Building

Started building at 2021/12/03 19:33:54
Calculating base
Updating mirror
Basing run on 7.1.0-1834 0a252cc55
Updating tree for run 03.12.2021-19.33
gocb is at 635fa6c, changes since last good build: none
libcouchbase is at f6822fc, changes since last good build: none
indexing is at a00b35f, changes since last good build: none
kv_engine is at 7e709d9, changes since last good build: none
ns_server is at ef29789, changes since last good build: none
couchstore is at 5aa7c4c, changes since last good build: 
 5aa7c4c MB-49810: commit time should be nanoseconds
couchdb is at fca2e6b, changes since last good build: none
testrunner is at c1989af, changes since last good build: 
 c1989af90 Fixed alter index cases for repetitive check
 8e4ec9e7c Fix for concurrent index case in free-tier tests
 ee48c0190 Eventing: fix testware issues
Switching eventing to unstable
eventing is at 9fce322, changes since last good build: none
Switching eventing-ee to unstable
eventing-ee is at 7060ef8, changes since last good build: none
Pulling in uncommitted change 160397 at refs/changes/97/160397/3
Total 23 (delta 15), reused 17 (delta 15)
[unstable df7f23d1] MB-100 : Do not fail n1ql timeout test if we are catching timeout exceptions successfully
 Author: abhijpes 
 Date: Mon Aug 30 08:24:38 2021 +0530
 1 file changed, 2 insertions(+), 2 deletions(-)
Pulling in uncommitted change 163449 at refs/changes/49/163449/1
Total 48 (delta 27), reused 39 (delta 27)
[unstable c95b4833] MB-100: collect data when test fails
 Author: AnkitPrabhu 
 Date: Tue Oct 12 14:02:31 2021 +0530
 1 file changed, 2 insertions(+)
Pulling in uncommitted change 165976 at refs/changes/76/165976/16
Total 5 (delta 3), reused 5 (delta 3)
[unstable c78efab5] MB-48538: Add new audit ids for different events
 Author: AnkitPrabhu 
 Date: Tue Nov 23 14:29:10 2021 +0530
 1 file changed, 337 insertions(+), 61 deletions(-)
Pulling in uncommitted change 166697 at refs/changes/97/166697/4
Total 10 (delta 9), reused 10 (delta 9)
[unstable 71fd2048] MB-49547: Check bucket instance to get kv node for dcp request
 Author: AnkitPrabhu 
 Date: Thu Dec 2 12:22:17 2021 +0530
 6 files changed, 38 insertions(+), 109 deletions(-)
Pulling in uncommitted change 166698 at refs/changes/98/166698/3
Total 4 (delta 3), reused 4 (delta 3)
[unstable e1e205a2] MB-48334 : Fail rebalance unless all functions deployed on current encryption level
 Author: abhijpes 
 Date: Wed Dec 1 14:51:29 2021 +0530
 1 file changed, 9 insertions(+)
Pulling in uncommitted change 166812 at refs/changes/12/166812/5
Total 24 (delta 20), reused 24 (delta 20)
[unstable 7ea2eb98] MB-49547: Refresh bucket info from streaming endpoint instance
 Author: AnkitPrabhu 
 Date: Thu Dec 2 13:55:55 2021 +0530
 7 files changed, 147 insertions(+), 235 deletions(-)
Building cmakefiles and deps
Building main product

Testing

Started testing at 2021/12/03 19:48:46
Testing mode: rebalance,n1ql,functional
Starting server
Starting server: attempt 1

Functional tests

2021/12/03 19:50:18 Started functional
Post "http://127.0.0.1:9000/pools/default": read tcp 127.0.0.1:56938->127.0.0.1:9000: read: connection reset by peer
Quota setup Post "http://127.0.0.1:9000/pools/default": read tcp 127.0.0.1:56938->127.0.0.1:9000: read: connection reset by peer
2021/12/03 19:50:27 Adding node: https://127.0.0.1:19001 with role: kv,index,n1ql to the cluster
2021/12/03 19:50:33 Error in adding nodes : , response: {"otpNode":"n_1@127.0.0.1"}
2021/12/03 19:50:33 Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/03 19:50:38 Removing node(s): [] from the cluster
2021/12/03 19:50:43 Rebalance progress: 2.748842592592592
2021/12/03 19:50:48 Rebalance progress: 7.288049768518519
2021/12/03 19:50:53 Rebalance progress: 10.61559606481481
2021/12/03 19:50:58 Rebalance progress: 15.13671875
2021/12/03 19:51:03 Rebalance progress: 18.51851851851852
2021/12/03 19:51:08 Rebalance progress: 23.05772569444444
2021/12/03 19:51:13 Rebalance progress: 27.66927083333333
2021/12/03 19:51:18 Rebalance progress: 100
=== RUN   TestAdvancedGetOps
2021/12/03 19:51:24 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/03 19:51:39 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/03 19:51:54 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/03 19:52:09 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/03 19:52:24 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/03 19:52:39 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/03 19:52:54 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/03 19:53:09 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/03 19:53:24 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/03 19:53:39 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/03 19:53:54 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/03 19:54:09 Deploying app: TestAdvancedGetOps. Item count src bucket: 0 dst bucket: 0
2021/12/03 19:54:10 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedGetOps' in metakv",
  "warnings": null
 }
}
2021/12/03 19:54:15 Waiting for app: TestAdvancedGetOps to get deployed
2021/12/03 19:54:20 Waiting for app: TestAdvancedGetOps to get deployed
2021/12/03 19:54:25 Waiting for app: TestAdvancedGetOps to get deployed
2021/12/03 19:54:30 Waiting for app: TestAdvancedGetOps to get deployed
2021/12/03 19:54:30 App: TestAdvancedGetOps got deployed
2021/12/03 19:54:30 Starting bucket ops to source bucket
2021/12/03 19:54:35 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/03 19:54:40 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/03 19:54:45 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/03 19:54:45 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/03 19:54:45 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 19:54:45 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/03 19:54:45 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 19:54:50 Waiting for app: TestAdvancedGetOps to get un-deployed
2021/12/03 19:54:55 Waiting for app: TestAdvancedGetOps to get un-deployed
2021/12/03 19:55:00 Waiting for app: TestAdvancedGetOps to get un-deployed
2021/12/03 19:55:05 Waiting for app: TestAdvancedGetOps to get un-deployed
2021/12/03 19:55:05 App: TestAdvancedGetOps got un-deployed
2021/12/03 19:55:05 No eventing-con process running
2021/12/03 19:55:05 Function: TestAdvancedGetOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedGetOps deleting in the background"
}
2021/12/03 19:55:07 Context: Bucket flush, response code: 200 dump: 
2021/12/03 19:55:17 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/03 19:55:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 19:55:19 Context: Bucket flush, response code: 200 dump: 
2021/12/03 19:55:29 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/03 19:55:34 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/03 19:55:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedGetOps (250.23s)
=== RUN   TestAdvancedInsertOps
2021/12/03 19:55:34 Deploying app: TestAdvancedInsertOps. Item count src bucket: 0 dst bucket: 0
2021/12/03 19:55:35 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedInsertOps' in metakv",
  "warnings": null
 }
}
2021/12/03 19:55:40 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/03 19:55:45 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/03 19:55:50 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/03 19:55:55 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/03 19:55:55 App: TestAdvancedInsertOps got deployed
2021/12/03 19:55:55 Starting bucket ops to source bucket
2021/12/03 19:56:01 Waiting for dst bucket item count to get to: 1024 curr count: 0
2021/12/03 19:56:01 src & dst bucket item count matched up. src bucket count: 1024 dst bucket count: 1024
2021/12/03 19:56:01 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 19:56:01 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/03 19:56:01 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 19:56:06 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/03 19:56:11 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/03 19:56:16 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/03 19:56:21 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/03 19:56:21 App: TestAdvancedInsertOps got un-deployed
2021/12/03 19:56:21 No eventing-con process running
2021/12/03 19:56:21 Function: TestAdvancedInsertOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedInsertOps deleting in the background"
}
2021/12/03 19:56:23 Context: Bucket flush, response code: 200 dump: 
2021/12/03 19:56:33 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/03 19:56:38 Waiting for dst bucket item count to get to: 0 curr count: 512
2021/12/03 19:56:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 19:56:40 Context: Bucket flush, response code: 200 dump: 
2021/12/03 19:56:50 Waiting for dst bucket item count to get to: 0 curr count: 512
2021/12/03 19:56:55 Waiting for dst bucket item count to get to: 0 curr count: 512
2021/12/03 19:56:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 19:56:55 Testing insert operation on source bucket
2021/12/03 19:56:55 Deploying app: TestAdvancedInsertOps. Item count src bucket: 0 dst bucket: 0
2021/12/03 19:56:56 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedInsertOps' in metakv",
  "warnings": null
 }
}
2021/12/03 19:57:01 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/03 19:57:06 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/03 19:57:11 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/03 19:57:16 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/03 19:57:16 App: TestAdvancedInsertOps got deployed
2021/12/03 19:57:16 Starting bucket ops to source bucket
2021/12/03 19:57:22 Waiting for dst bucket item count to get to: 2048 curr count: 0
2021/12/03 19:57:22 src & dst bucket item count matched up. src bucket count: 2048 dst bucket count: 2048
2021/12/03 19:57:22 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 19:57:22 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/03 19:57:22 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 19:57:27 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/03 19:57:32 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/03 19:57:37 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/03 19:57:42 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/03 19:57:42 App: TestAdvancedInsertOps got un-deployed
2021/12/03 19:57:42 No eventing-con process running
2021/12/03 19:57:42 Function: TestAdvancedInsertOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedInsertOps deleting in the background"
}
2021/12/03 19:57:44 Context: Bucket flush, response code: 200 dump: 
2021/12/03 19:57:54 Waiting for dst bucket item count to get to: 0 curr count: 2048
2021/12/03 19:57:59 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/03 19:57:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 19:58:01 Context: Bucket flush, response code: 200 dump: 
2021/12/03 19:58:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedInsertOps (151.70s)
=== RUN   TestAdvancedUpsertOps
2021/12/03 19:58:06 Starting bucket ops to source bucket
2021/12/03 19:58:06 Deploying app: TestAdvancedUpsertOps. Item count src bucket: 0 dst bucket: 0
2021/12/03 19:58:07 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedUpsertOps' in metakv",
  "warnings": null
 }
}
2021/12/03 19:58:12 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/03 19:58:17 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/03 19:58:22 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/03 19:58:27 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/03 19:58:27 App: TestAdvancedUpsertOps got deployed
2021/12/03 19:58:27 Starting bucket ops to source bucket
2021/12/03 19:58:33 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/03 19:58:39 Waiting for dst bucket item count to get to: 0 curr count: 512
2021/12/03 19:58:44 Waiting for dst bucket item count to get to: 0 curr count: 512
2021/12/03 19:58:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 19:58:44 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 19:58:44 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/03 19:58:44 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 19:58:49 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/03 19:58:54 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/03 19:58:59 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/03 19:59:04 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/03 19:59:04 App: TestAdvancedUpsertOps got un-deployed
2021/12/03 19:59:04 No eventing-con process running
2021/12/03 19:59:04 Function: TestAdvancedUpsertOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedUpsertOps deleting in the background"
}
2021/12/03 19:59:05 Context: Bucket flush, response code: 200 dump: 
2021/12/03 19:59:15 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/03 19:59:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 19:59:17 Context: Bucket flush, response code: 200 dump: 
2021/12/03 19:59:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 19:59:22 Testing upsert operation on source bucket
2021/12/03 19:59:22 Deploying app: TestAdvancedUpsertOps. Item count src bucket: 0 dst bucket: 0
2021/12/03 19:59:24 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedUpsertOps' in metakv",
  "warnings": null
 }
}
2021/12/03 19:59:29 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/03 19:59:34 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/03 19:59:39 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/03 19:59:44 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/03 19:59:44 App: TestAdvancedUpsertOps got deployed
2021/12/03 19:59:44 Starting bucket ops to source bucket
2021/12/03 19:59:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 19:59:45 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 19:59:45 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/03 19:59:45 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 19:59:50 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/03 19:59:55 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/03 20:00:00 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/03 20:00:05 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/03 20:00:05 App: TestAdvancedUpsertOps got un-deployed
2021/12/03 20:00:05 No eventing-con process running
2021/12/03 20:00:05 Function: TestAdvancedUpsertOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedUpsertOps deleting in the background"
}
2021/12/03 20:00:06 Context: Bucket flush, response code: 200 dump: 
2021/12/03 20:00:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 20:00:14 Context: Bucket flush, response code: 200 dump: 
2021/12/03 20:00:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedUpsertOps (133.15s)
=== RUN   TestAdvancedReplaceOps
2021/12/03 20:00:19 Starting bucket ops to source bucket
2021/12/03 20:00:20 Deploying app: TestAdvancedReplaceOps. Item count src bucket: 0 dst bucket: 0
2021/12/03 20:00:21 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedReplaceOps' in metakv",
  "warnings": null
 }
}
2021/12/03 20:00:26 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/03 20:00:31 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/03 20:00:36 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/03 20:00:41 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/03 20:00:41 App: TestAdvancedReplaceOps got deployed
2021/12/03 20:00:41 Starting bucket ops to source bucket
2021/12/03 20:00:47 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/03 20:00:52 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/03 20:00:57 Waiting for dst bucket item count to get to: 0 curr count: 512
2021/12/03 20:00:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 20:00:57 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 20:00:57 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/03 20:00:57 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 20:01:02 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/03 20:01:07 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/03 20:01:12 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/03 20:01:17 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/03 20:01:17 App: TestAdvancedReplaceOps got un-deployed
2021/12/03 20:01:17 No eventing-con process running
2021/12/03 20:01:17 Function: TestAdvancedReplaceOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedReplaceOps deleting in the background"
}
2021/12/03 20:01:19 Context: Bucket flush, response code: 200 dump: 
2021/12/03 20:01:29 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/03 20:01:34 Waiting for dst bucket item count to get to: 0 curr count: 512
2021/12/03 20:01:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 20:01:36 Context: Bucket flush, response code: 200 dump: 
2021/12/03 20:01:41 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 20:01:41 Testing upsert operation on source bucket
2021/12/03 20:01:41 Deploying app: TestAdvancedReplaceOps. Item count src bucket: 0 dst bucket: 0
2021/12/03 20:01:42 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedReplaceOps' in metakv",
  "warnings": null
 }
}
2021/12/03 20:01:47 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/03 20:01:52 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/03 20:01:57 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/03 20:02:02 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/03 20:02:02 App: TestAdvancedReplaceOps got deployed
2021/12/03 20:02:02 Starting bucket ops to source bucket
2021/12/03 20:02:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 20:02:03 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 20:02:03 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/03 20:02:03 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 20:02:08 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/03 20:02:13 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/03 20:02:18 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/03 20:02:23 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/03 20:02:23 App: TestAdvancedReplaceOps got un-deployed
2021/12/03 20:02:23 No eventing-con process running
2021/12/03 20:02:23 Function: TestAdvancedReplaceOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedReplaceOps deleting in the background"
}
2021/12/03 20:02:25 Context: Bucket flush, response code: 200 dump: 
2021/12/03 20:02:35 Waiting for dst bucket item count to get to: 0 curr count: 390
2021/12/03 20:02:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 20:02:37 Context: Bucket flush, response code: 200 dump: 
2021/12/03 20:02:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedReplaceOps (142.81s)
=== RUN   TestAdvancedDeleteOps
2021/12/03 20:02:42 Starting bucket ops to source bucket
2021/12/03 20:02:42 Deploying app: TestAdvancedDeleteOps. Item count src bucket: 0 dst bucket: 0
2021/12/03 20:02:43 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedDeleteOps' in metakv",
  "warnings": null
 }
}
2021/12/03 20:02:48 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/03 20:02:53 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/03 20:02:58 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/03 20:03:03 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/03 20:03:03 App: TestAdvancedDeleteOps got deployed
2021/12/03 20:03:03 Starting bucket ops to source bucket
2021/12/03 20:03:08 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/03 20:03:13 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/03 20:03:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 20:03:13 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 20:03:13 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/03 20:03:13 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 20:03:18 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/03 20:03:23 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/03 20:03:28 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/03 20:03:33 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/03 20:03:33 App: TestAdvancedDeleteOps got un-deployed
2021/12/03 20:03:33 No eventing-con process running
2021/12/03 20:03:33 Function: TestAdvancedDeleteOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedDeleteOps deleting in the background"
}
2021/12/03 20:03:35 Context: Bucket flush, response code: 200 dump: 
2021/12/03 20:03:45 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/03 20:03:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 20:03:47 Context: Bucket flush, response code: 200 dump: 
2021/12/03 20:03:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 20:03:52 Testing delete operation on source bucket
2021/12/03 20:03:52 Deploying app: TestAdvancedDeleteOps. Item count src bucket: 0 dst bucket: 0
2021/12/03 20:03:53 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedDeleteOps' in metakv",
  "warnings": null
 }
}
2021/12/03 20:03:58 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/03 20:04:03 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/03 20:04:08 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/03 20:04:13 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/03 20:04:13 App: TestAdvancedDeleteOps got deployed
2021/12/03 20:04:13 Starting bucket ops to source bucket
2021/12/03 20:04:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 20:04:13 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 20:04:13 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/03 20:04:13 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 20:04:18 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/03 20:04:23 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/03 20:04:28 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/03 20:04:33 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/03 20:04:33 App: TestAdvancedDeleteOps got un-deployed
2021/12/03 20:04:33 No eventing-con process running
2021/12/03 20:04:33 Function: TestAdvancedDeleteOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedDeleteOps deleting in the background"
}
2021/12/03 20:04:35 Context: Bucket flush, response code: 200 dump: 
2021/12/03 20:04:40 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 20:04:43 Context: Bucket flush, response code: 200 dump: 
2021/12/03 20:04:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedDeleteOps (125.93s)
=== RUN   TestEnoentAdvancedGet
2021/12/03 20:04:48 Deploying app: TestEnoentAdvancedGet. Item count src bucket: 0 dst bucket: 0
2021/12/03 20:04:49 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedGet' in metakv",
  "warnings": null
 }
}
2021/12/03 20:04:54 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/03 20:04:59 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/03 20:05:04 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/03 20:05:09 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/03 20:05:09 App: TestEnoentAdvancedGet got deployed
2021/12/03 20:05:09 Starting bucket ops to source bucket
2021/12/03 20:05:14 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/03 20:05:19 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/03 20:05:19 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/03 20:05:19 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 20:05:19 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/03 20:05:19 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 20:05:24 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/03 20:05:29 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/03 20:05:34 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/03 20:05:39 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/03 20:05:39 App: TestEnoentAdvancedGet got un-deployed
2021/12/03 20:05:39 No eventing-con process running
2021/12/03 20:05:39 Function: TestEnoentAdvancedGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedGet deleting in the background"
}
2021/12/03 20:05:41 Context: Bucket flush, response code: 200 dump: 
2021/12/03 20:05:51 Waiting for dst bucket item count to get to: 0 curr count: 92
2021/12/03 20:05:56 Waiting for dst bucket item count to get to: 0 curr count: 92
2021/12/03 20:05:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 20:05:58 Context: Bucket flush, response code: 200 dump: 
2021/12/03 20:06:08 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/03 20:06:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 20:06:08 Testing get enoent operation on source bucket
2021/12/03 20:06:08 Deploying app: TestEnoentAdvancedGet. Item count src bucket: 0 dst bucket: 0
2021/12/03 20:06:09 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedGet' in metakv",
  "warnings": null
 }
}
2021/12/03 20:06:14 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/03 20:06:19 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/03 20:06:24 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/03 20:06:29 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/03 20:06:29 App: TestEnoentAdvancedGet got deployed
2021/12/03 20:06:29 Starting bucket ops to source bucket
2021/12/03 20:06:34 Waiting for dst bucket item count to get to: 200 curr count: 0
2021/12/03 20:06:39 Waiting for dst bucket item count to get to: 200 curr count: 100
2021/12/03 20:06:39 src & dst bucket item count matched up. src bucket count: 200 dst bucket count: 200
2021/12/03 20:06:39 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 20:06:39 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/03 20:06:39 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 20:06:44 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/03 20:06:49 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/03 20:06:54 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/03 20:06:59 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/03 20:06:59 App: TestEnoentAdvancedGet got un-deployed
2021/12/03 20:06:59 No eventing-con process running
2021/12/03 20:06:59 Function: TestEnoentAdvancedGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedGet deleting in the background"
}
2021/12/03 20:07:01 Context: Bucket flush, response code: 200 dump: 
2021/12/03 20:07:11 Waiting for dst bucket item count to get to: 0 curr count: 186
2021/12/03 20:07:16 Waiting for dst bucket item count to get to: 0 curr count: 186
2021/12/03 20:07:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 20:07:18 Context: Bucket flush, response code: 200 dump: 
2021/12/03 20:07:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEnoentAdvancedGet (155.04s)
=== RUN   TestEnoentAdvancedReplace
2021/12/03 20:07:23 Deploying app: TestEnoentAdvancedReplace. Item count src bucket: 0 dst bucket: 0
2021/12/03 20:07:24 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedReplace' in metakv",
  "warnings": null
 }
}
2021/12/03 20:07:29 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/03 20:07:34 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/03 20:07:39 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/03 20:07:44 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/03 20:07:44 App: TestEnoentAdvancedReplace got deployed
2021/12/03 20:07:44 Starting bucket ops to source bucket
2021/12/03 20:07:49 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/03 20:07:54 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/03 20:07:54 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/03 20:07:54 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 20:07:54 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/03 20:07:54 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 20:07:59 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/03 20:08:04 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/03 20:08:09 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/03 20:08:14 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/03 20:08:14 App: TestEnoentAdvancedReplace got un-deployed
2021/12/03 20:08:14 No eventing-con process running
2021/12/03 20:08:14 Function: TestEnoentAdvancedReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedReplace deleting in the background"
}
2021/12/03 20:08:16 Context: Bucket flush, response code: 200 dump: 
2021/12/03 20:08:26 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/03 20:08:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 20:08:28 Context: Bucket flush, response code: 200 dump: 
2021/12/03 20:08:38 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/03 20:08:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 20:08:38 Testing get enoent operation on source bucket
2021/12/03 20:08:38 Deploying app: TestEnoentAdvancedReplace. Item count src bucket: 0 dst bucket: 0
2021/12/03 20:08:39 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedReplace' in metakv",
  "warnings": null
 }
}
2021/12/03 20:08:44 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/03 20:08:49 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/03 20:08:54 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/03 20:08:59 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/03 20:08:59 App: TestEnoentAdvancedReplace got deployed
2021/12/03 20:08:59 Starting bucket ops to source bucket
2021/12/03 20:09:04 Waiting for dst bucket item count to get to: 200 curr count: 0
2021/12/03 20:09:09 Waiting for dst bucket item count to get to: 200 curr count: 100
2021/12/03 20:09:09 src & dst bucket item count matched up. src bucket count: 200 dst bucket count: 200
2021/12/03 20:09:09 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 20:09:09 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/03 20:09:09 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 20:09:14 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/03 20:09:19 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/03 20:09:24 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/03 20:09:29 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/03 20:09:29 App: TestEnoentAdvancedReplace got un-deployed
2021/12/03 20:09:29 No eventing-con process running
2021/12/03 20:09:29 Function: TestEnoentAdvancedReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedReplace deleting in the background"
}
2021/12/03 20:09:31 Context: Bucket flush, response code: 200 dump: 
2021/12/03 20:09:41 Waiting for dst bucket item count to get to: 0 curr count: 190
2021/12/03 20:09:46 Waiting for dst bucket item count to get to: 0 curr count: 190
2021/12/03 20:09:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 20:09:48 Context: Bucket flush, response code: 200 dump: 
2021/12/03 20:09:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEnoentAdvancedReplace (150.09s)
=== RUN   TestEnoentAdvancedDelete
2021/12/03 20:09:53 Deploying app: TestEnoentAdvancedDelete. Item count src bucket: 0 dst bucket: 0
2021/12/03 20:09:54 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedDelete' in metakv",
  "warnings": null
 }
}
2021/12/03 20:09:59 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/03 20:10:04 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/03 20:10:09 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/03 20:10:14 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/03 20:10:14 App: TestEnoentAdvancedDelete got deployed
2021/12/03 20:10:14 Starting bucket ops to source bucket
2021/12/03 20:10:19 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/03 20:10:24 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/03 20:10:24 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/03 20:10:24 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 20:10:24 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/03 20:10:24 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 20:10:29 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/03 20:10:34 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/03 20:10:39 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/03 20:10:44 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/03 20:10:44 App: TestEnoentAdvancedDelete got un-deployed
2021/12/03 20:10:44 No eventing-con process running
2021/12/03 20:10:44 Function: TestEnoentAdvancedDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedDelete deleting in the background"
}
2021/12/03 20:10:46 Context: Bucket flush, response code: 200 dump: 
2021/12/03 20:10:56 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/03 20:10:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 20:10:58 Context: Bucket flush, response code: 200 dump: 
2021/12/03 20:11:08 Waiting for dst bucket item count to get to: 0 curr count: 54
2021/12/03 20:11:13 Waiting for dst bucket item count to get to: 0 curr count: 4
2021/12/03 20:11:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 20:11:13 Testing get enoent operation on source bucket
2021/12/03 20:11:13 Deploying app: TestEnoentAdvancedDelete. Item count src bucket: 0 dst bucket: 0
2021/12/03 20:11:14 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedDelete' in metakv",
  "warnings": null
 }
}
2021/12/03 20:11:19 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/03 20:11:24 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/03 20:11:29 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/03 20:11:34 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/03 20:11:34 App: TestEnoentAdvancedDelete got deployed
2021/12/03 20:11:34 Starting bucket ops to source bucket
2021/12/03 20:11:39 Waiting for dst bucket item count to get to: 200 curr count: 0
2021/12/03 20:11:44 Waiting for dst bucket item count to get to: 200 curr count: 100
2021/12/03 20:11:44 src & dst bucket item count matched up. src bucket count: 200 dst bucket count: 200
2021/12/03 20:11:44 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 20:11:44 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/03 20:11:44 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 20:11:49 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/03 20:11:54 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/03 20:11:59 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/03 20:12:04 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/03 20:12:04 App: TestEnoentAdvancedDelete got un-deployed
2021/12/03 20:12:04 No eventing-con process running
2021/12/03 20:12:04 Function: TestEnoentAdvancedDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedDelete deleting in the background"
}
2021/12/03 20:12:06 Context: Bucket flush, response code: 200 dump: 
2021/12/03 20:12:16 Waiting for dst bucket item count to get to: 0 curr count: 200
2021/12/03 20:12:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 20:12:18 Context: Bucket flush, response code: 200 dump: 
2021/12/03 20:12:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEnoentAdvancedDelete (150.09s)
=== RUN   TestAdvancedInsertKeyExist
2021/12/03 20:12:23 Deploying app: TestAdvancedInsertKeyExist. Item count src bucket: 0 dst bucket: 0
2021/12/03 20:12:24 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedInsertKeyExist' in metakv",
  "warnings": null
 }
}
2021/12/03 20:12:29 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/03 20:12:34 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/03 20:12:39 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/03 20:12:44 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/03 20:12:44 App: TestAdvancedInsertKeyExist got deployed
2021/12/03 20:12:44 Starting bucket ops to source bucket
2021/12/03 20:12:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 20:12:44 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 20:12:44 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/03 20:12:44 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 20:12:49 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/03 20:12:54 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/03 20:12:59 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/03 20:13:04 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/03 20:13:04 App: TestAdvancedInsertKeyExist got un-deployed
2021/12/03 20:13:04 No eventing-con process running
2021/12/03 20:13:04 Function: TestAdvancedInsertKeyExist delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedInsertKeyExist deleting in the background"
}
2021/12/03 20:13:06 Context: Bucket flush, response code: 200 dump: 
2021/12/03 20:13:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 20:13:13 Context: Bucket flush, response code: 200 dump: 
2021/12/03 20:13:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 20:13:18 Pumping document in the destination bucket
2021/12/03 20:13:18 Starting bucket ops to source bucket
2021/12/03 20:13:19 Deploying app: TestAdvancedInsertKeyExist. Item count src bucket: 0 dst bucket: 0
2021/12/03 20:13:20 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedInsertKeyExist' in metakv",
  "warnings": null
 }
}
2021/12/03 20:13:25 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/03 20:13:30 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/03 20:13:35 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/03 20:13:40 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/03 20:13:40 App: TestAdvancedInsertKeyExist got deployed
2021/12/03 20:13:40 Starting bucket ops to source bucket
2021/12/03 20:13:45 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/03 20:13:50 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/03 20:13:55 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/03 20:13:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 20:13:55 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 20:13:55 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/03 20:13:55 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 20:14:00 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/03 20:14:05 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/03 20:14:10 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/03 20:14:15 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/03 20:14:15 App: TestAdvancedInsertKeyExist got un-deployed
2021/12/03 20:14:15 No eventing-con process running
2021/12/03 20:14:15 Function: TestAdvancedInsertKeyExist delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedInsertKeyExist deleting in the background"
}
2021/12/03 20:14:17 Context: Bucket flush, response code: 200 dump: 
2021/12/03 20:14:27 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/03 20:14:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 20:14:28 Context: Bucket flush, response code: 200 dump: 
2021/12/03 20:14:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedInsertKeyExist (130.69s)
=== RUN   TestExipryGet
2021/12/03 20:14:33 Deploying app: TestExipryGet. Item count src bucket: 0 dst bucket: 0
2021/12/03 20:14:35 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryGet' in metakv",
  "warnings": null
 }
}
2021/12/03 20:14:40 Waiting for app: TestExipryGet to get deployed
2021/12/03 20:14:45 Waiting for app: TestExipryGet to get deployed
2021/12/03 20:14:50 Waiting for app: TestExipryGet to get deployed
2021/12/03 20:14:55 Waiting for app: TestExipryGet to get deployed
2021/12/03 20:14:55 App: TestExipryGet got deployed
2021/12/03 20:14:55 Starting bucket ops to source bucket
2021/12/03 20:15:40 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/03 20:15:40 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 20:15:40 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 20:15:40 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/03 20:15:40 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 20:15:45 Waiting for app: TestExipryGet to get un-deployed
2021/12/03 20:15:50 Waiting for app: TestExipryGet to get un-deployed
2021/12/03 20:15:55 Waiting for app: TestExipryGet to get un-deployed
2021/12/03 20:16:00 Waiting for app: TestExipryGet to get un-deployed
2021/12/03 20:16:00 App: TestExipryGet got un-deployed
2021/12/03 20:16:00 No eventing-con process running
2021/12/03 20:16:00 Function: TestExipryGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryGet deleting in the background"
}
2021/12/03 20:16:02 Context: Bucket flush, response code: 200 dump: 
2021/12/03 20:16:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 20:16:09 Context: Bucket flush, response code: 200 dump: 
2021/12/03 20:16:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 20:16:14 Pumping document in the destination bucket for expiry get
2021/12/03 20:16:14 Starting bucket ops to source bucket
2021/12/03 20:16:14 Deploying app: TestExipryGet. Item count src bucket: 0 dst bucket: 0
2021/12/03 20:16:15 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryGet' in metakv",
  "warnings": null
 }
}
2021/12/03 20:16:20 Waiting for app: TestExipryGet to get deployed
2021/12/03 20:16:25 Waiting for app: TestExipryGet to get deployed
2021/12/03 20:16:30 Waiting for app: TestExipryGet to get deployed
2021/12/03 20:16:35 Waiting for app: TestExipryGet to get deployed
2021/12/03 20:16:35 App: TestExipryGet got deployed
2021/12/03 20:16:35 Starting bucket ops to source bucket
2021/12/03 20:17:21 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/03 20:17:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 20:17:21 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 20:17:21 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/03 20:17:21 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 20:17:26 Waiting for app: TestExipryGet to get un-deployed
2021/12/03 20:17:31 Waiting for app: TestExipryGet to get un-deployed
2021/12/03 20:17:36 Waiting for app: TestExipryGet to get un-deployed
2021/12/03 20:17:41 Waiting for app: TestExipryGet to get un-deployed
2021/12/03 20:17:41 App: TestExipryGet got un-deployed
2021/12/03 20:17:41 No eventing-con process running
2021/12/03 20:17:41 Function: TestExipryGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryGet deleting in the background"
}
2021/12/03 20:17:43 Context: Bucket flush, response code: 200 dump: 
2021/12/03 20:17:53 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/03 20:17:58 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/03 20:17:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 20:17:59 Context: Bucket flush, response code: 200 dump: 
2021/12/03 20:18:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestExipryGet (210.89s)
=== RUN   TestExipryInsert
2021/12/03 20:18:04 Deploying app: TestExipryInsert. Item count src bucket: 0 dst bucket: 0
2021/12/03 20:18:06 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryInsert' in metakv",
  "warnings": null
 }
}
2021/12/03 20:18:11 Waiting for app: TestExipryInsert to get deployed
2021/12/03 20:18:16 Waiting for app: TestExipryInsert to get deployed
2021/12/03 20:18:21 Waiting for app: TestExipryInsert to get deployed
2021/12/03 20:18:26 Waiting for app: TestExipryInsert to get deployed
2021/12/03 20:18:26 App: TestExipryInsert got deployed
2021/12/03 20:18:26 Starting bucket ops to source bucket
2021/12/03 20:19:11 Waiting for dst bucket item count to get to: 0 curr count: 200
2021/12/03 20:19:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 20:19:11 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 20:19:11 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/03 20:19:11 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 20:19:16 Waiting for app: TestExipryInsert to get un-deployed
2021/12/03 20:19:21 Waiting for app: TestExipryInsert to get un-deployed
2021/12/03 20:19:26 Waiting for app: TestExipryInsert to get un-deployed
2021/12/03 20:19:31 Waiting for app: TestExipryInsert to get un-deployed
2021/12/03 20:19:31 App: TestExipryInsert got un-deployed
2021/12/03 20:19:31 No eventing-con process running
2021/12/03 20:19:31 Function: TestExipryInsert delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryInsert deleting in the background"
}
2021/12/03 20:19:33 Context: Bucket flush, response code: 200 dump: 
2021/12/03 20:19:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 20:19:40 Context: Bucket flush, response code: 200 dump: 
2021/12/03 20:19:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 20:19:45 Testing on the destination bucket for expiry insert
2021/12/03 20:19:45 Deploying app: TestExipryInsert. Item count src bucket: 0 dst bucket: 0
2021/12/03 20:19:46 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryInsert' in metakv",
  "warnings": null
 }
}
2021/12/03 20:19:51 Waiting for app: TestExipryInsert to get deployed
2021/12/03 20:19:56 Waiting for app: TestExipryInsert to get deployed
2021/12/03 20:20:01 Waiting for app: TestExipryInsert to get deployed
2021/12/03 20:20:06 Waiting for app: TestExipryInsert to get deployed
2021/12/03 20:20:06 App: TestExipryInsert got deployed
2021/12/03 20:20:06 Starting bucket ops to source bucket
2021/12/03 20:20:51 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/03 20:20:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 20:20:51 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 20:20:51 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/03 20:20:52 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 20:20:57 Waiting for app: TestExipryInsert to get un-deployed
2021/12/03 20:21:02 Waiting for app: TestExipryInsert to get un-deployed
2021/12/03 20:21:07 Waiting for app: TestExipryInsert to get un-deployed
2021/12/03 20:21:12 Waiting for app: TestExipryInsert to get un-deployed
2021/12/03 20:21:12 App: TestExipryInsert got un-deployed
2021/12/03 20:21:12 No eventing-con process running
2021/12/03 20:21:12 Function: TestExipryInsert delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryInsert deleting in the background"
}
2021/12/03 20:21:13 Context: Bucket flush, response code: 200 dump: 
2021/12/03 20:21:23 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/03 20:21:28 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/03 20:21:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 20:21:30 Context: Bucket flush, response code: 200 dump: 
2021/12/03 20:21:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestExipryInsert (210.79s)
=== RUN   TestExipryReplace
2021/12/03 20:21:35 Deploying app: TestExipryReplace. Item count src bucket: 0 dst bucket: 0
2021/12/03 20:21:36 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryReplace' in metakv",
  "warnings": null
 }
}
2021/12/03 20:21:41 Waiting for app: TestExipryReplace to get deployed
2021/12/03 20:21:46 Waiting for app: TestExipryReplace to get deployed
2021/12/03 20:21:51 Waiting for app: TestExipryReplace to get deployed
2021/12/03 20:21:56 Waiting for app: TestExipryReplace to get deployed
2021/12/03 20:21:56 App: TestExipryReplace got deployed
2021/12/03 20:21:56 Starting bucket ops to source bucket
2021/12/03 20:22:42 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/03 20:22:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 20:22:42 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 20:22:42 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/03 20:22:42 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 20:22:47 Waiting for app: TestExipryReplace to get un-deployed
2021/12/03 20:22:52 Waiting for app: TestExipryReplace to get un-deployed
2021/12/03 20:22:57 Waiting for app: TestExipryReplace to get un-deployed
2021/12/03 20:23:02 Waiting for app: TestExipryReplace to get un-deployed
2021/12/03 20:23:02 App: TestExipryReplace got un-deployed
2021/12/03 20:23:02 No eventing-con process running
2021/12/03 20:23:02 Function: TestExipryReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryReplace deleting in the background"
}
2021/12/03 20:23:04 Context: Bucket flush, response code: 200 dump: 
2021/12/03 20:23:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 20:23:11 Context: Bucket flush, response code: 200 dump: 
2021/12/03 20:23:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 20:23:16 Testing on the destination bucket for expiry insert
2021/12/03 20:23:16 Deploying app: TestExipryReplace. Item count src bucket: 0 dst bucket: 0
2021/12/03 20:23:17 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryReplace' in metakv",
  "warnings": null
 }
}
2021/12/03 20:23:22 Waiting for app: TestExipryReplace to get deployed
2021/12/03 20:23:27 Waiting for app: TestExipryReplace to get deployed
2021/12/03 20:23:32 Waiting for app: TestExipryReplace to get deployed
2021/12/03 20:23:37 Waiting for app: TestExipryReplace to get deployed
2021/12/03 20:23:37 App: TestExipryReplace got deployed
2021/12/03 20:23:37 Starting bucket ops to source bucket
2021/12/03 20:24:22 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/03 20:24:27 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/03 20:24:32 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/03 20:24:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 20:24:32 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 20:24:32 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/03 20:24:32 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 20:24:37 Waiting for app: TestExipryReplace to get un-deployed
2021/12/03 20:24:42 Waiting for app: TestExipryReplace to get un-deployed
2021/12/03 20:24:47 Waiting for app: TestExipryReplace to get un-deployed
2021/12/03 20:24:52 Waiting for app: TestExipryReplace to get un-deployed
2021/12/03 20:24:52 App: TestExipryReplace got un-deployed
2021/12/03 20:24:52 No eventing-con process running
2021/12/03 20:24:52 Function: TestExipryReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryReplace deleting in the background"
}
2021/12/03 20:24:54 Context: Bucket flush, response code: 200 dump: 
2021/12/03 20:25:04 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/03 20:25:09 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/03 20:25:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 20:25:11 Context: Bucket flush, response code: 200 dump: 
2021/12/03 20:25:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestExipryReplace (220.89s)
=== RUN   TestExipryUpsert
2021/12/03 20:25:16 Deploying app: TestExipryUpsert. Item count src bucket: 0 dst bucket: 0
2021/12/03 20:25:17 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryUpsert' in metakv",
  "warnings": null
 }
}
2021/12/03 20:25:22 Waiting for app: TestExipryUpsert to get deployed
2021/12/03 20:25:27 Waiting for app: TestExipryUpsert to get deployed
2021/12/03 20:25:32 Waiting for app: TestExipryUpsert to get deployed
2021/12/03 20:25:37 Waiting for app: TestExipryUpsert to get deployed
2021/12/03 20:25:37 App: TestExipryUpsert got deployed
2021/12/03 20:25:37 Starting bucket ops to source bucket
2021/12/03 20:26:22 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/03 20:26:27 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/03 20:26:32 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/03 20:26:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 20:26:33 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 20:26:33 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/03 20:26:33 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 20:26:38 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/03 20:26:43 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/03 20:26:48 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/03 20:26:53 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/03 20:26:53 App: TestExipryUpsert got un-deployed
2021/12/03 20:26:53 No eventing-con process running
2021/12/03 20:26:53 Function: TestExipryUpsert delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryUpsert deleting in the background"
}
2021/12/03 20:26:54 Context: Bucket flush, response code: 200 dump: 
2021/12/03 20:26:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 20:27:02 Context: Bucket flush, response code: 200 dump: 
2021/12/03 20:27:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 20:27:07 Testing on the destination bucket for expiry insert
2021/12/03 20:27:07 Deploying app: TestExipryUpsert. Item count src bucket: 0 dst bucket: 0
2021/12/03 20:27:08 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryUpsert' in metakv",
  "warnings": null
 }
}
2021/12/03 20:27:13 Waiting for app: TestExipryUpsert to get deployed
2021/12/03 20:27:18 Waiting for app: TestExipryUpsert to get deployed
2021/12/03 20:27:23 Waiting for app: TestExipryUpsert to get deployed
2021/12/03 20:27:28 Waiting for app: TestExipryUpsert to get deployed
2021/12/03 20:27:28 App: TestExipryUpsert got deployed
2021/12/03 20:27:28 Starting bucket ops to source bucket
2021/12/03 20:28:13 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/03 20:28:18 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/03 20:28:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 20:28:18 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 20:28:18 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/03 20:28:18 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 20:28:23 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/03 20:28:28 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/03 20:28:33 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/03 20:28:38 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/03 20:28:38 App: TestExipryUpsert got un-deployed
2021/12/03 20:28:38 No eventing-con process running
2021/12/03 20:28:38 Function: TestExipryUpsert delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryUpsert deleting in the background"
}
2021/12/03 20:28:40 Context: Bucket flush, response code: 200 dump: 
2021/12/03 20:28:50 Waiting for dst bucket item count to get to: 0 curr count: 63
2021/12/03 20:28:55 Waiting for dst bucket item count to get to: 0 curr count: 63
2021/12/03 20:28:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 20:28:57 Context: Bucket flush, response code: 200 dump: 
2021/12/03 20:29:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestExipryUpsert (225.85s)
=== RUN   TestCasReplace
2021/12/03 20:29:02 Deploying app: TestCasReplace. Item count src bucket: 0 dst bucket: 0
2021/12/03 20:29:03 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCasReplace' in metakv",
  "warnings": null
 }
}
2021/12/03 20:29:08 Waiting for app: TestCasReplace to get deployed
2021/12/03 20:29:13 Waiting for app: TestCasReplace to get deployed
2021/12/03 20:29:18 Waiting for app: TestCasReplace to get deployed
2021/12/03 20:29:23 Waiting for app: TestCasReplace to get deployed
2021/12/03 20:29:23 App: TestCasReplace got deployed
2021/12/03 20:29:23 Starting bucket ops to source bucket
2021/12/03 20:29:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 20:29:23 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 20:29:23 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/03 20:29:23 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 20:29:28 Waiting for app: TestCasReplace to get un-deployed
2021/12/03 20:29:33 Waiting for app: TestCasReplace to get un-deployed
2021/12/03 20:29:38 Waiting for app: TestCasReplace to get un-deployed
2021/12/03 20:29:43 Waiting for app: TestCasReplace to get un-deployed
2021/12/03 20:29:43 App: TestCasReplace got un-deployed
2021/12/03 20:29:43 No eventing-con process running
2021/12/03 20:29:43 Function: TestCasReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCasReplace deleting in the background"
}
2021/12/03 20:29:45 Context: Bucket flush, response code: 200 dump: 
2021/12/03 20:29:55 Waiting for dst bucket item count to get to: 0 curr count: 30
2021/12/03 20:29:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 20:29:57 Context: Bucket flush, response code: 200 dump: 
2021/12/03 20:30:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 20:30:02 Testing on the destination bucket for upsert with cas
2021/12/03 20:30:02 Starting bucket ops to source bucket
2021/12/03 20:30:02 Deploying app: TestCasReplace. Item count src bucket: 0 dst bucket: 0
2021/12/03 20:30:03 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCasReplace' in metakv",
  "warnings": null
 }
}
2021/12/03 20:30:08 Waiting for app: TestCasReplace to get deployed
2021/12/03 20:30:13 Waiting for app: TestCasReplace to get deployed
2021/12/03 20:30:18 Waiting for app: TestCasReplace to get deployed
2021/12/03 20:30:23 Waiting for app: TestCasReplace to get deployed
2021/12/03 20:30:23 App: TestCasReplace got deployed
2021/12/03 20:30:23 Starting bucket ops to source bucket
2021/12/03 20:30:29 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/03 20:30:34 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/03 20:30:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 20:30:34 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 20:30:34 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/03 20:30:34 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 20:30:39 Waiting for app: TestCasReplace to get un-deployed
2021/12/03 20:30:44 Waiting for app: TestCasReplace to get un-deployed
2021/12/03 20:30:49 Waiting for app: TestCasReplace to get un-deployed
2021/12/03 20:30:54 Waiting for app: TestCasReplace to get un-deployed
2021/12/03 20:30:54 App: TestCasReplace got un-deployed
2021/12/03 20:30:54 No eventing-con process running
2021/12/03 20:30:54 Function: TestCasReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCasReplace deleting in the background"
}
2021/12/03 20:30:55 Context: Bucket flush, response code: 200 dump: 
2021/12/03 20:31:05 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/03 20:31:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 20:31:07 Context: Bucket flush, response code: 200 dump: 
2021/12/03 20:31:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCasReplace (130.37s)
=== RUN   TestCasDelete
2021/12/03 20:31:12 Deploying app: TestCasDelete. Item count src bucket: 0 dst bucket: 0
2021/12/03 20:31:13 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCasDelete' in metakv",
  "warnings": null
 }
}
2021/12/03 20:31:18 Waiting for app: TestCasDelete to get deployed
2021/12/03 20:31:23 Waiting for app: TestCasDelete to get deployed
2021/12/03 20:31:28 Waiting for app: TestCasDelete to get deployed
2021/12/03 20:31:33 Waiting for app: TestCasDelete to get deployed
2021/12/03 20:31:33 App: TestCasDelete got deployed
2021/12/03 20:31:33 Starting bucket ops to source bucket
2021/12/03 20:31:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 20:31:34 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 20:31:34 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/03 20:31:34 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 20:31:39 Waiting for app: TestCasDelete to get un-deployed
2021/12/03 20:31:44 Waiting for app: TestCasDelete to get un-deployed
2021/12/03 20:31:49 Waiting for app: TestCasDelete to get un-deployed
2021/12/03 20:31:54 Waiting for app: TestCasDelete to get un-deployed
2021/12/03 20:31:54 App: TestCasDelete got un-deployed
2021/12/03 20:31:54 No eventing-con process running
2021/12/03 20:31:54 Function: TestCasDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCasDelete deleting in the background"
}
2021/12/03 20:31:56 Context: Bucket flush, response code: 200 dump: 
2021/12/03 20:32:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 20:32:03 Context: Bucket flush, response code: 200 dump: 
2021/12/03 20:32:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 20:32:08 Testing on the destination bucket for delete with cas
2021/12/03 20:32:08 Starting bucket ops to source bucket
2021/12/03 20:32:08 Deploying app: TestCasDelete. Item count src bucket: 0 dst bucket: 0
2021/12/03 20:32:09 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCasDelete' in metakv",
  "warnings": null
 }
}
2021/12/03 20:32:14 Waiting for app: TestCasDelete to get deployed
2021/12/03 20:32:19 Waiting for app: TestCasDelete to get deployed
2021/12/03 20:32:24 Waiting for app: TestCasDelete to get deployed
2021/12/03 20:32:29 Waiting for app: TestCasDelete to get deployed
2021/12/03 20:32:29 App: TestCasDelete got deployed
2021/12/03 20:32:29 Starting bucket ops to source bucket
2021/12/03 20:32:34 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/03 20:32:39 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/03 20:32:44 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/03 20:32:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 20:32:44 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 20:32:44 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/03 20:32:45 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 20:32:50 Waiting for app: TestCasDelete to get un-deployed
2021/12/03 20:32:55 Waiting for app: TestCasDelete to get un-deployed
2021/12/03 20:33:00 Waiting for app: TestCasDelete to get un-deployed
2021/12/03 20:33:05 Waiting for app: TestCasDelete to get un-deployed
2021/12/03 20:33:05 App: TestCasDelete got un-deployed
2021/12/03 20:33:05 No eventing-con process running
2021/12/03 20:33:05 Function: TestCasDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCasDelete deleting in the background"
}
2021/12/03 20:33:06 Context: Bucket flush, response code: 200 dump: 
2021/12/03 20:33:16 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/03 20:33:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 20:33:18 Context: Bucket flush, response code: 200 dump: 
2021/12/03 20:33:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCasDelete (130.90s)
=== RUN   TestCountersIncrement
2021/12/03 20:33:23 Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/03 20:33:29 Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/03 20:33:29 Removing node(s): [] from the cluster
2021/12/03 20:33:34 Rebalance progress: 100
2021/12/03 20:33:34 Deploying app: TestCountersIncrement. Item count src bucket: 0 dst bucket: 0
2021/12/03 20:33:35 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCountersIncrement' in metakv",
  "warnings": null
 }
}
2021/12/03 20:33:40 Waiting for app: TestCountersIncrement to get deployed
2021/12/03 20:33:45 Waiting for app: TestCountersIncrement to get deployed
2021/12/03 20:33:50 Waiting for app: TestCountersIncrement to get deployed
2021/12/03 20:33:55 Waiting for app: TestCountersIncrement to get deployed
2021/12/03 20:33:55 App: TestCountersIncrement got deployed
2021/12/03 20:33:55 Starting bucket ops to source bucket
2021/12/03 20:34:02 Waiting for dst bucket item count to get to: 1024 curr count: 0
2021/12/03 20:34:07 Waiting for dst bucket item count to get to: 1024 curr count: 512
2021/12/03 20:34:07 src & dst bucket item count matched up. src bucket count: 1024 dst bucket count: 1024
2021/12/03 20:34:07 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 20:34:07 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 20:34:12 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/03 20:34:17 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/03 20:34:22 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/03 20:34:27 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/03 20:34:27 App: TestCountersIncrement got un-deployed
2021/12/03 20:34:27 No eventing-con process running
2021/12/03 20:34:27 Function: TestCountersIncrement delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCountersIncrement deleting in the background"
}
2021/12/03 20:34:29 Context: Bucket flush, response code: 200 dump: 
2021/12/03 20:34:39 Waiting for dst bucket item count to get to: 0 curr count: 885
2021/12/03 20:34:44 Waiting for dst bucket item count to get to: 0 curr count: 372
2021/12/03 20:34:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 20:34:45 Context: Bucket flush, response code: 200 dump: 
2021/12/03 20:34:55 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/03 20:34:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 20:34:55 Deploying app: TestCountersIncrement. Item count src bucket: 0 dst bucket: 0
2021/12/03 20:34:57 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCountersIncrement' in metakv",
  "warnings": null
 }
}
2021/12/03 20:35:02 Waiting for app: TestCountersIncrement to get deployed
2021/12/03 20:35:07 Waiting for app: TestCountersIncrement to get deployed
2021/12/03 20:35:12 Waiting for app: TestCountersIncrement to get deployed
2021/12/03 20:35:17 Waiting for app: TestCountersIncrement to get deployed
2021/12/03 20:35:17 App: TestCountersIncrement got deployed
2021/12/03 20:35:17 Starting bucket ops to source bucket
2021/12/03 20:35:23 Waiting for dst bucket item count to get to: 2048 curr count: 0
2021/12/03 20:35:28 Waiting for dst bucket item count to get to: 2048 curr count: 1453
2021/12/03 20:35:33 Waiting for dst bucket item count to get to: 2048 curr count: 1453
2021/12/03 20:35:33 src & dst bucket item count matched up. src bucket count: 2048 dst bucket count: 2048
2021/12/03 20:35:33 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 20:35:33 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 20:35:38 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/03 20:35:43 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/03 20:35:48 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/03 20:35:53 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/03 20:35:53 App: TestCountersIncrement got un-deployed
2021/12/03 20:35:53 No eventing-con process running
2021/12/03 20:35:53 Function: TestCountersIncrement delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCountersIncrement deleting in the background"
}
2021/12/03 20:35:55 Context: Bucket flush, response code: 200 dump: 
2021/12/03 20:36:05 Waiting for dst bucket item count to get to: 0 curr count: 2048
2021/12/03 20:36:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 20:36:07 Context: Bucket flush, response code: 200 dump: 
2021/12/03 20:36:17 Waiting for dst bucket item count to get to: 0 curr count: 1
2021/12/03 20:36:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 20:36:17 Removing node(s): [http://127.0.0.1:19002 http://127.0.0.1:19003] from the cluster
2021/12/03 20:36:22 Rebalance progress: 100
--- PASS: TestCountersIncrement (178.45s)
=== RUN   TestCountersDecrement
2021/12/03 20:36:22 Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/03 20:36:22 Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/03 20:36:22 Removing node(s): [] from the cluster
2021/12/03 20:36:27 Rebalance progress: 100
2021/12/03 20:36:27 Deploying app: TestCountersDecrement. Item count src bucket: 0 dst bucket: 0
2021/12/03 20:36:29 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCountersDecrement' in metakv",
  "warnings": null
 }
}
2021/12/03 20:36:34 Waiting for app: TestCountersDecrement to get deployed
2021/12/03 20:36:39 Waiting for app: TestCountersDecrement to get deployed
2021/12/03 20:36:44 Waiting for app: TestCountersDecrement to get deployed
2021/12/03 20:36:49 Waiting for app: TestCountersDecrement to get deployed
2021/12/03 20:36:49 App: TestCountersDecrement got deployed
2021/12/03 20:36:49 Starting bucket ops to source bucket
2021/12/03 20:36:55 Waiting for dst bucket item count to get to: 1024 curr count: 0
2021/12/03 20:37:00 Waiting for dst bucket item count to get to: 1024 curr count: 278
2021/12/03 20:37:05 Waiting for dst bucket item count to get to: 1024 curr count: 278
2021/12/03 20:37:05 src & dst bucket item count matched up. src bucket count: 1024 dst bucket count: 1024
2021/12/03 20:37:05 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 20:37:05 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 20:37:10 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/03 20:37:15 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/03 20:37:20 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/03 20:37:25 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/03 20:37:25 App: TestCountersDecrement got un-deployed
2021/12/03 20:37:25 No eventing-con process running
2021/12/03 20:37:25 Function: TestCountersDecrement delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCountersDecrement deleting in the background"
}
2021/12/03 20:37:27 Context: Bucket flush, response code: 200 dump: 
2021/12/03 20:37:37 Waiting for dst bucket item count to get to: 0 curr count: 1025
2021/12/03 20:37:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 20:37:39 Context: Bucket flush, response code: 200 dump: 
2021/12/03 20:37:49 Waiting for dst bucket item count to get to: 0 curr count: 894
2021/12/03 20:37:54 Waiting for dst bucket item count to get to: 0 curr count: 382
2021/12/03 20:37:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 20:37:54 Deploying app: TestCountersDecrement. Item count src bucket: 0 dst bucket: 0
2021/12/03 20:37:55 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCountersDecrement' in metakv",
  "warnings": null
 }
}
2021/12/03 20:38:00 Waiting for app: TestCountersDecrement to get deployed
2021/12/03 20:38:05 Waiting for app: TestCountersDecrement to get deployed
2021/12/03 20:38:10 Waiting for app: TestCountersDecrement to get deployed
2021/12/03 20:38:15 Waiting for app: TestCountersDecrement to get deployed
2021/12/03 20:38:15 App: TestCountersDecrement got deployed
2021/12/03 20:38:15 Starting bucket ops to source bucket
2021/12/03 20:38:21 Waiting for dst bucket item count to get to: 2048 curr count: 0
2021/12/03 20:38:26 Waiting for dst bucket item count to get to: 2048 curr count: 1024
2021/12/03 20:38:26 src & dst bucket item count matched up. src bucket count: 2048 dst bucket count: 2048
2021/12/03 20:38:26 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 20:38:26 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 20:38:31 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/03 20:38:36 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/03 20:38:41 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/03 20:38:46 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/03 20:38:46 App: TestCountersDecrement got un-deployed
2021/12/03 20:38:46 No eventing-con process running
2021/12/03 20:38:46 Function: TestCountersDecrement delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCountersDecrement deleting in the background"
}
2021/12/03 20:38:48 Context: Bucket flush, response code: 200 dump: 
2021/12/03 20:38:58 Waiting for dst bucket item count to get to: 0 curr count: 1361
2021/12/03 20:39:03 Waiting for dst bucket item count to get to: 0 curr count: 337
2021/12/03 20:39:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 20:39:05 Context: Bucket flush, response code: 200 dump: 
2021/12/03 20:39:15 Waiting for dst bucket item count to get to: 0 curr count: 1
2021/12/03 20:39:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 20:39:15 Removing node(s): [http://127.0.0.1:19002 http://127.0.0.1:19003] from the cluster
2021/12/03 20:39:20 Rebalance progress: 100
--- PASS: TestCountersDecrement (178.33s)
=== RUN   TestCacheGet
2021/12/03 20:39:20 Deploying app: TestCacheGet. Item count src bucket: 0 dst bucket: 0
2021/12/03 20:39:21 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCacheGet' in metakv",
  "warnings": null
 }
}
2021/12/03 20:39:26 Waiting for app: TestCacheGet to get deployed
2021/12/03 20:39:31 Waiting for app: TestCacheGet to get deployed
2021/12/03 20:39:36 Waiting for app: TestCacheGet to get deployed
2021/12/03 20:39:41 Waiting for app: TestCacheGet to get deployed
2021/12/03 20:39:41 App: TestCacheGet got deployed
2021/12/03 20:39:41 Starting bucket ops to source bucket
2021/12/03 20:39:46 Waiting for dst bucket item count to get to: 20 curr count: 0
2021/12/03 20:39:51 Waiting for dst bucket item count to get to: 20 curr count: 0
2021/12/03 20:39:56 Waiting for dst bucket item count to get to: 20 curr count: 5
2021/12/03 20:40:01 Waiting for dst bucket item count to get to: 20 curr count: 5
2021/12/03 20:40:06 Waiting for dst bucket item count to get to: 20 curr count: 18
2021/12/03 20:40:11 Waiting for dst bucket item count to get to: 20 curr count: 18
2021/12/03 20:40:11 src & dst bucket item count matched up. src bucket count: 20 dst bucket count: 20
2021/12/03 20:40:11 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 20:40:12 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 20:40:17 Waiting for app: TestCacheGet to get un-deployed
2021/12/03 20:40:22 Waiting for app: TestCacheGet to get un-deployed
2021/12/03 20:40:27 Waiting for app: TestCacheGet to get un-deployed
2021/12/03 20:40:32 Waiting for app: TestCacheGet to get un-deployed
2021/12/03 20:40:32 App: TestCacheGet got un-deployed
2021/12/03 20:40:32 No eventing-con process running
2021/12/03 20:40:32 Function: TestCacheGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCacheGet deleting in the background"
}
2021/12/03 20:40:33 Context: Bucket flush, response code: 200 dump: 
2021/12/03 20:40:43 Waiting for dst bucket item count to get to: 0 curr count: 20
2021/12/03 20:40:48 Waiting for dst bucket item count to get to: 0 curr count: 10
2021/12/03 20:40:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 20:40:50 Context: Bucket flush, response code: 200 dump: 
2021/12/03 20:41:00 Waiting for dst bucket item count to get to: 0 curr count: 16
2021/12/03 20:41:05 Waiting for dst bucket item count to get to: 0 curr count: 16
2021/12/03 20:41:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCacheGet (105.34s)
=== RUN   TestCacheRYOW
2021/12/03 20:41:05 Deploying app: TestCacheRYOW. Item count src bucket: 0 dst bucket: 0
2021/12/03 20:41:07 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCacheRYOW' in metakv",
  "warnings": null
 }
}
2021/12/03 20:41:12 Waiting for app: TestCacheRYOW to get deployed
2021/12/03 20:41:17 Waiting for app: TestCacheRYOW to get deployed
2021/12/03 20:41:22 Waiting for app: TestCacheRYOW to get deployed
2021/12/03 20:41:27 Waiting for app: TestCacheRYOW to get deployed
2021/12/03 20:41:27 App: TestCacheRYOW got deployed
2021/12/03 20:41:27 Starting bucket ops to source bucket
2021/12/03 20:41:32 Waiting for dst bucket item count to get to: 50 curr count: 0
2021/12/03 20:41:32 src & dst bucket item count matched up. src bucket count: 50 dst bucket count: 50
2021/12/03 20:41:32 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 20:41:32 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 20:41:37 Waiting for app: TestCacheRYOW to get un-deployed
2021/12/03 20:41:42 Waiting for app: TestCacheRYOW to get un-deployed
2021/12/03 20:41:47 Waiting for app: TestCacheRYOW to get un-deployed
2021/12/03 20:41:52 Waiting for app: TestCacheRYOW to get un-deployed
2021/12/03 20:41:52 App: TestCacheRYOW got un-deployed
2021/12/03 20:41:52 No eventing-con process running
2021/12/03 20:41:52 Function: TestCacheRYOW delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCacheRYOW deleting in the background"
}
2021/12/03 20:41:54 Context: Bucket flush, response code: 200 dump: 
2021/12/03 20:42:04 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/03 20:42:09 Waiting for dst bucket item count to get to: 0 curr count: 25
2021/12/03 20:42:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 20:42:11 Context: Bucket flush, response code: 200 dump: 
2021/12/03 20:42:21 Waiting for dst bucket item count to get to: 0 curr count: 41
2021/12/03 20:42:26 Waiting for dst bucket item count to get to: 0 curr count: 41
2021/12/03 20:42:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCacheRYOW (80.29s)
=== RUN   TestCacheExpiry
2021/12/03 20:42:26 Deploying app: TestCacheExpiry. Item count src bucket: 0 dst bucket: 0
2021/12/03 20:42:27 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCacheExpiry' in metakv",
  "warnings": null
 }
}
2021/12/03 20:42:32 Waiting for app: TestCacheExpiry to get deployed
2021/12/03 20:42:37 Waiting for app: TestCacheExpiry to get deployed
2021/12/03 20:42:42 Waiting for app: TestCacheExpiry to get deployed
2021/12/03 20:42:47 Waiting for app: TestCacheExpiry to get deployed
2021/12/03 20:42:47 App: TestCacheExpiry got deployed
2021/12/03 20:42:47 Starting bucket ops to source bucket
2021/12/03 20:42:52 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/03 20:42:57 Waiting for dst bucket item count to get to: 100 curr count: 19
2021/12/03 20:43:02 Waiting for dst bucket item count to get to: 100 curr count: 19
2021/12/03 20:43:07 Waiting for dst bucket item count to get to: 100 curr count: 94
2021/12/03 20:43:12 Waiting for dst bucket item count to get to: 100 curr count: 94
2021/12/03 20:43:12 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/03 20:43:12 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 20:43:12 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 20:43:17 Waiting for app: TestCacheExpiry to get un-deployed
2021/12/03 20:43:22 Waiting for app: TestCacheExpiry to get un-deployed
2021/12/03 20:43:27 Waiting for app: TestCacheExpiry to get un-deployed
2021/12/03 20:43:32 Waiting for app: TestCacheExpiry to get un-deployed
2021/12/03 20:43:32 App: TestCacheExpiry got un-deployed
2021/12/03 20:43:32 No eventing-con process running
2021/12/03 20:43:32 Function: TestCacheExpiry delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCacheExpiry deleting in the background"
}
2021/12/03 20:43:34 Context: Bucket flush, response code: 200 dump: 
2021/12/03 20:43:44 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/03 20:43:49 Waiting for dst bucket item count to get to: 0 curr count: 25
2021/12/03 20:43:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 20:43:51 Context: Bucket flush, response code: 200 dump: 
2021/12/03 20:44:01 Waiting for dst bucket item count to get to: 0 curr count: 93
2021/12/03 20:44:06 Waiting for dst bucket item count to get to: 0 curr count: 93
2021/12/03 20:44:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCacheExpiry (100.31s)
=== RUN   TestCacheExpiryCustom
2021/12/03 20:44:06 Deploying app: TestCacheExpiryCustom. Item count src bucket: 0 dst bucket: 0
2021/12/03 20:44:07 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCacheExpiryCustom' in metakv",
  "warnings": null
 }
}
2021/12/03 20:44:12 Waiting for app: TestCacheExpiryCustom to get deployed
2021/12/03 20:44:17 Waiting for app: TestCacheExpiryCustom to get deployed
2021/12/03 20:44:22 Waiting for app: TestCacheExpiryCustom to get deployed
2021/12/03 20:44:27 Waiting for app: TestCacheExpiryCustom to get deployed
2021/12/03 20:44:27 App: TestCacheExpiryCustom got deployed
2021/12/03 20:44:27 Starting bucket ops to source bucket
2021/12/03 20:44:32 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/03 20:44:37 Waiting for dst bucket item count to get to: 100 curr count: 11
2021/12/03 20:44:42 Waiting for dst bucket item count to get to: 100 curr count: 11
2021/12/03 20:44:47 Waiting for dst bucket item count to get to: 100 curr count: 86
2021/12/03 20:44:52 Waiting for dst bucket item count to get to: 100 curr count: 86
2021/12/03 20:44:52 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/03 20:44:52 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 20:44:53 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 20:44:58 Waiting for app: TestCacheExpiryCustom to get un-deployed
2021/12/03 20:45:03 Waiting for app: TestCacheExpiryCustom to get un-deployed
2021/12/03 20:45:08 Waiting for app: TestCacheExpiryCustom to get un-deployed
2021/12/03 20:45:13 Waiting for app: TestCacheExpiryCustom to get un-deployed
2021/12/03 20:45:13 App: TestCacheExpiryCustom got un-deployed
2021/12/03 20:45:13 No eventing-con process running
2021/12/03 20:45:13 Function: TestCacheExpiryCustom delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCacheExpiryCustom deleting in the background"
}
2021/12/03 20:45:14 Context: Bucket flush, response code: 200 dump: 
2021/12/03 20:45:24 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/03 20:45:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 20:45:26 Context: Bucket flush, response code: 200 dump: 
2021/12/03 20:45:36 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/03 20:45:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCacheExpiryCustom (90.35s)
=== RUN   TestCacheOverflowCustom
2021/12/03 20:45:36 Deploying app: TestCacheOverflowCustom. Item count src bucket: 0 dst bucket: 0
2021/12/03 20:45:37 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCacheOverflowCustom' in metakv",
  "warnings": null
 }
}
2021/12/03 20:45:42 Waiting for app: TestCacheOverflowCustom to get deployed
2021/12/03 20:45:48 Waiting for app: TestCacheOverflowCustom to get deployed
2021/12/03 20:45:53 Waiting for app: TestCacheOverflowCustom to get deployed
2021/12/03 20:45:58 Waiting for app: TestCacheOverflowCustom to get deployed
2021/12/03 20:45:58 App: TestCacheOverflowCustom got deployed
2021/12/03 20:45:58 Starting bucket ops to source bucket
2021/12/03 20:46:03 Waiting for dst bucket item count to get to: 35 curr count: 0
2021/12/03 20:46:08 Waiting for dst bucket item count to get to: 35 curr count: 5
2021/12/03 20:46:13 Waiting for dst bucket item count to get to: 35 curr count: 5
2021/12/03 20:46:18 Waiting for dst bucket item count to get to: 35 curr count: 25
2021/12/03 20:46:23 Waiting for dst bucket item count to get to: 35 curr count: 25
2021/12/03 20:46:28 Waiting for dst bucket item count to get to: 35 curr count: 34
2021/12/03 20:46:33 Waiting for dst bucket item count to get to: 35 curr count: 34
2021/12/03 20:46:33 src & dst bucket item count matched up. src bucket count: 35 dst bucket count: 35
2021/12/03 20:46:33 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 20:46:33 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 20:46:38 Waiting for app: TestCacheOverflowCustom to get un-deployed
2021/12/03 20:46:43 Waiting for app: TestCacheOverflowCustom to get un-deployed
2021/12/03 20:46:48 Waiting for app: TestCacheOverflowCustom to get un-deployed
2021/12/03 20:46:53 Waiting for app: TestCacheOverflowCustom to get un-deployed
2021/12/03 20:46:53 App: TestCacheOverflowCustom got un-deployed
2021/12/03 20:46:53 No eventing-con process running
2021/12/03 20:46:53 Function: TestCacheOverflowCustom delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCacheOverflowCustom deleting in the background"
}
2021/12/03 20:46:55 Context: Bucket flush, response code: 200 dump: 
2021/12/03 20:47:05 Waiting for dst bucket item count to get to: 0 curr count: 5
2021/12/03 20:47:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 20:47:07 Context: Bucket flush, response code: 200 dump: 
2021/12/03 20:47:17 Waiting for dst bucket item count to get to: 0 curr count: 35
2021/12/03 20:47:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCacheOverflowCustom (100.51s)
=== RUN   TestStrictMode
2021/12/03 20:47:17 Deploying app: TestStrictMode. Item count src bucket: 0 dst bucket: 0
2021/12/03 20:47:18 Context: Post to main store, response code: 422 dump: {
 "name": "ERR_HANDLER_COMPILATION",
 "code": 27,
 "description": "Function compilation failed",
 "attributes": null,
 "runtime_info": {
  "code": 27,
  "info": {
   "area": "handlerCode",
   "column_number": 63,
   "compile_success": false,
   "description": "        destination[meta.id] = 00000000000000000000000000000101;\n                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nSyntaxError: Octal literals are not allowed in strict mode.\n",
   "index": 132,
   "language": "JavaScript",
   "line_number": 4
  }
 }
}
--- PASS: TestStrictMode (1.05s)
=== RUN   TestDataTypes
2021/12/03 20:47:18 Deploying app: TestDataTypes. Item count src bucket: 0 dst bucket: 0
2021/12/03 20:47:19 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDataTypes' in metakv",
  "warnings": null
 }
}
2021/12/03 20:47:24 Waiting for app: TestDataTypes to get deployed
2021/12/03 20:47:29 Waiting for app: TestDataTypes to get deployed
2021/12/03 20:47:34 Waiting for app: TestDataTypes to get deployed
2021/12/03 20:47:39 Waiting for app: TestDataTypes to get deployed
2021/12/03 20:47:39 App: TestDataTypes got deployed
2021/12/03 20:47:39 Starting bucket ops to source bucket
2021/12/03 20:47:44 Waiting for dst bucket item count to get to: 18 curr count: 0
2021/12/03 20:47:49 Waiting for dst bucket item count to get to: 18 curr count: 10
2021/12/03 20:47:54 Waiting for dst bucket item count to get to: 18 curr count: 10
2021/12/03 20:47:54 src & dst bucket item count matched up. src bucket count: 18 dst bucket count: 18
2021/12/03 20:47:54 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 20:47:54 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 20:47:59 Waiting for app: TestDataTypes to get un-deployed
2021/12/03 20:48:04 Waiting for app: TestDataTypes to get un-deployed
2021/12/03 20:48:09 Waiting for app: TestDataTypes to get un-deployed
2021/12/03 20:48:14 Waiting for app: TestDataTypes to get un-deployed
2021/12/03 20:48:14 App: TestDataTypes got un-deployed
2021/12/03 20:48:14 No eventing-con process running
2021/12/03 20:48:14 Function: TestDataTypes delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDataTypes deleting in the background"
}
2021/12/03 20:48:16 Context: Bucket flush, response code: 200 dump: 
2021/12/03 20:48:26 Waiting for dst bucket item count to get to: 0 curr count: 1
2021/12/03 20:48:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 20:48:28 Context: Bucket flush, response code: 200 dump: 
2021/12/03 20:48:38 Waiting for dst bucket item count to get to: 0 curr count: 11
2021/12/03 20:48:43 Waiting for dst bucket item count to get to: 0 curr count: 1
2021/12/03 20:48:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDataTypes (85.11s)
=== RUN   TestEnoentGet
2021/12/03 20:48:43 Deploying app: TestEnoentGet. Item count src bucket: 0 dst bucket: 0
2021/12/03 20:48:44 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentGet' in metakv",
  "warnings": null
 }
}
2021/12/03 20:48:49 Waiting for app: TestEnoentGet to get deployed
2021/12/03 20:48:54 Waiting for app: TestEnoentGet to get deployed
2021/12/03 20:48:59 Waiting for app: TestEnoentGet to get deployed
2021/12/03 20:49:04 Waiting for app: TestEnoentGet to get deployed
2021/12/03 20:49:04 App: TestEnoentGet got deployed
2021/12/03 20:49:04 Starting bucket ops to source bucket
2021/12/03 20:49:09 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/03 20:49:14 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/03 20:49:14 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/03 20:49:14 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 20:49:14 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 20:49:19 Waiting for app: TestEnoentGet to get un-deployed
2021/12/03 20:49:24 Waiting for app: TestEnoentGet to get un-deployed
2021/12/03 20:49:29 Waiting for app: TestEnoentGet to get un-deployed
2021/12/03 20:49:34 Waiting for app: TestEnoentGet to get un-deployed
2021/12/03 20:49:34 App: TestEnoentGet got un-deployed
2021/12/03 20:49:34 No eventing-con process running
2021/12/03 20:49:35 Function: TestEnoentGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentGet deleting in the background"
}
2021/12/03 20:49:36 Context: Bucket flush, response code: 200 dump: 
2021/12/03 20:49:46 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/03 20:49:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 20:49:48 Context: Bucket flush, response code: 200 dump: 
2021/12/03 20:49:58 Waiting for dst bucket item count to get to: 0 curr count: 75
2021/12/03 20:50:03 Waiting for dst bucket item count to get to: 0 curr count: 25
2021/12/03 20:50:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 20:50:03 Deploying app: TestEnoentGet. Item count src bucket: 0 dst bucket: 0
2021/12/03 20:50:04 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentGet' in metakv",
  "warnings": null
 }
}
2021/12/03 20:50:09 Waiting for app: TestEnoentGet to get deployed
2021/12/03 20:50:14 Waiting for app: TestEnoentGet to get deployed
2021/12/03 20:50:19 Waiting for app: TestEnoentGet to get deployed
2021/12/03 20:50:24 Waiting for app: TestEnoentGet to get deployed
2021/12/03 20:50:24 App: TestEnoentGet got deployed
2021/12/03 20:50:24 Starting bucket ops to source bucket
2021/12/03 20:50:29 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/03 20:50:34 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/03 20:50:34 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/03 20:50:35 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 20:50:35 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 20:50:40 Waiting for app: TestEnoentGet to get un-deployed
2021/12/03 20:50:45 Waiting for app: TestEnoentGet to get un-deployed
2021/12/03 20:50:50 Waiting for app: TestEnoentGet to get un-deployed
2021/12/03 20:50:55 Waiting for app: TestEnoentGet to get un-deployed
2021/12/03 20:50:55 App: TestEnoentGet got un-deployed
2021/12/03 20:50:55 No eventing-con process running
2021/12/03 20:50:55 Function: TestEnoentGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentGet deleting in the background"
}
2021/12/03 20:50:56 Context: Bucket flush, response code: 200 dump: 
2021/12/03 20:51:06 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/03 20:51:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 20:51:08 Context: Bucket flush, response code: 200 dump: 
2021/12/03 20:51:18 Waiting for dst bucket item count to get to: 0 curr count: 79
2021/12/03 20:51:23 Waiting for dst bucket item count to get to: 0 curr count: 29
2021/12/03 20:51:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEnoentGet (160.41s)
=== RUN   TestEnoentDelete
2021/12/03 20:51:23 Deploying app: TestEnoentDelete. Item count src bucket: 0 dst bucket: 0
2021/12/03 20:51:24 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentDelete' in metakv",
  "warnings": null
 }
}
2021/12/03 20:51:29 Waiting for app: TestEnoentDelete to get deployed
2021/12/03 20:51:34 Waiting for app: TestEnoentDelete to get deployed
2021/12/03 20:51:39 Waiting for app: TestEnoentDelete to get deployed
2021/12/03 20:51:44 Waiting for app: TestEnoentDelete to get deployed
2021/12/03 20:51:44 App: TestEnoentDelete got deployed
2021/12/03 20:51:44 Starting bucket ops to source bucket
2021/12/03 20:51:50 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/03 20:51:55 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/03 20:51:55 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/03 20:51:55 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 20:51:55 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 20:52:00 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/03 20:52:05 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/03 20:52:10 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/03 20:52:15 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/03 20:52:15 App: TestEnoentDelete got un-deployed
2021/12/03 20:52:15 No eventing-con process running
2021/12/03 20:52:15 Function: TestEnoentDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentDelete deleting in the background"
}
2021/12/03 20:52:17 Context: Bucket flush, response code: 200 dump: 
2021/12/03 20:52:27 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/03 20:52:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 20:52:28 Context: Bucket flush, response code: 200 dump: 
2021/12/03 20:52:38 Waiting for dst bucket item count to get to: 0 curr count: 82
2021/12/03 20:52:43 Waiting for dst bucket item count to get to: 0 curr count: 32
2021/12/03 20:52:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 20:52:43 Deploying app: TestEnoentDelete. Item count src bucket: 0 dst bucket: 0
2021/12/03 20:52:45 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentDelete' in metakv",
  "warnings": null
 }
}
2021/12/03 20:52:50 Waiting for app: TestEnoentDelete to get deployed
2021/12/03 20:52:55 Waiting for app: TestEnoentDelete to get deployed
2021/12/03 20:53:00 Waiting for app: TestEnoentDelete to get deployed
2021/12/03 20:53:05 Waiting for app: TestEnoentDelete to get deployed
2021/12/03 20:53:05 App: TestEnoentDelete got deployed
2021/12/03 20:53:05 Starting bucket ops to source bucket
2021/12/03 20:53:10 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/03 20:53:15 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/03 20:53:15 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/03 20:53:15 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 20:53:15 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 20:53:20 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/03 20:53:25 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/03 20:53:30 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/03 20:53:35 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/03 20:53:35 App: TestEnoentDelete got un-deployed
2021/12/03 20:53:35 No eventing-con process running
2021/12/03 20:53:35 Function: TestEnoentDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentDelete deleting in the background"
}
2021/12/03 20:53:37 Context: Bucket flush, response code: 200 dump: 
2021/12/03 20:53:47 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/03 20:53:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 20:53:49 Context: Bucket flush, response code: 200 dump: 
2021/12/03 20:53:59 Waiting for dst bucket item count to get to: 0 curr count: 84
2021/12/03 20:54:04 Waiting for dst bucket item count to get to: 0 curr count: 34
2021/12/03 20:54:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEnoentDelete (160.27s)
=== RUN   TestError
2021/12/03 20:54:14 Waiting for app: TestError to get un-deployed
2021/12/03 20:54:14 App: TestError got un-deployed
2021/12/03 20:54:14 No eventing-con process running
2021/12/03 20:54:14 Function: TestError delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestError not found"
 }
}
2021/12/03 20:54:15 Context: Bucket flush, response code: 200 dump: 
2021/12/03 20:54:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 20:54:22 Context: Bucket flush, response code: 200 dump: 
2021/12/03 20:54:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 20:54:27 Deploying app: TestError. Item count src bucket: 0 dst bucket: 0
2021/12/03 20:54:28 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestError' in metakv",
  "warnings": null
 }
}
2021/12/03 20:54:33 Waiting for app: TestError to get deployed
2021/12/03 20:54:38 Waiting for app: TestError to get deployed
2021/12/03 20:54:43 Waiting for app: TestError to get deployed
2021/12/03 20:54:48 Waiting for app: TestError to get deployed
2021/12/03 20:54:48 App: TestError got deployed
2021/12/03 20:54:48 Starting bucket ops to source bucket
2021/12/03 20:54:54 Waiting for dst bucket item count to get to: 300 curr count: 0
2021/12/03 20:54:59 Waiting for dst bucket item count to get to: 300 curr count: 144
2021/12/03 20:54:59 src & dst bucket item count matched up. src bucket count: 300 dst bucket count: 300
2021/12/03 20:54:59 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 20:54:59 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 20:55:04 Waiting for app: TestError to get un-deployed
2021/12/03 20:55:09 Waiting for app: TestError to get un-deployed
2021/12/03 20:55:14 Waiting for app: TestError to get un-deployed
2021/12/03 20:55:19 Waiting for app: TestError to get un-deployed
2021/12/03 20:55:19 App: TestError got un-deployed
2021/12/03 20:55:19 No eventing-con process running
2021/12/03 20:55:19 Function: TestError delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestError deleting in the background"
}
2021/12/03 20:55:21 Context: Bucket flush, response code: 200 dump: 
2021/12/03 20:55:31 Waiting for dst bucket item count to get to: 0 curr count: 89
2021/12/03 20:55:36 Waiting for dst bucket item count to get to: 0 curr count: 89
2021/12/03 20:55:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 20:55:38 Context: Bucket flush, response code: 200 dump: 
2021/12/03 20:55:48 Waiting for dst bucket item count to get to: 0 curr count: 144
2021/12/03 20:55:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestError (103.97s)
=== RUN   TestCRLF
2021/12/03 20:55:58 Waiting for app: TestCRLF to get un-deployed
2021/12/03 20:55:58 App: TestCRLF got un-deployed
2021/12/03 20:55:58 No eventing-con process running
2021/12/03 20:55:58 Function: TestCRLF delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestCRLF not found"
 }
}
2021/12/03 20:55:59 Context: Bucket flush, response code: 200 dump: 
2021/12/03 20:56:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 20:56:06 Context: Bucket flush, response code: 200 dump: 
2021/12/03 20:56:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 20:56:11 Deploying app: TestCRLF. Item count src bucket: 0 dst bucket: 0
2021/12/03 20:56:12 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCRLF' in metakv",
  "warnings": null
 }
}
2021/12/03 20:56:17 Waiting for app: TestCRLF to get deployed
2021/12/03 20:56:22 Waiting for app: TestCRLF to get deployed
2021/12/03 20:56:27 Waiting for app: TestCRLF to get deployed
2021/12/03 20:56:32 Waiting for app: TestCRLF to get deployed
2021/12/03 20:56:32 App: TestCRLF got deployed
2021/12/03 20:56:32 Starting bucket ops to source bucket
2021/12/03 20:56:38 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/03 20:56:43 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/03 20:56:43 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/03 20:56:43 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 20:56:43 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 20:56:48 Waiting for app: TestCRLF to get un-deployed
2021/12/03 20:56:53 Waiting for app: TestCRLF to get un-deployed
2021/12/03 20:56:58 Waiting for app: TestCRLF to get un-deployed
2021/12/03 20:57:03 Waiting for app: TestCRLF to get un-deployed
2021/12/03 20:57:03 App: TestCRLF got un-deployed
2021/12/03 20:57:03 No eventing-con process running
2021/12/03 20:57:03 Function: TestCRLF delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCRLF deleting in the background"
}
2021/12/03 20:57:05 Context: Bucket flush, response code: 200 dump: 
2021/12/03 20:57:15 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/03 20:57:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 20:57:17 Context: Bucket flush, response code: 200 dump: 
2021/12/03 20:57:27 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/03 20:57:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCRLF (99.00s)
=== RUN   TestImportExport
2021/12/03 20:57:37 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/03 20:57:37 App: TestImportExport_n1ql_insert_on_update got un-deployed
2021/12/03 20:57:37 No eventing-con process running
2021/12/03 20:57:37 Function: TestImportExport_n1ql_insert_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestImportExport_n1ql_insert_on_update not found"
 }
}
2021/12/03 20:57:38 Context: Bucket flush, response code: 200 dump: 
2021/12/03 20:57:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 20:57:45 Context: Bucket flush, response code: 200 dump: 
2021/12/03 20:57:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 20:57:55 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/03 20:57:55 App: TestImportExport_bucket_op_on_update got un-deployed
2021/12/03 20:57:55 No eventing-con process running
2021/12/03 20:57:55 Function: TestImportExport_bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestImportExport_bucket_op_on_update not found"
 }
}
2021/12/03 20:57:57 Context: Bucket flush, response code: 200 dump: 
2021/12/03 20:58:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 20:58:04 Context: Bucket flush, response code: 200 dump: 
2021/12/03 20:58:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 20:58:09 Deploying app: TestImportExport_n1ql_insert_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/03 20:58:10 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestImportExport_n1ql_insert_on_update' in metakv",
  "warnings": null
 }
}
2021/12/03 20:58:10 Deploying app: TestImportExport_bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/03 20:58:11 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestImportExport_bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/03 20:58:16 Waiting for app: TestImportExport_n1ql_insert_on_update to get deployed
2021/12/03 20:58:21 Waiting for app: TestImportExport_n1ql_insert_on_update to get deployed
2021/12/03 20:58:26 Waiting for app: TestImportExport_n1ql_insert_on_update to get deployed
2021/12/03 20:58:31 Waiting for app: TestImportExport_n1ql_insert_on_update to get deployed
2021/12/03 20:58:31 App: TestImportExport_n1ql_insert_on_update got deployed
2021/12/03 20:58:36 Waiting for app: TestImportExport_bucket_op_on_update to get deployed
2021/12/03 20:58:41 Waiting for app: TestImportExport_bucket_op_on_update to get deployed
2021/12/03 20:58:46 Waiting for app: TestImportExport_bucket_op_on_update to get deployed
2021/12/03 20:58:46 App: TestImportExport_bucket_op_on_update got deployed
2021/12/03 20:58:47 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 20:58:52 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/03 20:58:57 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/03 20:59:02 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/03 20:59:07 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/03 20:59:07 App: TestImportExport_n1ql_insert_on_update got un-deployed
2021/12/03 20:59:07 No eventing-con process running
2021/12/03 20:59:07 Function: TestImportExport_n1ql_insert_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestImportExport_n1ql_insert_on_update deleting in the background"
}
2021/12/03 20:59:07 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 20:59:12 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/03 20:59:17 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/03 20:59:22 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/03 20:59:27 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/03 20:59:27 App: TestImportExport_bucket_op_on_update got un-deployed
2021/12/03 20:59:27 No eventing-con process running
2021/12/03 20:59:27 Function: TestImportExport_bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestImportExport_bucket_op_on_update deleting in the background"
}
2021/12/03 20:59:44 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/03 20:59:44 App: TestImportExport_bucket_op_on_update got un-deployed
2021/12/03 20:59:44 No eventing-con process running
2021/12/03 20:59:44 Function: TestImportExport_bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestImportExport_bucket_op_on_update deleting in the background"
}
2021/12/03 20:59:46 Context: Bucket flush, response code: 200 dump: 
2021/12/03 20:59:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 20:59:53 Context: Bucket flush, response code: 200 dump: 
2021/12/03 20:59:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 21:00:03 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/03 21:00:03 App: TestImportExport_n1ql_insert_on_update got un-deployed
2021/12/03 21:00:03 No eventing-con process running
2021/12/03 21:00:03 Function: TestImportExport_n1ql_insert_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestImportExport_n1ql_insert_on_update deleting in the background"
}
2021/12/03 21:00:05 Context: Bucket flush, response code: 200 dump: 
2021/12/03 21:00:10 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 21:00:12 Context: Bucket flush, response code: 200 dump: 
2021/12/03 21:00:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestImportExport (170.01s)
=== RUN   TestDeployUndeployLoopNonDefaultSettings
2021/12/03 21:00:27 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/03 21:00:27 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/03 21:00:27 No eventing-con process running
2021/12/03 21:00:27 Function: TestDeployUndeployLoopNonDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDeployUndeployLoopNonDefaultSettings not found"
 }
}
2021/12/03 21:00:28 Context: Bucket flush, response code: 200 dump: 
2021/12/03 21:00:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 21:00:35 Context: Bucket flush, response code: 200 dump: 
2021/12/03 21:00:40 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 21:00:40 Deploying app: TestDeployUndeployLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/03 21:00:41 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/03 21:00:41 Starting bucket ops to source bucket
2021/12/03 21:00:49 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 21:00:54 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 21:00:59 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 21:01:04 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 21:01:04 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/03 21:01:09 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get deployed
2021/12/03 21:01:09 App: TestDeployUndeployLoopNonDefaultSettings got deployed
2021/12/03 21:01:09 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 21:01:09 Undeploying app: bucket_op_on_update
2021/12/03 21:01:10 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 21:01:15 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/03 21:01:20 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/03 21:01:25 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/03 21:01:30 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/03 21:01:30 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/03 21:01:30 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 21:01:30 No eventing-con process running
2021/12/03 21:01:31 Context: Bucket flush, response code: 200 dump: 
2021/12/03 21:01:39 Context: Bucket flush, response code: 200 dump: 
2021/12/03 21:02:14 Deploying app: TestDeployUndeployLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/03 21:02:15 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/03 21:02:15 Starting bucket ops to source bucket
2021/12/03 21:02:23 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 21:02:28 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 21:02:33 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 21:02:38 Waiting for dst bucket item count to get to: 5000 curr count: 3273
2021/12/03 21:02:43 Waiting for dst bucket item count to get to: 5000 curr count: 3273
2021/12/03 21:02:43 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/03 21:02:48 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get deployed
2021/12/03 21:02:48 App: TestDeployUndeployLoopNonDefaultSettings got deployed
2021/12/03 21:02:48 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 21:02:48 Undeploying app: bucket_op_on_update
2021/12/03 21:02:48 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 21:02:53 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/03 21:02:58 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/03 21:03:03 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/03 21:03:08 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/03 21:03:08 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/03 21:03:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 21:03:08 No eventing-con process running
2021/12/03 21:03:10 Context: Bucket flush, response code: 200 dump: 
2021/12/03 21:03:17 Context: Bucket flush, response code: 200 dump: 
2021/12/03 21:03:52 Deploying app: TestDeployUndeployLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/03 21:03:53 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/03 21:03:53 Starting bucket ops to source bucket
2021/12/03 21:04:01 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 21:04:06 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 21:04:11 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 21:04:16 Waiting for dst bucket item count to get to: 5000 curr count: 2423
2021/12/03 21:04:21 Waiting for dst bucket item count to get to: 5000 curr count: 4923
2021/12/03 21:04:21 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/03 21:04:26 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get deployed
2021/12/03 21:04:26 App: TestDeployUndeployLoopNonDefaultSettings got deployed
2021/12/03 21:04:26 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 21:04:26 Undeploying app: bucket_op_on_update
2021/12/03 21:04:27 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 21:04:32 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/03 21:04:37 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/03 21:04:42 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/03 21:04:47 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/03 21:04:47 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/03 21:04:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 21:04:47 No eventing-con process running
2021/12/03 21:04:48 Context: Bucket flush, response code: 200 dump: 
2021/12/03 21:04:56 Context: Bucket flush, response code: 200 dump: 
2021/12/03 21:05:31 Deploying app: TestDeployUndeployLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/03 21:05:32 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/03 21:05:32 Starting bucket ops to source bucket
2021/12/03 21:05:40 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 21:05:45 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 21:05:50 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 21:05:55 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 21:05:55 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/03 21:06:00 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get deployed
2021/12/03 21:06:00 App: TestDeployUndeployLoopNonDefaultSettings got deployed
2021/12/03 21:06:00 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 21:06:00 Undeploying app: bucket_op_on_update
2021/12/03 21:06:00 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 21:06:05 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/03 21:06:10 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/03 21:06:15 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/03 21:06:20 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/03 21:06:20 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/03 21:06:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 21:06:20 No eventing-con process running
2021/12/03 21:06:22 Context: Bucket flush, response code: 200 dump: 
2021/12/03 21:06:29 Context: Bucket flush, response code: 200 dump: 
2021/12/03 21:07:04 Deploying app: TestDeployUndeployLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/03 21:07:05 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/03 21:07:05 Starting bucket ops to source bucket
2021/12/03 21:07:13 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 21:07:18 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 21:07:23 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 21:07:28 Waiting for dst bucket item count to get to: 5000 curr count: 2707
2021/12/03 21:07:33 Waiting for dst bucket item count to get to: 5000 curr count: 2707
2021/12/03 21:07:33 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/03 21:07:38 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get deployed
2021/12/03 21:07:38 App: TestDeployUndeployLoopNonDefaultSettings got deployed
2021/12/03 21:07:38 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 21:07:38 Undeploying app: bucket_op_on_update
2021/12/03 21:07:39 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 21:07:44 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/03 21:07:49 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/03 21:07:54 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/03 21:07:59 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/03 21:07:59 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/03 21:07:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 21:07:59 No eventing-con process running
2021/12/03 21:08:00 Context: Bucket flush, response code: 200 dump: 
2021/12/03 21:08:08 Context: Bucket flush, response code: 200 dump: 
2021/12/03 21:08:43 Function: TestDeployUndeployLoopNonDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDeployUndeployLoopNonDefaultSettings deleting in the background"
}
--- PASS: TestDeployUndeployLoopNonDefaultSettings (506.21s)
=== RUN   TestOnUpdateN1QLOp
2021/12/03 21:08:53 Waiting for app: TestOnUpdateN1QLOp to get un-deployed
2021/12/03 21:08:53 App: TestOnUpdateN1QLOp got un-deployed
2021/12/03 21:08:53 No eventing-con process running
2021/12/03 21:08:53 Function: TestOnUpdateN1QLOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateN1QLOp not found"
 }
}
2021/12/03 21:08:55 Context: Bucket flush, response code: 200 dump: 
2021/12/03 21:09:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 21:09:01 Context: Bucket flush, response code: 200 dump: 
2021/12/03 21:09:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 21:09:06 Deploying app: TestOnUpdateN1QLOp. Item count src bucket: 0 dst bucket: 0
2021/12/03 21:09:08 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateN1QLOp' in metakv",
  "warnings": null
 }
}
2021/12/03 21:09:08 Starting bucket ops to source bucket
2021/12/03 21:09:16 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 21:09:21 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 21:09:26 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 21:09:31 Waiting for dst bucket item count to get to: 5000 curr count: 12
2021/12/03 21:09:36 Waiting for dst bucket item count to get to: 5000 curr count: 12
2021/12/03 21:09:36 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/03 21:09:36 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 21:09:36 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 21:09:41 Waiting for app: TestOnUpdateN1QLOp to get un-deployed
2021/12/03 21:09:46 Waiting for app: TestOnUpdateN1QLOp to get un-deployed
2021/12/03 21:09:51 Waiting for app: TestOnUpdateN1QLOp to get un-deployed
2021/12/03 21:09:56 Waiting for app: TestOnUpdateN1QLOp to get un-deployed
2021/12/03 21:09:56 App: TestOnUpdateN1QLOp got un-deployed
2021/12/03 21:09:56 No eventing-con process running
2021/12/03 21:09:56 Function: TestOnUpdateN1QLOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateN1QLOp deleting in the background"
}
2021/12/03 21:09:58 Context: Bucket flush, response code: 200 dump: 
2021/12/03 21:10:08 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/03 21:10:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 21:10:09 Context: Bucket flush, response code: 200 dump: 
2021/12/03 21:10:19 Waiting for dst bucket item count to get to: 0 curr count: 2513
2021/12/03 21:10:25 Waiting for dst bucket item count to get to: 0 curr count: 2513
2021/12/03 21:10:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateN1QLOp (101.74s)
=== RUN   TestOnUpdateBucketOpDefaultSettings
2021/12/03 21:10:35 Waiting for app: TestOnUpdateBucketOpDefaultSettings to get un-deployed
2021/12/03 21:10:35 App: TestOnUpdateBucketOpDefaultSettings got un-deployed
2021/12/03 21:10:35 No eventing-con process running
2021/12/03 21:10:35 Function: TestOnUpdateBucketOpDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateBucketOpDefaultSettings not found"
 }
}
2021/12/03 21:10:36 Context: Bucket flush, response code: 200 dump: 
2021/12/03 21:10:41 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 21:10:43 Context: Bucket flush, response code: 200 dump: 
2021/12/03 21:10:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 21:10:48 Deploying app: TestOnUpdateBucketOpDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/03 21:10:49 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateBucketOpDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/03 21:10:49 Starting bucket ops to source bucket
2021/12/03 21:10:57 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 21:11:02 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 21:11:07 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 21:11:12 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 21:11:17 Waiting for dst bucket item count to get to: 5000 curr count: 2500
2021/12/03 21:11:17 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/03 21:11:17 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 21:11:17 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 21:11:22 Waiting for app: TestOnUpdateBucketOpDefaultSettings to get un-deployed
2021/12/03 21:11:27 Waiting for app: TestOnUpdateBucketOpDefaultSettings to get un-deployed
2021/12/03 21:11:32 Waiting for app: TestOnUpdateBucketOpDefaultSettings to get un-deployed
2021/12/03 21:11:37 Waiting for app: TestOnUpdateBucketOpDefaultSettings to get un-deployed
2021/12/03 21:11:37 App: TestOnUpdateBucketOpDefaultSettings got un-deployed
2021/12/03 21:11:37 No eventing-con process running
2021/12/03 21:11:37 Function: TestOnUpdateBucketOpDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateBucketOpDefaultSettings deleting in the background"
}
2021/12/03 21:11:39 Context: Bucket flush, response code: 200 dump: 
2021/12/03 21:11:49 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/03 21:11:54 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/03 21:11:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 21:11:56 Context: Bucket flush, response code: 200 dump: 
2021/12/03 21:12:06 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/03 21:12:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateBucketOpDefaultSettings (101.68s)
=== RUN   TestOnUpdateBucketOpNonDefaultSettings
2021/12/03 21:12:16 Waiting for app: TestOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/03 21:12:16 App: TestOnUpdateBucketOpNonDefaultSettings got un-deployed
2021/12/03 21:12:16 No eventing-con process running
2021/12/03 21:12:16 Function: TestOnUpdateBucketOpNonDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateBucketOpNonDefaultSettings not found"
 }
}
2021/12/03 21:12:18 Context: Bucket flush, response code: 200 dump: 
2021/12/03 21:12:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 21:12:25 Context: Bucket flush, response code: 200 dump: 
2021/12/03 21:12:30 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 21:12:30 Deploying app: TestOnUpdateBucketOpNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/03 21:12:31 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateBucketOpNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/03 21:12:31 Starting bucket ops to source bucket
2021/12/03 21:12:39 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 21:12:44 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 21:12:49 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 21:12:54 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 21:12:54 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/03 21:12:54 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 21:12:54 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 21:12:59 Waiting for app: TestOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/03 21:13:04 Waiting for app: TestOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/03 21:13:09 Waiting for app: TestOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/03 21:13:14 Waiting for app: TestOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/03 21:13:14 App: TestOnUpdateBucketOpNonDefaultSettings got un-deployed
2021/12/03 21:13:14 No eventing-con process running
2021/12/03 21:13:14 Function: TestOnUpdateBucketOpNonDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateBucketOpNonDefaultSettings deleting in the background"
}
2021/12/03 21:13:16 Context: Bucket flush, response code: 200 dump: 
2021/12/03 21:13:26 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/03 21:13:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 21:13:28 Context: Bucket flush, response code: 200 dump: 
2021/12/03 21:13:38 Waiting for dst bucket item count to get to: 0 curr count: 3379
2021/12/03 21:13:43 Waiting for dst bucket item count to get to: 0 curr count: 879
2021/12/03 21:13:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateBucketOpNonDefaultSettings (96.76s)
=== RUN   TestOnUpdateBucketOpDefaultSettings10K
2021/12/03 21:13:53 Waiting for app: TestOnUpdateBucketOpDefaultSettings10K to get un-deployed
2021/12/03 21:13:53 App: TestOnUpdateBucketOpDefaultSettings10K got un-deployed
2021/12/03 21:13:53 No eventing-con process running
2021/12/03 21:13:53 Function: TestOnUpdateBucketOpDefaultSettings10K delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateBucketOpDefaultSettings10K not found"
 }
}
2021/12/03 21:13:55 Context: Bucket flush, response code: 200 dump: 
2021/12/03 21:14:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 21:14:02 Context: Bucket flush, response code: 200 dump: 
2021/12/03 21:14:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 21:14:07 Deploying app: TestOnUpdateBucketOpDefaultSettings10K. Item count src bucket: 0 dst bucket: 0
2021/12/03 21:14:08 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateBucketOpDefaultSettings10K' in metakv",
  "warnings": null
 }
}
2021/12/03 21:14:08 Starting bucket ops to source bucket
2021/12/03 21:14:16 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 21:14:21 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 21:14:26 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 21:14:31 Waiting for dst bucket item count to get to: 5000 curr count: 236
2021/12/03 21:14:36 Waiting for dst bucket item count to get to: 5000 curr count: 236
2021/12/03 21:14:36 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/03 21:14:36 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 21:14:36 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 21:14:41 Waiting for app: TestOnUpdateBucketOpDefaultSettings10K to get un-deployed
2021/12/03 21:14:46 Waiting for app: TestOnUpdateBucketOpDefaultSettings10K to get un-deployed
2021/12/03 21:14:51 Waiting for app: TestOnUpdateBucketOpDefaultSettings10K to get un-deployed
2021/12/03 21:14:56 Waiting for app: TestOnUpdateBucketOpDefaultSettings10K to get un-deployed
2021/12/03 21:14:56 App: TestOnUpdateBucketOpDefaultSettings10K got un-deployed
2021/12/03 21:14:56 No eventing-con process running
2021/12/03 21:14:56 Function: TestOnUpdateBucketOpDefaultSettings10K delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateBucketOpDefaultSettings10K deleting in the background"
}
2021/12/03 21:14:58 Context: Bucket flush, response code: 200 dump: 
2021/12/03 21:15:08 Waiting for dst bucket item count to get to: 0 curr count: 2977
2021/12/03 21:15:13 Waiting for dst bucket item count to get to: 0 curr count: 477
2021/12/03 21:15:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 21:15:15 Context: Bucket flush, response code: 200 dump: 
2021/12/03 21:15:25 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/03 21:15:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateBucketOpDefaultSettings10K (101.80s)
=== RUN   TestOnUpdateBucketOpDefaultSettings100K
2021/12/03 21:15:35 Waiting for app: TestOnUpdateBucketOpDefaultSettings100K to get un-deployed
2021/12/03 21:15:35 App: TestOnUpdateBucketOpDefaultSettings100K got un-deployed
2021/12/03 21:15:35 No eventing-con process running
2021/12/03 21:15:35 Function: TestOnUpdateBucketOpDefaultSettings100K delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateBucketOpDefaultSettings100K not found"
 }
}
2021/12/03 21:15:37 Context: Bucket flush, response code: 200 dump: 
2021/12/03 21:15:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 21:15:43 Context: Bucket flush, response code: 200 dump: 
2021/12/03 21:15:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 21:15:48 Deploying app: TestOnUpdateBucketOpDefaultSettings100K. Item count src bucket: 0 dst bucket: 0
2021/12/03 21:15:50 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateBucketOpDefaultSettings100K' in metakv",
  "warnings": null
 }
}
2021/12/03 21:15:50 Starting bucket ops to source bucket
2021/12/03 21:15:58 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 21:16:03 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 21:16:08 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 21:16:13 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 21:16:18 Waiting for dst bucket item count to get to: 5000 curr count: 2500
2021/12/03 21:16:18 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/03 21:16:18 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 21:16:18 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 21:16:23 Waiting for app: TestOnUpdateBucketOpDefaultSettings100K to get un-deployed
2021/12/03 21:16:28 Waiting for app: TestOnUpdateBucketOpDefaultSettings100K to get un-deployed
2021/12/03 21:16:33 Waiting for app: TestOnUpdateBucketOpDefaultSettings100K to get un-deployed
2021/12/03 21:16:38 Waiting for app: TestOnUpdateBucketOpDefaultSettings100K to get un-deployed
2021/12/03 21:16:38 App: TestOnUpdateBucketOpDefaultSettings100K got un-deployed
2021/12/03 21:16:38 No eventing-con process running
2021/12/03 21:16:38 Function: TestOnUpdateBucketOpDefaultSettings100K delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateBucketOpDefaultSettings100K deleting in the background"
}
2021/12/03 21:16:40 Context: Bucket flush, response code: 200 dump: 
2021/12/03 21:16:50 Waiting for dst bucket item count to get to: 0 curr count: 3148
2021/12/03 21:16:55 Waiting for dst bucket item count to get to: 0 curr count: 3148
2021/12/03 21:16:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 21:16:57 Context: Bucket flush, response code: 200 dump: 
2021/12/03 21:17:07 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/03 21:17:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateBucketOpDefaultSettings100K (102.23s)
=== RUN   TestOnDeleteBucketOp
2021/12/03 21:17:17 Waiting for app: TestOnDeleteBucketOp to get un-deployed
2021/12/03 21:17:17 App: TestOnDeleteBucketOp got un-deployed
2021/12/03 21:17:17 No eventing-con process running
2021/12/03 21:17:17 Function: TestOnDeleteBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnDeleteBucketOp not found"
 }
}
2021/12/03 21:17:19 Context: Bucket flush, response code: 200 dump: 
2021/12/03 21:17:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 21:17:26 Context: Bucket flush, response code: 200 dump: 
2021/12/03 21:17:31 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 21:17:31 Deploying app: TestOnDeleteBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/03 21:17:32 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnDeleteBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/03 21:17:32 Starting bucket ops to source bucket
2021/12/03 21:17:43 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 21:17:48 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 21:17:53 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 21:17:53 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/03 21:17:53 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 21:17:53 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 21:17:58 Waiting for app: TestOnDeleteBucketOp to get un-deployed
2021/12/03 21:18:03 Waiting for app: TestOnDeleteBucketOp to get un-deployed
2021/12/03 21:18:08 Waiting for app: TestOnDeleteBucketOp to get un-deployed
2021/12/03 21:18:13 Waiting for app: TestOnDeleteBucketOp to get un-deployed
2021/12/03 21:18:13 App: TestOnDeleteBucketOp got un-deployed
2021/12/03 21:18:13 No eventing-con process running
2021/12/03 21:18:13 Function: TestOnDeleteBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnDeleteBucketOp deleting in the background"
}
2021/12/03 21:18:15 Context: Bucket flush, response code: 200 dump: 
2021/12/03 21:18:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 21:18:22 Context: Bucket flush, response code: 200 dump: 
2021/12/03 21:18:32 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/03 21:18:37 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/03 21:18:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnDeleteBucketOp (90.02s)
=== RUN   TestOnDeleteBucketOp5K
2021/12/03 21:18:47 Waiting for app: TestOnDeleteBucketOp5K to get un-deployed
2021/12/03 21:18:47 App: TestOnDeleteBucketOp5K got un-deployed
2021/12/03 21:18:47 No eventing-con process running
2021/12/03 21:18:47 Function: TestOnDeleteBucketOp5K delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnDeleteBucketOp5K not found"
 }
}
2021/12/03 21:18:49 Context: Bucket flush, response code: 200 dump: 
2021/12/03 21:18:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 21:18:56 Context: Bucket flush, response code: 200 dump: 
2021/12/03 21:19:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 21:19:01 Deploying app: TestOnDeleteBucketOp5K. Item count src bucket: 0 dst bucket: 0
2021/12/03 21:19:02 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnDeleteBucketOp5K' in metakv",
  "warnings": null
 }
}
2021/12/03 21:19:02 Starting bucket ops to source bucket
2021/12/03 21:19:13 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 21:19:18 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 21:19:23 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 21:19:23 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/03 21:19:23 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 21:19:23 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 21:19:28 Waiting for app: TestOnDeleteBucketOp5K to get un-deployed
2021/12/03 21:19:33 Waiting for app: TestOnDeleteBucketOp5K to get un-deployed
2021/12/03 21:19:38 Waiting for app: TestOnDeleteBucketOp5K to get un-deployed
2021/12/03 21:19:43 Waiting for app: TestOnDeleteBucketOp5K to get un-deployed
2021/12/03 21:19:43 App: TestOnDeleteBucketOp5K got un-deployed
2021/12/03 21:19:43 No eventing-con process running
2021/12/03 21:19:43 Function: TestOnDeleteBucketOp5K delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnDeleteBucketOp5K deleting in the background"
}
2021/12/03 21:19:45 Context: Bucket flush, response code: 200 dump: 
2021/12/03 21:19:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 21:19:52 Context: Bucket flush, response code: 200 dump: 
2021/12/03 21:20:02 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/03 21:20:07 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/03 21:20:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnDeleteBucketOp5K (90.15s)
=== RUN   TestTimerBucketOp
2021/12/03 21:20:17 Waiting for app: TestTimerBucketOp to get un-deployed
2021/12/03 21:20:17 App: TestTimerBucketOp got un-deployed
2021/12/03 21:20:17 No eventing-con process running
2021/12/03 21:20:17 Function: TestTimerBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestTimerBucketOp not found"
 }
}
2021/12/03 21:20:19 Context: Bucket flush, response code: 200 dump: 
2021/12/03 21:20:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 21:20:26 Context: Bucket flush, response code: 200 dump: 
2021/12/03 21:20:31 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 21:20:31 Deploying app: TestTimerBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/03 21:20:32 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestTimerBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/03 21:20:32 Starting bucket ops to source bucket
2021/12/03 21:20:40 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 21:20:45 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 21:20:50 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 21:20:55 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 21:21:00 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 21:21:05 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 21:21:10 Waiting for dst bucket item count to get to: 5000 curr count: 1892
2021/12/03 21:21:15 Waiting for dst bucket item count to get to: 5000 curr count: 1892
2021/12/03 21:21:15 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/03 21:21:15 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 21:21:15 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 21:21:20 Waiting for app: TestTimerBucketOp to get un-deployed
2021/12/03 21:21:25 Waiting for app: TestTimerBucketOp to get un-deployed
2021/12/03 21:21:30 Waiting for app: TestTimerBucketOp to get un-deployed
2021/12/03 21:21:35 Waiting for app: TestTimerBucketOp to get un-deployed
2021/12/03 21:21:35 App: TestTimerBucketOp got un-deployed
2021/12/03 21:21:35 No eventing-con process running
2021/12/03 21:21:35 Function: TestTimerBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestTimerBucketOp deleting in the background"
}
2021/12/03 21:21:37 Context: Bucket flush, response code: 200 dump: 
2021/12/03 21:21:47 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/03 21:21:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 21:21:49 Context: Bucket flush, response code: 200 dump: 
2021/12/03 21:21:59 Waiting for dst bucket item count to get to: 0 curr count: 4900
2021/12/03 21:22:04 Waiting for dst bucket item count to get to: 0 curr count: 2400
2021/12/03 21:22:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestTimerBucketOp (116.71s)
=== RUN   TestTimerInPastBucketOp
2021/12/03 21:22:14 Waiting for app: TestTimerInPastBucketOp to get un-deployed
2021/12/03 21:22:14 App: TestTimerInPastBucketOp got un-deployed
2021/12/03 21:22:14 No eventing-con process running
2021/12/03 21:22:14 Function: TestTimerInPastBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestTimerInPastBucketOp not found"
 }
}
2021/12/03 21:22:16 Context: Bucket flush, response code: 200 dump: 
2021/12/03 21:22:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 21:22:23 Context: Bucket flush, response code: 200 dump: 
2021/12/03 21:22:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 21:22:28 Deploying app: TestTimerInPastBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/03 21:22:29 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestTimerInPastBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/03 21:22:29 Starting bucket ops to source bucket
2021/12/03 21:22:37 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 21:22:42 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 21:22:47 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 21:22:52 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 21:22:57 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 21:23:02 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 21:23:07 Waiting for dst bucket item count to get to: 5000 curr count: 2133
2021/12/03 21:23:12 Waiting for dst bucket item count to get to: 5000 curr count: 4633
2021/12/03 21:23:12 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/03 21:23:12 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 21:23:12 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 21:23:17 Waiting for app: TestTimerInPastBucketOp to get un-deployed
2021/12/03 21:23:22 Waiting for app: TestTimerInPastBucketOp to get un-deployed
2021/12/03 21:23:27 Waiting for app: TestTimerInPastBucketOp to get un-deployed
2021/12/03 21:23:32 Waiting for app: TestTimerInPastBucketOp to get un-deployed
2021/12/03 21:23:32 App: TestTimerInPastBucketOp got un-deployed
2021/12/03 21:23:32 No eventing-con process running
2021/12/03 21:23:32 Function: TestTimerInPastBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestTimerInPastBucketOp deleting in the background"
}
2021/12/03 21:23:34 Context: Bucket flush, response code: 200 dump: 
2021/12/03 21:23:44 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/03 21:23:49 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/03 21:23:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 21:23:51 Context: Bucket flush, response code: 200 dump: 
2021/12/03 21:24:01 Waiting for dst bucket item count to get to: 0 curr count: 4244
2021/12/03 21:24:06 Waiting for dst bucket item count to get to: 0 curr count: 4244
2021/12/03 21:24:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestTimerInPastBucketOp (121.71s)
=== RUN   TestDeployUndeployLoopTimer
2021/12/03 21:24:16 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/03 21:24:16 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/03 21:24:16 No eventing-con process running
2021/12/03 21:24:16 Function: TestDeployUndeployLoopTimer delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDeployUndeployLoopTimer not found"
 }
}
2021/12/03 21:24:17 Context: Bucket flush, response code: 200 dump: 
2021/12/03 21:24:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 21:24:24 Context: Bucket flush, response code: 200 dump: 
2021/12/03 21:24:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 21:24:29 Deploying app: TestDeployUndeployLoopTimer. Item count src bucket: 0 dst bucket: 0
2021/12/03 21:24:30 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopTimer' in metakv",
  "warnings": null
 }
}
2021/12/03 21:24:30 Starting bucket ops to source bucket
2021/12/03 21:24:38 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 21:24:43 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 21:24:48 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 21:24:53 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 21:24:58 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 21:25:03 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 21:25:08 Waiting for dst bucket item count to get to: 5000 curr count: 3237
2021/12/03 21:25:13 Waiting for dst bucket item count to get to: 5000 curr count: 3237
2021/12/03 21:25:13 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/03 21:25:18 Waiting for app: TestDeployUndeployLoopTimer to get deployed
2021/12/03 21:25:18 App: TestDeployUndeployLoopTimer got deployed
2021/12/03 21:25:18 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 21:25:18 Undeploying app: bucket_op_with_timer
2021/12/03 21:25:19 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 21:25:24 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/03 21:25:29 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/03 21:25:34 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/03 21:25:39 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/03 21:25:39 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/03 21:25:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 21:25:39 No eventing-con process running
2021/12/03 21:25:40 Context: Bucket flush, response code: 200 dump: 
2021/12/03 21:25:48 Context: Bucket flush, response code: 200 dump: 
2021/12/03 21:26:23 Deploying app: TestDeployUndeployLoopTimer. Item count src bucket: 0 dst bucket: 0
2021/12/03 21:26:24 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopTimer' in metakv",
  "warnings": null
 }
}
2021/12/03 21:26:24 Starting bucket ops to source bucket
2021/12/03 21:26:32 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 21:26:37 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 21:26:42 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 21:26:47 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 21:26:52 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 21:26:57 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 21:27:02 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 21:27:07 Waiting for dst bucket item count to get to: 5000 curr count: 2500
2021/12/03 21:27:07 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/03 21:27:12 Waiting for app: TestDeployUndeployLoopTimer to get deployed
2021/12/03 21:27:12 App: TestDeployUndeployLoopTimer got deployed
2021/12/03 21:27:12 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 21:27:12 Undeploying app: bucket_op_with_timer
2021/12/03 21:27:12 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 21:27:17 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/03 21:27:22 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/03 21:27:27 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/03 21:27:32 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/03 21:27:32 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/03 21:27:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 21:27:32 No eventing-con process running
2021/12/03 21:27:34 Context: Bucket flush, response code: 200 dump: 
2021/12/03 21:27:41 Context: Bucket flush, response code: 200 dump: 
2021/12/03 21:28:16 Deploying app: TestDeployUndeployLoopTimer. Item count src bucket: 0 dst bucket: 0
2021/12/03 21:28:17 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopTimer' in metakv",
  "warnings": null
 }
}
2021/12/03 21:28:17 Starting bucket ops to source bucket
2021/12/03 21:28:25 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 21:28:30 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 21:28:35 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 21:28:40 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 21:28:45 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 21:28:50 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 21:28:55 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 21:28:55 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/03 21:29:00 Waiting for app: TestDeployUndeployLoopTimer to get deployed
2021/12/03 21:29:00 App: TestDeployUndeployLoopTimer got deployed
2021/12/03 21:29:00 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 21:29:01 Undeploying app: bucket_op_with_timer
2021/12/03 21:29:01 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 21:29:06 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/03 21:29:11 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/03 21:29:16 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/03 21:29:21 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/03 21:29:21 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/03 21:29:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 21:29:21 No eventing-con process running
2021/12/03 21:29:22 Context: Bucket flush, response code: 200 dump: 
2021/12/03 21:29:30 Context: Bucket flush, response code: 200 dump: 
2021/12/03 21:30:05 Deploying app: TestDeployUndeployLoopTimer. Item count src bucket: 0 dst bucket: 0
2021/12/03 21:30:06 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopTimer' in metakv",
  "warnings": null
 }
}
2021/12/03 21:30:06 Starting bucket ops to source bucket
2021/12/03 21:30:14 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 21:30:19 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 21:30:24 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 21:30:29 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 21:30:34 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 21:30:39 Waiting for dst bucket item count to get to: 5000 curr count: 1278
2021/12/03 21:30:44 Waiting for dst bucket item count to get to: 5000 curr count: 1278
2021/12/03 21:30:44 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/03 21:30:49 Waiting for app: TestDeployUndeployLoopTimer to get deployed
2021/12/03 21:30:49 App: TestDeployUndeployLoopTimer got deployed
2021/12/03 21:30:49 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 21:30:49 Undeploying app: bucket_op_with_timer
2021/12/03 21:30:49 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 21:30:54 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/03 21:30:59 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/03 21:31:04 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/03 21:31:09 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/03 21:31:09 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/03 21:31:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 21:31:09 No eventing-con process running
2021/12/03 21:31:11 Context: Bucket flush, response code: 200 dump: 
2021/12/03 21:31:18 Context: Bucket flush, response code: 200 dump: 
2021/12/03 21:31:53 Deploying app: TestDeployUndeployLoopTimer. Item count src bucket: 0 dst bucket: 0
2021/12/03 21:31:54 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopTimer' in metakv",
  "warnings": null
 }
}
2021/12/03 21:31:54 Starting bucket ops to source bucket
2021/12/03 21:32:02 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 21:32:07 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 21:32:12 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 21:32:17 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 21:32:22 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 21:32:27 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 21:32:32 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 21:32:37 Waiting for dst bucket item count to get to: 5000 curr count: 2500
2021/12/03 21:32:37 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/03 21:32:42 Waiting for app: TestDeployUndeployLoopTimer to get deployed
2021/12/03 21:32:42 App: TestDeployUndeployLoopTimer got deployed
2021/12/03 21:32:42 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 21:32:42 Undeploying app: bucket_op_with_timer
2021/12/03 21:32:42 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 21:32:47 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/03 21:32:52 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/03 21:32:58 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/03 21:33:03 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/03 21:33:03 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/03 21:33:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 21:33:03 No eventing-con process running
2021/12/03 21:33:04 Context: Bucket flush, response code: 200 dump: 
2021/12/03 21:33:12 Context: Bucket flush, response code: 200 dump: 
2021/12/03 21:33:47 Function: TestDeployUndeployLoopTimer delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDeployUndeployLoopTimer deleting in the background"
}
--- PASS: TestDeployUndeployLoopTimer (581.07s)
=== RUN   TestMultipleHandlers
2021/12/03 21:33:57 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get un-deployed
2021/12/03 21:33:57 App: TestMultipleHandlers_bucket_op_on_update got un-deployed
2021/12/03 21:33:57 No eventing-con process running
2021/12/03 21:33:57 Function: TestMultipleHandlers_bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMultipleHandlers_bucket_op_on_update not found"
 }
}
2021/12/03 21:33:58 Context: Bucket flush, response code: 200 dump: 
2021/12/03 21:34:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 21:34:05 Context: Bucket flush, response code: 200 dump: 
2021/12/03 21:34:10 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 21:34:15 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get un-deployed
2021/12/03 21:34:15 App: TestMultipleHandlers_bucket_op_on_delete got un-deployed
2021/12/03 21:34:15 No eventing-con process running
2021/12/03 21:34:15 Function: TestMultipleHandlers_bucket_op_on_delete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMultipleHandlers_bucket_op_on_delete not found"
 }
}
2021/12/03 21:34:17 Context: Bucket flush, response code: 200 dump: 
2021/12/03 21:34:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 21:34:24 Context: Bucket flush, response code: 200 dump: 
2021/12/03 21:34:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 21:34:29 Deploying app: TestMultipleHandlers_bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/03 21:34:30 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMultipleHandlers_bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/03 21:34:30 Deploying app: TestMultipleHandlers_bucket_op_on_delete. Item count src bucket: 0 dst bucket: 0
2021/12/03 21:34:31 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMultipleHandlers_bucket_op_on_delete' in metakv",
  "warnings": null
 }
}
2021/12/03 21:34:36 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get deployed
2021/12/03 21:34:41 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get deployed
2021/12/03 21:34:46 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get deployed
2021/12/03 21:34:51 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get deployed
2021/12/03 21:34:51 App: TestMultipleHandlers_bucket_op_on_update got deployed
2021/12/03 21:34:56 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get deployed
2021/12/03 21:35:01 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get deployed
2021/12/03 21:35:06 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get deployed
2021/12/03 21:35:06 App: TestMultipleHandlers_bucket_op_on_delete got deployed
2021/12/03 21:35:06 Starting bucket ops to source bucket
2021/12/03 21:35:17 Waiting for dst bucket item count to get to: 5000 curr count: 1711
2021/12/03 21:35:22 Waiting for dst bucket item count to get to: 5000 curr count: 1711
2021/12/03 21:35:22 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/03 21:35:22 Starting bucket ops to source bucket
2021/12/03 21:35:37 Waiting for dst bucket item count to get to: 10000 curr count: 6564
2021/12/03 21:35:42 Waiting for dst bucket item count to get to: 10000 curr count: 6564
2021/12/03 21:35:42 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/03 21:35:42 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 21:35:42 Function: TestMultipleHandlers_bucket_op_on_update update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/03 21:35:42 Function: TestMultipleHandlers_bucket_op_on_delete update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/03 21:35:42 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 21:35:47 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get un-deployed
2021/12/03 21:35:52 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get un-deployed
2021/12/03 21:35:57 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get un-deployed
2021/12/03 21:36:02 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get un-deployed
2021/12/03 21:36:02 App: TestMultipleHandlers_bucket_op_on_update got un-deployed
2021/12/03 21:36:02 No eventing-con process running
2021/12/03 21:36:02 Function: TestMultipleHandlers_bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMultipleHandlers_bucket_op_on_update deleting in the background"
}
2021/12/03 21:36:04 Context: Bucket flush, response code: 200 dump: 
2021/12/03 21:36:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 21:36:11 Context: Bucket flush, response code: 200 dump: 
2021/12/03 21:36:21 Waiting for dst bucket item count to get to: 0 curr count: 9085
2021/12/03 21:36:26 Waiting for dst bucket item count to get to: 0 curr count: 9085
2021/12/03 21:36:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 21:36:26 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 21:36:31 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get un-deployed
2021/12/03 21:36:36 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get un-deployed
2021/12/03 21:36:41 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get un-deployed
2021/12/03 21:36:46 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get un-deployed
2021/12/03 21:36:46 App: TestMultipleHandlers_bucket_op_on_delete got un-deployed
2021/12/03 21:36:46 No eventing-con process running
2021/12/03 21:36:46 Function: TestMultipleHandlers_bucket_op_on_delete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMultipleHandlers_bucket_op_on_delete deleting in the background"
}
2021/12/03 21:36:48 Context: Bucket flush, response code: 200 dump: 
2021/12/03 21:36:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 21:36:55 Context: Bucket flush, response code: 200 dump: 
2021/12/03 21:37:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestMultipleHandlers (193.36s)
=== RUN   TestPauseResumeLoopDefaultSettings
2021/12/03 21:37:10 Waiting for app: TestPauseResumeLoopDefaultSettings to get un-deployed
2021/12/03 21:37:10 App: TestPauseResumeLoopDefaultSettings got un-deployed
2021/12/03 21:37:10 No eventing-con process running
2021/12/03 21:37:10 Function: TestPauseResumeLoopDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestPauseResumeLoopDefaultSettings not found"
 }
}
2021/12/03 21:37:12 Context: Bucket flush, response code: 200 dump: 
2021/12/03 21:37:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 21:37:19 Context: Bucket flush, response code: 200 dump: 
2021/12/03 21:37:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 21:37:24 Deploying app: TestPauseResumeLoopDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/03 21:37:25 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestPauseResumeLoopDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/03 21:37:25 Starting bucket ops to source bucket
2021/12/03 21:37:33 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 21:37:38 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 21:37:43 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 21:37:48 Waiting for dst bucket item count to get to: 5000 curr count: 1084
2021/12/03 21:37:53 Waiting for dst bucket item count to get to: 5000 curr count: 3584
2021/12/03 21:37:53 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/03 21:37:53 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 21:37:58 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to deployed
2021/12/03 21:37:58 Function: TestPauseResumeLoopDefaultSettings status changed to deployed
2021/12/03 21:37:58 Pausing app: bucket_op_on_update
2021/12/03 21:37:58 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/03 21:38:03 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to paused
2021/12/03 21:38:03 Function: TestPauseResumeLoopDefaultSettings status changed to paused
2021/12/03 21:38:03 Resuming app: bucket_op_on_update
2021/12/03 21:38:04 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/03 21:38:04 Starting bucket ops to source bucket
2021/12/03 21:38:12 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/03 21:38:17 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/03 21:38:22 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/03 21:38:22 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/03 21:38:22 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 21:38:27 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to deployed
2021/12/03 21:38:27 Function: TestPauseResumeLoopDefaultSettings status changed to deployed
2021/12/03 21:38:27 Pausing app: bucket_op_on_update
2021/12/03 21:38:27 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/03 21:38:32 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to paused
2021/12/03 21:38:32 Function: TestPauseResumeLoopDefaultSettings status changed to paused
2021/12/03 21:38:32 Resuming app: bucket_op_on_update
2021/12/03 21:38:34 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/03 21:38:34 Starting bucket ops to source bucket
2021/12/03 21:38:41 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/03 21:38:46 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/03 21:38:51 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/03 21:38:51 src & dst bucket item count matched up. src bucket count: 15000 dst bucket count: 15000
2021/12/03 21:38:51 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 21:38:56 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to deployed
2021/12/03 21:38:57 Function: TestPauseResumeLoopDefaultSettings status changed to deployed
2021/12/03 21:38:57 Pausing app: bucket_op_on_update
2021/12/03 21:38:57 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/03 21:39:02 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to paused
2021/12/03 21:39:02 Function: TestPauseResumeLoopDefaultSettings status changed to paused
2021/12/03 21:39:02 Resuming app: bucket_op_on_update
2021/12/03 21:39:03 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/03 21:39:03 Starting bucket ops to source bucket
2021/12/03 21:39:11 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/03 21:39:16 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/03 21:39:21 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/03 21:39:26 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/03 21:39:26 src & dst bucket item count matched up. src bucket count: 20000 dst bucket count: 20000
2021/12/03 21:39:26 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 21:39:31 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to deployed
2021/12/03 21:39:31 Function: TestPauseResumeLoopDefaultSettings status changed to deployed
2021/12/03 21:39:31 Pausing app: bucket_op_on_update
2021/12/03 21:39:31 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/03 21:39:36 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to paused
2021/12/03 21:39:36 Function: TestPauseResumeLoopDefaultSettings status changed to paused
2021/12/03 21:39:36 Resuming app: bucket_op_on_update
2021/12/03 21:39:37 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/03 21:39:37 Starting bucket ops to source bucket
2021/12/03 21:39:45 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/03 21:39:50 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/03 21:39:55 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/03 21:39:55 src & dst bucket item count matched up. src bucket count: 25000 dst bucket count: 25000
2021/12/03 21:39:55 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 21:40:00 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to deployed
2021/12/03 21:40:00 Function: TestPauseResumeLoopDefaultSettings status changed to deployed
2021/12/03 21:40:00 Pausing app: bucket_op_on_update
2021/12/03 21:40:00 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/03 21:40:05 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to paused
2021/12/03 21:40:05 Function: TestPauseResumeLoopDefaultSettings status changed to paused
2021/12/03 21:40:05 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 21:40:10 Waiting for app: TestPauseResumeLoopDefaultSettings to get un-deployed
2021/12/03 21:40:15 Waiting for app: TestPauseResumeLoopDefaultSettings to get un-deployed
2021/12/03 21:40:20 Waiting for app: TestPauseResumeLoopDefaultSettings to get un-deployed
2021/12/03 21:40:25 Waiting for app: TestPauseResumeLoopDefaultSettings to get un-deployed
2021/12/03 21:40:25 App: TestPauseResumeLoopDefaultSettings got un-deployed
2021/12/03 21:40:25 No eventing-con process running
2021/12/03 21:40:25 Function: TestPauseResumeLoopDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestPauseResumeLoopDefaultSettings deleting in the background"
}
2021/12/03 21:40:27 Context: Bucket flush, response code: 200 dump: 
2021/12/03 21:40:37 Waiting for dst bucket item count to get to: 0 curr count: 25000
2021/12/03 21:40:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 21:40:39 Context: Bucket flush, response code: 200 dump: 
2021/12/03 21:40:49 Waiting for dst bucket item count to get to: 0 curr count: 25000
2021/12/03 21:40:54 Waiting for dst bucket item count to get to: 0 curr count: 12500
2021/12/03 21:40:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestPauseResumeLoopDefaultSettings (233.96s)
=== RUN   TestPauseResumeLoopNonDefaultSettings
2021/12/03 21:41:04 Waiting for app: TestPauseResumeLoopNonDefaultSettings to get un-deployed
2021/12/03 21:41:04 App: TestPauseResumeLoopNonDefaultSettings got un-deployed
2021/12/03 21:41:04 No eventing-con process running
2021/12/03 21:41:04 Function: TestPauseResumeLoopNonDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestPauseResumeLoopNonDefaultSettings not found"
 }
}
2021/12/03 21:41:06 Context: Bucket flush, response code: 200 dump: 
2021/12/03 21:41:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 21:41:13 Context: Bucket flush, response code: 200 dump: 
2021/12/03 21:41:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 21:41:18 Deploying app: TestPauseResumeLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/03 21:41:19 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestPauseResumeLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/03 21:41:19 Starting bucket ops to source bucket
2021/12/03 21:41:27 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 21:41:32 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 21:41:37 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 21:41:42 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 21:41:47 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 21:41:52 Waiting for dst bucket item count to get to: 5000 curr count: 62
2021/12/03 21:41:57 Waiting for dst bucket item count to get to: 5000 curr count: 2562
2021/12/03 21:41:57 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/03 21:41:57 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 21:42:02 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to deployed
2021/12/03 21:42:02 Function: TestPauseResumeLoopNonDefaultSettings status changed to deployed
2021/12/03 21:42:02 Pausing app: bucket_op_with_timer
2021/12/03 21:42:02 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/03 21:42:07 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to paused
2021/12/03 21:42:07 Function: TestPauseResumeLoopNonDefaultSettings status changed to paused
2021/12/03 21:42:07 Resuming app: bucket_op_with_timer
2021/12/03 21:42:08 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/03 21:42:08 Starting bucket ops to source bucket
2021/12/03 21:42:16 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/03 21:42:21 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/03 21:42:26 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/03 21:42:31 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/03 21:42:36 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/03 21:42:41 Waiting for dst bucket item count to get to: 10000 curr count: 5082
2021/12/03 21:42:46 Waiting for dst bucket item count to get to: 10000 curr count: 7582
2021/12/03 21:42:46 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/03 21:42:46 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 21:42:51 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to deployed
2021/12/03 21:42:51 Function: TestPauseResumeLoopNonDefaultSettings status changed to deployed
2021/12/03 21:42:51 Pausing app: bucket_op_with_timer
2021/12/03 21:42:51 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/03 21:42:56 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to paused
2021/12/03 21:42:56 Function: TestPauseResumeLoopNonDefaultSettings status changed to paused
2021/12/03 21:42:56 Resuming app: bucket_op_with_timer
2021/12/03 21:42:58 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/03 21:42:58 Starting bucket ops to source bucket
2021/12/03 21:43:06 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/03 21:43:11 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/03 21:43:16 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/03 21:43:21 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/03 21:43:26 Waiting for dst bucket item count to get to: 15000 curr count: 10580
2021/12/03 21:43:31 Waiting for dst bucket item count to get to: 15000 curr count: 10580
2021/12/03 21:43:31 src & dst bucket item count matched up. src bucket count: 15000 dst bucket count: 15000
2021/12/03 21:43:31 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 21:43:36 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to deployed
2021/12/03 21:43:36 Function: TestPauseResumeLoopNonDefaultSettings status changed to deployed
2021/12/03 21:43:36 Pausing app: bucket_op_with_timer
2021/12/03 21:43:36 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/03 21:43:41 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to paused
2021/12/03 21:43:41 Function: TestPauseResumeLoopNonDefaultSettings status changed to paused
2021/12/03 21:43:41 Resuming app: bucket_op_with_timer
2021/12/03 21:43:42 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/03 21:43:42 Starting bucket ops to source bucket
2021/12/03 21:43:50 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/03 21:43:55 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/03 21:44:00 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/03 21:44:05 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/03 21:44:10 Waiting for dst bucket item count to get to: 20000 curr count: 15388
2021/12/03 21:44:15 Waiting for dst bucket item count to get to: 20000 curr count: 15388
2021/12/03 21:44:20 Waiting for dst bucket item count to get to: 20000 curr count: 18044
2021/12/03 21:44:25 Waiting for dst bucket item count to get to: 20000 curr count: 18044
2021/12/03 21:44:25 src & dst bucket item count matched up. src bucket count: 20000 dst bucket count: 20000
2021/12/03 21:44:25 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 21:44:30 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to deployed
2021/12/03 21:44:30 Function: TestPauseResumeLoopNonDefaultSettings status changed to deployed
2021/12/03 21:44:30 Pausing app: bucket_op_with_timer
2021/12/03 21:44:30 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/03 21:44:35 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to paused
2021/12/03 21:44:35 Function: TestPauseResumeLoopNonDefaultSettings status changed to paused
2021/12/03 21:44:35 Resuming app: bucket_op_with_timer
2021/12/03 21:44:36 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/03 21:44:36 Starting bucket ops to source bucket
2021/12/03 21:44:44 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/03 21:44:49 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/03 21:44:54 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/03 21:44:59 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/03 21:45:04 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/03 21:45:09 Waiting for dst bucket item count to get to: 25000 curr count: 22434
2021/12/03 21:45:14 Waiting for dst bucket item count to get to: 25000 curr count: 22434
2021/12/03 21:45:14 src & dst bucket item count matched up. src bucket count: 25000 dst bucket count: 25000
2021/12/03 21:45:14 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 21:45:20 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to deployed
2021/12/03 21:45:20 Function: TestPauseResumeLoopNonDefaultSettings status changed to deployed
2021/12/03 21:45:20 Pausing app: bucket_op_with_timer
2021/12/03 21:45:20 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/03 21:45:25 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to paused
2021/12/03 21:45:25 Function: TestPauseResumeLoopNonDefaultSettings status changed to paused
2021/12/03 21:45:25 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 21:45:30 Waiting for app: TestPauseResumeLoopNonDefaultSettings to get un-deployed
2021/12/03 21:45:35 Waiting for app: TestPauseResumeLoopNonDefaultSettings to get un-deployed
2021/12/03 21:45:40 Waiting for app: TestPauseResumeLoopNonDefaultSettings to get un-deployed
2021/12/03 21:45:45 Waiting for app: TestPauseResumeLoopNonDefaultSettings to get un-deployed
2021/12/03 21:45:45 App: TestPauseResumeLoopNonDefaultSettings got un-deployed
2021/12/03 21:45:45 No eventing-con process running
2021/12/03 21:45:45 Function: TestPauseResumeLoopNonDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestPauseResumeLoopNonDefaultSettings deleting in the background"
}
2021/12/03 21:45:47 Context: Bucket flush, response code: 200 dump: 
2021/12/03 21:45:57 Waiting for dst bucket item count to get to: 0 curr count: 25000
2021/12/03 21:45:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 21:45:58 Context: Bucket flush, response code: 200 dump: 
2021/12/03 21:46:08 Waiting for dst bucket item count to get to: 0 curr count: 20886
2021/12/03 21:46:13 Waiting for dst bucket item count to get to: 0 curr count: 8386
2021/12/03 21:46:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestPauseResumeLoopNonDefaultSettings (319.42s)
=== RUN   TestPauseAndResumeWithWorkerCountChange
2021/12/03 21:46:23 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get un-deployed
2021/12/03 21:46:23 App: TestPauseAndResumeWithWorkerCountChange got un-deployed
2021/12/03 21:46:23 No eventing-con process running
2021/12/03 21:46:23 Function: TestPauseAndResumeWithWorkerCountChange delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestPauseAndResumeWithWorkerCountChange not found"
 }
}
2021/12/03 21:46:25 Context: Bucket flush, response code: 200 dump: 
2021/12/03 21:46:30 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 21:46:32 Context: Bucket flush, response code: 200 dump: 
2021/12/03 21:46:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 21:46:37 Deploying app: TestPauseAndResumeWithWorkerCountChange. Item count src bucket: 0 dst bucket: 0
2021/12/03 21:46:38 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestPauseAndResumeWithWorkerCountChange' in metakv",
  "warnings": null
 }
}
2021/12/03 21:46:43 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get deployed
2021/12/03 21:46:48 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get deployed
2021/12/03 21:46:53 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get deployed
2021/12/03 21:46:58 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get deployed
2021/12/03 21:46:58 App: TestPauseAndResumeWithWorkerCountChange got deployed
2021/12/03 21:46:58 Starting bucket ops to source bucket
2021/12/03 21:47:08 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 21:47:13 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 21:47:18 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 21:47:23 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 21:47:28 Waiting for dst bucket item count to get to: 5000 curr count: 648
2021/12/03 21:47:33 Waiting for dst bucket item count to get to: 5000 curr count: 2072
2021/12/03 21:47:38 Waiting for dst bucket item count to get to: 5000 curr count: 3924
2021/12/03 21:47:38 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/03 21:47:38 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 21:47:38 Pausing app: bucket_op_with_timer
2021/12/03 21:47:38 Function: TestPauseAndResumeWithWorkerCountChange update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/03 21:47:43 Waiting for function: TestPauseAndResumeWithWorkerCountChange status to change to paused
2021/12/03 21:47:43 Function: TestPauseAndResumeWithWorkerCountChange status changed to paused
2021/12/03 21:47:43 Resuming app: bucket_op_with_timer
2021/12/03 21:47:44 Function: TestPauseAndResumeWithWorkerCountChange update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:6] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/03 21:47:44 Starting bucket ops to source bucket
2021/12/03 21:47:56 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/03 21:48:01 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/03 21:48:06 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/03 21:48:11 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/03 21:48:16 Waiting for dst bucket item count to get to: 10000 curr count: 6177
2021/12/03 21:48:21 Waiting for dst bucket item count to get to: 10000 curr count: 8639
2021/12/03 21:48:26 Waiting for dst bucket item count to get to: 10000 curr count: 9962
2021/12/03 21:48:26 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/03 21:48:26 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 21:48:31 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get un-deployed
2021/12/03 21:48:36 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get un-deployed
2021/12/03 21:48:41 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get un-deployed
2021/12/03 21:48:46 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get un-deployed
2021/12/03 21:48:46 App: TestPauseAndResumeWithWorkerCountChange got un-deployed
2021/12/03 21:48:46 No eventing-con process running
2021/12/03 21:48:46 Function: TestPauseAndResumeWithWorkerCountChange delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestPauseAndResumeWithWorkerCountChange deleting in the background"
}
2021/12/03 21:48:48 Context: Bucket flush, response code: 200 dump: 
2021/12/03 21:48:58 Waiting for dst bucket item count to get to: 0 curr count: 5760
2021/12/03 21:49:03 Waiting for dst bucket item count to get to: 0 curr count: 760
2021/12/03 21:49:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 21:49:05 Context: Bucket flush, response code: 200 dump: 
2021/12/03 21:49:15 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/03 21:49:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestPauseAndResumeWithWorkerCountChange (181.17s)
=== RUN   TestPauseResumeWithEventingReb
2021/12/03 21:49:25 Waiting for app: TestPauseResumeWithEventingReb to get un-deployed
2021/12/03 21:49:25 App: TestPauseResumeWithEventingReb got un-deployed
2021/12/03 21:49:25 No eventing-con process running
2021/12/03 21:49:25 Function: TestPauseResumeWithEventingReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestPauseResumeWithEventingReb not found"
 }
}
2021/12/03 21:49:26 Context: Bucket flush, response code: 200 dump: 
2021/12/03 21:49:31 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 21:49:33 Context: Bucket flush, response code: 200 dump: 
2021/12/03 21:49:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 21:49:38 Deploying app: TestPauseResumeWithEventingReb. Item count src bucket: 0 dst bucket: 0
2021/12/03 21:49:39 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestPauseResumeWithEventingReb' in metakv",
  "warnings": null
 }
}
2021/12/03 21:49:44 Waiting for app: TestPauseResumeWithEventingReb to get deployed
2021/12/03 21:49:49 Waiting for app: TestPauseResumeWithEventingReb to get deployed
2021/12/03 21:49:54 Waiting for app: TestPauseResumeWithEventingReb to get deployed
2021/12/03 21:49:59 Waiting for app: TestPauseResumeWithEventingReb to get deployed
2021/12/03 21:49:59 App: TestPauseResumeWithEventingReb got deployed
2021/12/03 21:49:59 Starting bucket ops to source bucket
2021/12/03 21:50:09 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 21:50:14 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 21:50:19 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 21:50:24 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 21:50:29 Waiting for dst bucket item count to get to: 5000 curr count: 1187
2021/12/03 21:50:34 Waiting for dst bucket item count to get to: 5000 curr count: 2374
2021/12/03 21:50:39 Waiting for dst bucket item count to get to: 5000 curr count: 3687
2021/12/03 21:50:39 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/03 21:50:39 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 21:50:39 Pausing app: bucket_op_with_timer
2021/12/03 21:50:39 Function: TestPauseResumeWithEventingReb update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/03 21:50:44 Waiting for function: TestPauseResumeWithEventingReb status to change to paused
2021/12/03 21:50:44 Function: TestPauseResumeWithEventingReb status changed to paused
2021/12/03 21:50:44 Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/03 21:50:44 Removing node(s): [] from the cluster
2021/12/03 21:50:49 Rebalance progress: 29.16666666666666
2021/12/03 21:50:54 Rebalance progress: 29.16666666666666
2021/12/03 21:50:59 Rebalance progress: 100
2021/12/03 21:50:59 Resuming app: bucket_op_with_timer
2021/12/03 21:51:01 Function: TestPauseResumeWithEventingReb update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:6] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/03 21:51:01 Starting bucket ops to source bucket
2021/12/03 21:51:12 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/03 21:51:17 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/03 21:51:22 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/03 21:51:28 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/03 21:51:33 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/03 21:51:33 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/03 21:51:33 Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/03 21:51:38 Rebalance progress: 29.16666666666666
2021/12/03 21:51:43 Rebalance progress: 29.16666666666666
2021/12/03 21:51:48 Rebalance progress: 100

VB distribution across eventing nodes and their workers::

2021/12/03 21:51:53 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:44} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/03 21:52:08 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:44} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/03 21:52:23 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:44} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/03 21:52:38 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:44} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/03 21:52:53 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:44} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/03 21:53:08 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:44} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/03 21:53:23 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:44} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/03 21:53:38 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:44} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/03 21:53:53 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:44} {Name:#primary Bucket:hello-world Completion:100}]

DCP Stream statuses:
	stream status: running
	len: 1024
	vb list dump: [0-1023]

vbucket curr owner:
Producer node: 127.0.0.1:9302	Node UUID: 8f4040b40e22991ccc6417cab9eed230
	workerID: worker_TestPauseResumeWithEventingReb_0
	len: 57
	v: [0-56]
	workerID: worker_TestPauseResumeWithEventingReb_1
	len: 57
	v: [57-113]
	workerID: worker_TestPauseResumeWithEventingReb_2
	len: 57
	v: [114-170]
	workerID: worker_TestPauseResumeWithEventingReb_3
	len: 57
	v: [171-227]
	workerID: worker_TestPauseResumeWithEventingReb_4
	len: 57
	v: [228-284]
	workerID: worker_TestPauseResumeWithEventingReb_5
	len: 57
	v: [285-341]
Producer node: 172.31.2.134:9300	Node UUID: 17ef4d05f9a1c14fa6d6f1f8b465e112
	workerID: worker_TestPauseResumeWithEventingReb_5
	len: 56
	v: [968-1023]
	workerID: worker_TestPauseResumeWithEventingReb_0
	len: 57
	v: [683-739]
	workerID: worker_TestPauseResumeWithEventingReb_1
	len: 57
	v: [740-796]
	workerID: worker_TestPauseResumeWithEventingReb_2
	len: 57
	v: [797-853]
	workerID: worker_TestPauseResumeWithEventingReb_3
	len: 57
	v: [854-910]
	workerID: worker_TestPauseResumeWithEventingReb_4
	len: 57
	v: [911-967]
Producer node: 127.0.0.1:9303	Node UUID: 3bdd07c6c33df1dbf7c4f258364e903b
	workerID: worker_TestPauseResumeWithEventingReb_0
	len: 57
	v: [342-398]
	workerID: worker_TestPauseResumeWithEventingReb_1
	len: 57
	v: [399-455]
	workerID: worker_TestPauseResumeWithEventingReb_2
	len: 57
	v: [456-512]
	workerID: worker_TestPauseResumeWithEventingReb_3
	len: 57
	v: [513-569]
	workerID: worker_TestPauseResumeWithEventingReb_4
	len: 57
	v: [570-626]
	workerID: worker_TestPauseResumeWithEventingReb_5
	len: 56
	v: [627-682]


2021/12/03 21:54:08 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 21:54:08 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 21:54:13 clusterSetup.go:397: Waiting for app: TestPauseResumeWithEventingReb to get un-deployed
2021/12/03 21:54:18 clusterSetup.go:397: Waiting for app: TestPauseResumeWithEventingReb to get un-deployed
2021/12/03 21:54:23 clusterSetup.go:397: Waiting for app: TestPauseResumeWithEventingReb to get un-deployed
2021/12/03 21:54:28 clusterSetup.go:397: Waiting for app: TestPauseResumeWithEventingReb to get un-deployed
2021/12/03 21:54:28 clusterSetup.go:405: App: TestPauseResumeWithEventingReb got un-deployed
2021/12/03 21:54:28 eventingSetup.go:46: No eventing-con process running
2021/12/03 21:54:28 eventingSetup.go:418: Function: TestPauseResumeWithEventingReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestPauseResumeWithEventingReb deleting in the background"
}
2021/12/03 21:54:30 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/03 21:54:40 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5351
2021/12/03 21:54:45 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5351
2021/12/03 21:54:45 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 21:54:47 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/03 21:54:57 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/03 21:54:57 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestPauseResumeWithEventingReb (342.04s)
=== RUN   TestBucketDeleteAfterPause
2021/12/03 21:55:07 clusterSetup.go:397: Waiting for app: TestBucketDeleteAfterPause to get un-deployed
2021/12/03 21:55:07 clusterSetup.go:405: App: TestBucketDeleteAfterPause got un-deployed
2021/12/03 21:55:07 eventingSetup.go:46: No eventing-con process running
2021/12/03 21:55:07 eventingSetup.go:418: Function: TestBucketDeleteAfterPause delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestBucketDeleteAfterPause not found"
 }
}
2021/12/03 21:55:09 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/03 21:55:14 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 21:55:15 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/03 21:55:20 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 21:55:20 eventingSetup.go:125: Deploying app: TestBucketDeleteAfterPause. Item count src bucket: 0 dst bucket: 0
2021/12/03 21:55:22 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestBucketDeleteAfterPause' in metakv",
  "warnings": null
 }
}
2021/12/03 21:55:27 clusterSetup.go:346: Waiting for app: TestBucketDeleteAfterPause to get deployed
2021/12/03 21:55:32 clusterSetup.go:346: Waiting for app: TestBucketDeleteAfterPause to get deployed
2021/12/03 21:55:37 clusterSetup.go:346: Waiting for app: TestBucketDeleteAfterPause to get deployed
2021/12/03 21:55:42 clusterSetup.go:346: Waiting for app: TestBucketDeleteAfterPause to get deployed
2021/12/03 21:55:42 clusterSetup.go:354: App: TestBucketDeleteAfterPause got deployed
2021/12/03 21:55:42 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/03 21:55:51 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 21:55:51 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/03 21:55:51 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 21:55:51 handler_test.go:712: Pausing app: bucket_op_on_update
2021/12/03 21:55:51 eventingSetup.go:270: Function: TestBucketDeleteAfterPause update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/03 21:55:56 eventingSetup.go:719: Waiting for function: TestBucketDeleteAfterPause status to change to paused
2021/12/03 21:55:57 eventingSetup.go:727: Function: TestBucketDeleteAfterPause status changed to paused
2021/12/03 21:55:57 handler_test.go:716: Deleting source bucket: default
2021/12/03 21:56:03 eventingSetup.go:719: Waiting for function: TestBucketDeleteAfterPause status to change to undeployed
2021/12/03 21:56:03 eventingSetup.go:727: Function: TestBucketDeleteAfterPause status changed to undeployed
2021/12/03 21:56:08 clusterSetup.go:397: Waiting for app: TestBucketDeleteAfterPause to get un-deployed
2021/12/03 21:56:08 clusterSetup.go:405: App: TestBucketDeleteAfterPause got un-deployed
2021/12/03 21:56:08 eventingSetup.go:46: No eventing-con process running
2021/12/03 21:56:08 eventingSetup.go:418: Function: TestBucketDeleteAfterPause delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestBucketDeleteAfterPause deleting in the background"
}
2021/12/03 21:56:10 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/03 21:56:15 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 21:56:16 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/03 21:56:26 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/03 21:56:26 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestBucketDeleteAfterPause (89.89s)
=== RUN   TestChangeFnCodeBetweenPauseResume
2021/12/03 21:56:37 clusterSetup.go:397: Waiting for app: TestChangeFnCodeBetweenPauseResume to get un-deployed
2021/12/03 21:56:37 clusterSetup.go:405: App: TestChangeFnCodeBetweenPauseResume got un-deployed
2021/12/03 21:56:37 eventingSetup.go:46: No eventing-con process running
2021/12/03 21:56:37 eventingSetup.go:418: Function: TestChangeFnCodeBetweenPauseResume delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestChangeFnCodeBetweenPauseResume not found"
 }
}
2021/12/03 21:56:38 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/03 21:56:43 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 21:56:45 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/03 21:56:50 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 21:56:50 eventingSetup.go:125: Deploying app: TestChangeFnCodeBetweenPauseResume. Item count src bucket: 0 dst bucket: 0
2021/12/03 21:56:51 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestChangeFnCodeBetweenPauseResume' in metakv",
  "warnings": null
 }
}
2021/12/03 21:56:56 clusterSetup.go:346: Waiting for app: TestChangeFnCodeBetweenPauseResume to get deployed
2021/12/03 21:57:01 clusterSetup.go:346: Waiting for app: TestChangeFnCodeBetweenPauseResume to get deployed
2021/12/03 21:57:06 clusterSetup.go:346: Waiting for app: TestChangeFnCodeBetweenPauseResume to get deployed
2021/12/03 21:57:11 clusterSetup.go:346: Waiting for app: TestChangeFnCodeBetweenPauseResume to get deployed
2021/12/03 21:57:11 clusterSetup.go:354: App: TestChangeFnCodeBetweenPauseResume got deployed
2021/12/03 21:57:11 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/03 21:57:46 handler_test.go:739: Pausing function: TestChangeFnCodeBetweenPauseResume
2021/12/03 21:57:46 eventingSetup.go:270: Function: TestChangeFnCodeBetweenPauseResume update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/03 21:57:51 eventingSetup.go:719: Waiting for function: TestChangeFnCodeBetweenPauseResume status to change to paused
2021/12/03 21:57:51 eventingSetup.go:727: Function: TestChangeFnCodeBetweenPauseResume status changed to paused
2021/12/03 22:00:51 handler_test.go:748: Resuming function: TestChangeFnCodeBetweenPauseResume
2021/12/03 22:00:51 eventingSetup.go:125: Deploying app: TestChangeFnCodeBetweenPauseResume. Item count src bucket: 5000 dst bucket: 0
2021/12/03 22:00:52 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestChangeFnCodeBetweenPauseResume' in metakv",
  "warnings": null
 }
}
2021/12/03 22:00:57 eventingSetup.go:795: Waiting for function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter to get to 5000
2021/12/03 22:00:57 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 22:00:57 eventingSetup.go:799: Function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter got to 0 expected 5000
2021/12/03 22:01:02 eventingSetup.go:795: Waiting for function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter to get to 5000
2021/12/03 22:01:02 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 22:01:02 eventingSetup.go:799: Function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter got to 3256 expected 5000
2021/12/03 22:01:07 eventingSetup.go:795: Waiting for function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter to get to 5000
2021/12/03 22:01:07 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 22:01:07 eventingSetup.go:799: Function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter got to 4859 expected 5000
2021/12/03 22:01:12 eventingSetup.go:795: Waiting for function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter to get to 5000
2021/12/03 22:01:12 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 22:01:12 eventingSetup.go:804: Function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter got to 5000
2021/12/03 22:01:12 handler_test.go:753: Undeploying function: TestChangeFnCodeBetweenPauseResume
2021/12/03 22:01:12 eventingSetup.go:270: Function: TestChangeFnCodeBetweenPauseResume update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/03 22:01:22 clusterSetup.go:397: Waiting for app: TestChangeFnCodeBetweenPauseResume to get un-deployed
2021/12/03 22:01:27 clusterSetup.go:397: Waiting for app: TestChangeFnCodeBetweenPauseResume to get un-deployed
2021/12/03 22:01:32 clusterSetup.go:397: Waiting for app: TestChangeFnCodeBetweenPauseResume to get un-deployed
2021/12/03 22:01:32 clusterSetup.go:405: App: TestChangeFnCodeBetweenPauseResume got un-deployed
2021/12/03 22:01:32 eventingSetup.go:46: No eventing-con process running
2021/12/03 22:01:32 eventingSetup.go:418: Function: TestChangeFnCodeBetweenPauseResume delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestChangeFnCodeBetweenPauseResume deleting in the background"
}
2021/12/03 22:01:34 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/03 22:01:44 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/03 22:01:49 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/03 22:01:49 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 22:01:51 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/03 22:01:56 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestChangeFnCodeBetweenPauseResume (329.04s)
=== RUN   TestCommentUnCommentOnDelete
2021/12/03 22:02:06 clusterSetup.go:397: Waiting for app: TestCommentUnCommentOnDelete to get un-deployed
2021/12/03 22:02:06 clusterSetup.go:405: App: TestCommentUnCommentOnDelete got un-deployed
2021/12/03 22:02:06 eventingSetup.go:46: No eventing-con process running
2021/12/03 22:02:06 eventingSetup.go:418: Function: TestCommentUnCommentOnDelete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestCommentUnCommentOnDelete not found"
 }
}
2021/12/03 22:02:07 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/03 22:02:12 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 22:02:14 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/03 22:02:19 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 22:02:19 eventingSetup.go:125: Deploying app: TestCommentUnCommentOnDelete. Item count src bucket: 0 dst bucket: 0
2021/12/03 22:02:20 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCommentUnCommentOnDelete' in metakv",
  "warnings": null
 }
}
2021/12/03 22:02:25 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/03 22:02:30 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/03 22:02:35 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/03 22:02:40 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/03 22:02:40 clusterSetup.go:354: App: TestCommentUnCommentOnDelete got deployed
2021/12/03 22:02:40 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/03 22:02:49 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 22:02:54 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 22:02:59 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 2500
2021/12/03 22:02:59 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/03 22:02:59 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 22:02:59 handler_test.go:780: Undeploying app: TestCommentUnCommentOnDelete
2021/12/03 22:02:59 eventingSetup.go:270: Function: TestCommentUnCommentOnDelete update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/03 22:03:29 eventingSetup.go:125: Deploying app: TestCommentUnCommentOnDelete. Item count src bucket: 5000 dst bucket: 5000
2021/12/03 22:03:30 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCommentUnCommentOnDelete' in metakv",
  "warnings": null
 }
}
2021/12/03 22:03:35 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/03 22:03:40 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/03 22:03:45 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/03 22:03:50 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/03 22:03:50 clusterSetup.go:354: App: TestCommentUnCommentOnDelete got deployed
2021/12/03 22:03:50 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/03 22:04:03 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/03 22:04:08 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 716
2021/12/03 22:04:13 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 716
2021/12/03 22:04:13 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 22:04:13 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 22:04:13 handler_test.go:799: Undeploying app: TestCommentUnCommentOnDelete
2021/12/03 22:04:13 eventingSetup.go:270: Function: TestCommentUnCommentOnDelete update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/03 22:04:23 clusterSetup.go:397: Waiting for app: TestCommentUnCommentOnDelete to get un-deployed
2021/12/03 22:04:28 clusterSetup.go:397: Waiting for app: TestCommentUnCommentOnDelete to get un-deployed
2021/12/03 22:04:33 clusterSetup.go:397: Waiting for app: TestCommentUnCommentOnDelete to get un-deployed
2021/12/03 22:04:33 clusterSetup.go:405: App: TestCommentUnCommentOnDelete got un-deployed
2021/12/03 22:04:33 eventingSetup.go:46: No eventing-con process running
2021/12/03 22:04:33 eventingSetup.go:418: Function: TestCommentUnCommentOnDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCommentUnCommentOnDelete deleting in the background"
}
2021/12/03 22:04:35 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/03 22:04:40 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 22:04:42 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/03 22:04:47 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCommentUnCommentOnDelete (171.00s)
=== RUN   TestCPPWorkerCleanup
2021/12/03 22:04:57 clusterSetup.go:397: Waiting for app: TestCPPWorkerCleanup to get un-deployed
2021/12/03 22:04:57 clusterSetup.go:405: App: TestCPPWorkerCleanup got un-deployed
2021/12/03 22:04:57 eventingSetup.go:46: No eventing-con process running
2021/12/03 22:04:57 eventingSetup.go:418: Function: TestCPPWorkerCleanup delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestCPPWorkerCleanup not found"
 }
}
2021/12/03 22:04:58 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/03 22:05:03 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 22:05:05 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/03 22:05:10 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 22:05:10 eventingSetup.go:125: Deploying app: TestCPPWorkerCleanup. Item count src bucket: 0 dst bucket: 0
2021/12/03 22:05:11 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCPPWorkerCleanup' in metakv",
  "warnings": null
 }
}
2021/12/03 22:05:11 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/03 22:05:18 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 22:05:23 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 22:05:28 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 22:05:33 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 22:05:38 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 4191
2021/12/03 22:05:43 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 4191
2021/12/03 22:05:43 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/03 22:05:43 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 22:05:43 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 22:05:48 clusterSetup.go:397: Waiting for app: TestCPPWorkerCleanup to get un-deployed
2021/12/03 22:05:53 clusterSetup.go:397: Waiting for app: TestCPPWorkerCleanup to get un-deployed
2021/12/03 22:05:58 clusterSetup.go:397: Waiting for app: TestCPPWorkerCleanup to get un-deployed
2021/12/03 22:06:03 clusterSetup.go:397: Waiting for app: TestCPPWorkerCleanup to get un-deployed
2021/12/03 22:06:03 clusterSetup.go:405: App: TestCPPWorkerCleanup got un-deployed
2021/12/03 22:06:03 eventingSetup.go:46: No eventing-con process running
2021/12/03 22:06:03 eventingSetup.go:418: Function: TestCPPWorkerCleanup delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCPPWorkerCleanup deleting in the background"
}
2021/12/03 22:06:05 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/03 22:06:15 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/03 22:06:15 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 22:06:17 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/03 22:06:27 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/03 22:06:27 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCPPWorkerCleanup (129.98s)
=== RUN   TestWithUserXattrs
2021/12/03 22:07:07 clusterSetup.go:397: Waiting for app: TestWithUserXattrs to get un-deployed
2021/12/03 22:07:07 clusterSetup.go:405: App: TestWithUserXattrs got un-deployed
2021/12/03 22:07:07 eventingSetup.go:46: No eventing-con process running
2021/12/03 22:07:07 eventingSetup.go:418: Function: TestWithUserXattrs delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestWithUserXattrs not found"
 }
}
2021/12/03 22:07:08 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/03 22:07:13 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 22:07:15 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/03 22:07:20 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 22:07:20 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/03 22:07:22 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/03 22:07:24 eventingSetup.go:125: Deploying app: TestWithUserXattrs. Item count src bucket: 0 dst bucket: 0
2021/12/03 22:07:25 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestWithUserXattrs' in metakv",
  "warnings": null
 }
}
2021/12/03 22:07:30 clusterSetup.go:346: Waiting for app: TestWithUserXattrs to get deployed
2021/12/03 22:07:35 clusterSetup.go:346: Waiting for app: TestWithUserXattrs to get deployed
2021/12/03 22:07:40 clusterSetup.go:346: Waiting for app: TestWithUserXattrs to get deployed
2021/12/03 22:07:45 clusterSetup.go:346: Waiting for app: TestWithUserXattrs to get deployed
2021/12/03 22:07:45 clusterSetup.go:354: App: TestWithUserXattrs got deployed
2021/12/03 22:07:45 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/03 22:07:54 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 22:07:59 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 3764
2021/12/03 22:08:04 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 3764
2021/12/03 22:08:04 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/03 22:08:04 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 22:08:04 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 22:08:09 clusterSetup.go:397: Waiting for app: TestWithUserXattrs to get un-deployed
2021/12/03 22:08:14 clusterSetup.go:397: Waiting for app: TestWithUserXattrs to get un-deployed
2021/12/03 22:08:19 clusterSetup.go:397: Waiting for app: TestWithUserXattrs to get un-deployed
2021/12/03 22:08:24 clusterSetup.go:397: Waiting for app: TestWithUserXattrs to get un-deployed
2021/12/03 22:08:24 clusterSetup.go:405: App: TestWithUserXattrs got un-deployed
2021/12/03 22:08:24 eventingSetup.go:46: No eventing-con process running
2021/12/03 22:08:25 eventingSetup.go:418: Function: TestWithUserXattrs delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestWithUserXattrs deleting in the background"
}
2021/12/03 22:08:26 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/03 22:08:36 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/03 22:08:36 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 22:08:38 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/03 22:08:48 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/03 22:08:48 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestWithUserXattrs (111.09s)
=== RUN   TestEventProcessingPostBucketFlush
2021/12/03 22:08:58 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/03 22:08:58 clusterSetup.go:405: App: TestEventProcessingPostBucketFlush got un-deployed
2021/12/03 22:08:58 eventingSetup.go:46: No eventing-con process running
2021/12/03 22:08:58 eventingSetup.go:418: Function: TestEventProcessingPostBucketFlush delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventProcessingPostBucketFlush not found"
 }
}
2021/12/03 22:08:59 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/03 22:09:04 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 22:09:06 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/03 22:09:11 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 22:09:11 eventingSetup.go:125: Deploying app: TestEventProcessingPostBucketFlush. Item count src bucket: 0 dst bucket: 0
2021/12/03 22:09:12 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventProcessingPostBucketFlush' in metakv",
  "warnings": null
 }
}
2021/12/03 22:09:17 clusterSetup.go:346: Waiting for app: TestEventProcessingPostBucketFlush to get deployed
2021/12/03 22:09:22 clusterSetup.go:346: Waiting for app: TestEventProcessingPostBucketFlush to get deployed
2021/12/03 22:09:27 clusterSetup.go:346: Waiting for app: TestEventProcessingPostBucketFlush to get deployed
2021/12/03 22:09:32 clusterSetup.go:346: Waiting for app: TestEventProcessingPostBucketFlush to get deployed
2021/12/03 22:09:32 clusterSetup.go:354: App: TestEventProcessingPostBucketFlush got deployed
2021/12/03 22:09:32 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/03 22:09:41 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 22:09:46 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 22:09:46 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/03 22:09:46 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 22:09:50 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/03 22:10:00 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1654
2021/12/03 22:10:05 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1654
2021/12/03 22:10:05 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 22:10:06 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/03 22:10:16 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/03 22:10:16 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 22:10:16 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/03 22:10:43 eventingSetup.go:441: Waiting for dst bucket item count to get to: 25000 curr count: 13904
2021/12/03 22:10:48 eventingSetup.go:441: Waiting for dst bucket item count to get to: 25000 curr count: 24811
2021/12/03 22:10:53 eventingSetup.go:441: Waiting for dst bucket item count to get to: 25000 curr count: 24811
2021/12/03 22:10:53 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 25000 dst bucket count: 25000
2021/12/03 22:10:53 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 22:10:53 eventingSetup.go:270: Function: TestEventProcessingPostBucketFlush update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/03 22:10:58 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/03 22:11:03 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/03 22:11:08 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/03 22:11:13 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/03 22:11:13 clusterSetup.go:405: App: TestEventProcessingPostBucketFlush got un-deployed
2021/12/03 22:11:13 eventingSetup.go:46: No eventing-con process running
2021/12/03 22:11:18 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/03 22:11:18 clusterSetup.go:405: App: TestEventProcessingPostBucketFlush got un-deployed
2021/12/03 22:11:18 eventingSetup.go:46: No eventing-con process running
2021/12/03 22:11:18 eventingSetup.go:418: Function: TestEventProcessingPostBucketFlush delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventProcessingPostBucketFlush deleting in the background"
}
2021/12/03 22:11:19 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/03 22:11:29 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 25000
2021/12/03 22:11:34 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12500
2021/12/03 22:11:34 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 22:11:36 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/03 22:11:46 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 25000
2021/12/03 22:11:46 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventProcessingPostBucketFlush (178.23s)
=== RUN   TestMetaBucketDelete
2021/12/03 22:11:56 clusterSetup.go:397: Waiting for app: TestMetaBucketDelete to get un-deployed
2021/12/03 22:11:56 clusterSetup.go:405: App: TestMetaBucketDelete got un-deployed
2021/12/03 22:11:56 eventingSetup.go:46: No eventing-con process running
2021/12/03 22:11:56 eventingSetup.go:418: Function: TestMetaBucketDelete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMetaBucketDelete not found"
 }
}
2021/12/03 22:11:57 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/03 22:12:02 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 22:12:04 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/03 22:12:09 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 22:12:09 eventingSetup.go:125: Deploying app: TestMetaBucketDelete. Item count src bucket: 0 dst bucket: 0
2021/12/03 22:12:10 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMetaBucketDelete' in metakv",
  "warnings": null
 }
}
2021/12/03 22:12:15 clusterSetup.go:346: Waiting for app: TestMetaBucketDelete to get deployed
2021/12/03 22:12:20 clusterSetup.go:346: Waiting for app: TestMetaBucketDelete to get deployed
2021/12/03 22:12:25 clusterSetup.go:346: Waiting for app: TestMetaBucketDelete to get deployed
2021/12/03 22:12:30 clusterSetup.go:346: Waiting for app: TestMetaBucketDelete to get deployed
2021/12/03 22:12:30 clusterSetup.go:354: App: TestMetaBucketDelete got deployed
2021/12/03 22:12:30 handler_test.go:902: Deleting metadata bucket: eventing
2021/12/03 22:12:32 handler_test.go:904: Deleted metadata bucket: eventing
2021/12/03 22:12:37 clusterSetup.go:397: Waiting for app: TestMetaBucketDelete to get un-deployed
2021/12/03 22:12:37 clusterSetup.go:405: App: TestMetaBucketDelete got un-deployed
2021/12/03 22:12:52 clusterSetup.go:397: Waiting for app: TestMetaBucketDelete to get un-deployed
2021/12/03 22:12:52 clusterSetup.go:405: App: TestMetaBucketDelete got un-deployed
2021/12/03 22:12:52 eventingSetup.go:46: No eventing-con process running
2021/12/03 22:12:52 eventingSetup.go:418: Function: TestMetaBucketDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMetaBucketDelete deleting in the background"
}
2021/12/03 22:12:53 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/03 22:12:58 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 22:13:00 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/03 22:13:05 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestMetaBucketDelete (79.28s)
=== RUN   TestMetaBucketDeleteWithBootstrap
2021/12/03 22:13:15 clusterSetup.go:397: Waiting for app: TestMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/03 22:13:15 clusterSetup.go:405: App: TestMetaBucketDeleteWithBootstrap got un-deployed
2021/12/03 22:13:15 eventingSetup.go:46: No eventing-con process running
2021/12/03 22:13:15 eventingSetup.go:418: Function: TestMetaBucketDeleteWithBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMetaBucketDeleteWithBootstrap not found"
 }
}
2021/12/03 22:13:17 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/03 22:13:22 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 22:13:23 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/03 22:13:28 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 22:13:28 eventingSetup.go:125: Deploying app: TestMetaBucketDeleteWithBootstrap. Item count src bucket: 0 dst bucket: 0
2021/12/03 22:13:29 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMetaBucketDeleteWithBootstrap' in metakv",
  "warnings": null
 }
}
2021/12/03 22:13:48 handler_test.go:923: Deleting metadata bucket: eventing
2021/12/03 22:13:49 handler_test.go:925: Deleted metadata bucket: eventing
2021/12/03 22:14:04 clusterSetup.go:397: Waiting for app: TestMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/03 22:14:04 clusterSetup.go:405: App: TestMetaBucketDeleteWithBootstrap got un-deployed
2021/12/03 22:14:19 clusterSetup.go:397: Waiting for app: TestMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/03 22:14:19 clusterSetup.go:405: App: TestMetaBucketDeleteWithBootstrap got un-deployed
2021/12/03 22:14:19 eventingSetup.go:46: No eventing-con process running
2021/12/03 22:14:19 eventingSetup.go:418: Function: TestMetaBucketDeleteWithBootstrap delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMetaBucketDeleteWithBootstrap deleting in the background"
}
2021/12/03 22:14:20 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/03 22:14:25 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 22:14:27 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/03 22:14:32 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestMetaBucketDeleteWithBootstrap (87.00s)
=== RUN   TestSourceBucketDelete
2021/12/03 22:14:42 clusterSetup.go:397: Waiting for app: TestSourceBucketDelete to get un-deployed
2021/12/03 22:14:42 clusterSetup.go:405: App: TestSourceBucketDelete got un-deployed
2021/12/03 22:14:42 eventingSetup.go:46: No eventing-con process running
2021/12/03 22:14:42 eventingSetup.go:418: Function: TestSourceBucketDelete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestSourceBucketDelete not found"
 }
}
2021/12/03 22:14:44 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/03 22:14:49 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 22:14:50 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/03 22:14:55 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 22:14:55 eventingSetup.go:125: Deploying app: TestSourceBucketDelete. Item count src bucket: 0 dst bucket: 0
2021/12/03 22:14:56 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestSourceBucketDelete' in metakv",
  "warnings": null
 }
}
2021/12/03 22:15:01 clusterSetup.go:346: Waiting for app: TestSourceBucketDelete to get deployed
2021/12/03 22:15:07 clusterSetup.go:346: Waiting for app: TestSourceBucketDelete to get deployed
2021/12/03 22:15:12 clusterSetup.go:346: Waiting for app: TestSourceBucketDelete to get deployed
2021/12/03 22:15:17 clusterSetup.go:346: Waiting for app: TestSourceBucketDelete to get deployed
2021/12/03 22:15:17 clusterSetup.go:354: App: TestSourceBucketDelete got deployed
2021/12/03 22:15:17 handler_test.go:945: Deleting source bucket: default
2021/12/03 22:15:18 handler_test.go:947: Deleted source bucket: default
2021/12/03 22:15:23 clusterSetup.go:397: Waiting for app: TestSourceBucketDelete to get un-deployed
2021/12/03 22:15:23 clusterSetup.go:405: App: TestSourceBucketDelete got un-deployed
2021/12/03 22:15:38 clusterSetup.go:397: Waiting for app: TestSourceBucketDelete to get un-deployed
2021/12/03 22:15:38 clusterSetup.go:405: App: TestSourceBucketDelete got un-deployed
2021/12/03 22:15:38 eventingSetup.go:46: No eventing-con process running
2021/12/03 22:15:38 eventingSetup.go:418: Function: TestSourceBucketDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestSourceBucketDelete deleting in the background"
}
2021/12/03 22:15:39 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/03 22:15:44 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 22:15:46 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/03 22:15:51 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestSourceBucketDelete (79.04s)
=== RUN   TestSourceBucketDeleteWithBootstrap
2021/12/03 22:16:01 clusterSetup.go:397: Waiting for app: TestSourceBucketDeleteWithBootstrap to get un-deployed
2021/12/03 22:16:01 clusterSetup.go:405: App: TestSourceBucketDeleteWithBootstrap got un-deployed
2021/12/03 22:16:01 eventingSetup.go:46: No eventing-con process running
2021/12/03 22:16:01 eventingSetup.go:418: Function: TestSourceBucketDeleteWithBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestSourceBucketDeleteWithBootstrap not found"
 }
}
2021/12/03 22:16:03 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/03 22:16:08 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 22:16:09 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/03 22:16:14 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 22:16:14 eventingSetup.go:125: Deploying app: TestSourceBucketDeleteWithBootstrap. Item count src bucket: 0 dst bucket: 0
2021/12/03 22:16:15 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestSourceBucketDeleteWithBootstrap' in metakv",
  "warnings": null
 }
}
2021/12/03 22:16:34 handler_test.go:966: Deleting source bucket: default
2021/12/03 22:16:36 handler_test.go:968: Deleted source bucket: default
2021/12/03 22:16:51 clusterSetup.go:397: Waiting for app: TestSourceBucketDeleteWithBootstrap to get un-deployed
2021/12/03 22:16:51 clusterSetup.go:405: App: TestSourceBucketDeleteWithBootstrap got un-deployed
2021/12/03 22:17:06 clusterSetup.go:397: Waiting for app: TestSourceBucketDeleteWithBootstrap to get un-deployed
2021/12/03 22:17:06 clusterSetup.go:405: App: TestSourceBucketDeleteWithBootstrap got un-deployed
2021/12/03 22:17:06 eventingSetup.go:46: No eventing-con process running
2021/12/03 22:17:06 eventingSetup.go:418: Function: TestSourceBucketDeleteWithBootstrap delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestSourceBucketDeleteWithBootstrap deleting in the background"
}
2021/12/03 22:17:07 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/03 22:17:12 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 22:17:14 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/03 22:17:19 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestSourceBucketDeleteWithBootstrap (87.76s)
=== RUN   TestSourceAndMetaBucketDeleteWithBootstrap
2021/12/03 22:17:29 clusterSetup.go:397: Waiting for app: TestSourceAndMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/03 22:17:29 clusterSetup.go:405: App: TestSourceAndMetaBucketDeleteWithBootstrap got un-deployed
2021/12/03 22:17:29 eventingSetup.go:46: No eventing-con process running
2021/12/03 22:17:29 eventingSetup.go:418: Function: TestSourceAndMetaBucketDeleteWithBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestSourceAndMetaBucketDeleteWithBootstrap not found"
 }
}
2021/12/03 22:17:30 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/03 22:17:35 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 22:17:37 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/03 22:17:42 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 22:17:42 eventingSetup.go:125: Deploying app: TestSourceAndMetaBucketDeleteWithBootstrap. Item count src bucket: 0 dst bucket: 0
2021/12/03 22:17:43 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestSourceAndMetaBucketDeleteWithBootstrap' in metakv",
  "warnings": null
 }
}
2021/12/03 22:18:02 handler_test.go:988: Deleting source bucket: default
2021/12/03 22:18:03 handler_test.go:990: Deleted source bucket: default
2021/12/03 22:18:03 handler_test.go:992: Deleting metadata bucket: eventing
2021/12/03 22:18:04 handler_test.go:994: Deleted metadata bucket: eventing
2021/12/03 22:18:19 clusterSetup.go:397: Waiting for app: TestSourceAndMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/03 22:18:19 clusterSetup.go:405: App: TestSourceAndMetaBucketDeleteWithBootstrap got un-deployed
2021/12/03 22:18:34 clusterSetup.go:397: Waiting for app: TestSourceAndMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/03 22:18:34 clusterSetup.go:405: App: TestSourceAndMetaBucketDeleteWithBootstrap got un-deployed
2021/12/03 22:18:34 eventingSetup.go:46: No eventing-con process running
2021/12/03 22:18:34 eventingSetup.go:418: Function: TestSourceAndMetaBucketDeleteWithBootstrap delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestSourceAndMetaBucketDeleteWithBootstrap deleting in the background"
}
2021/12/03 22:18:35 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/03 22:18:40 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 22:18:42 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/03 22:18:47 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestSourceAndMetaBucketDeleteWithBootstrap (88.13s)
=== RUN   TestUndeployDuringBootstrap
2021/12/03 22:18:57 clusterSetup.go:397: Waiting for app: TestUndeployDuringBootstrap to get un-deployed
2021/12/03 22:18:57 clusterSetup.go:405: App: TestUndeployDuringBootstrap got un-deployed
2021/12/03 22:18:57 eventingSetup.go:46: No eventing-con process running
2021/12/03 22:18:57 eventingSetup.go:418: Function: TestUndeployDuringBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestUndeployDuringBootstrap not found"
 }
}
2021/12/03 22:18:58 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/03 22:19:03 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 22:19:05 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/03 22:19:10 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 22:19:10 eventingSetup.go:125: Deploying app: TestUndeployDuringBootstrap. Item count src bucket: 0 dst bucket: 0
2021/12/03 22:19:11 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestUndeployDuringBootstrap' in metakv",
  "warnings": null
 }
}
2021/12/03 22:19:11 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/03 22:19:23 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 22:19:23 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 22:19:23 clusterSetup.go:375: Apps undergoing bootstrap: true
2021/12/03 22:19:23 clusterSetup.go:378: App: TestUndeployDuringBootstrap started bootstrap
2021/12/03 22:19:23 clusterSetup.go:375: Apps undergoing bootstrap: true
2021/12/03 22:19:28 clusterSetup.go:375: Apps undergoing bootstrap: true
2021/12/03 22:19:33 clusterSetup.go:367: No apps undergoing bootstrap
2021/12/03 22:19:38 clusterSetup.go:397: Waiting for app: TestUndeployDuringBootstrap to get un-deployed
2021/12/03 22:19:43 clusterSetup.go:397: Waiting for app: TestUndeployDuringBootstrap to get un-deployed
2021/12/03 22:19:48 clusterSetup.go:397: Waiting for app: TestUndeployDuringBootstrap to get un-deployed
2021/12/03 22:19:48 clusterSetup.go:405: App: TestUndeployDuringBootstrap got un-deployed
2021/12/03 22:19:48 eventingSetup.go:46: No eventing-con process running
2021/12/03 22:20:13 clusterSetup.go:397: Waiting for app: TestUndeployDuringBootstrap to get un-deployed
2021/12/03 22:20:13 clusterSetup.go:405: App: TestUndeployDuringBootstrap got un-deployed
2021/12/03 22:20:13 eventingSetup.go:46: No eventing-con process running
2021/12/03 22:20:14 eventingSetup.go:418: Function: TestUndeployDuringBootstrap delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestUndeployDuringBootstrap deleting in the background"
}
2021/12/03 22:20:15 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/03 22:20:25 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/03 22:20:25 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 22:20:27 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/03 22:20:37 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/03 22:20:37 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestUndeployDuringBootstrap (109.53s)
=== RUN   TestDeleteBeforeUndeploy
2021/12/03 22:20:42 eventingSetup.go:125: Deploying app: TestDeleteBeforeUndeploy. Item count src bucket: 0 dst bucket: 0
2021/12/03 22:20:43 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeleteBeforeUndeploy' in metakv",
  "warnings": null
 }
}
2021/12/03 22:20:48 clusterSetup.go:346: Waiting for app: TestDeleteBeforeUndeploy to get deployed
2021/12/03 22:20:53 clusterSetup.go:346: Waiting for app: TestDeleteBeforeUndeploy to get deployed
2021/12/03 22:20:58 clusterSetup.go:346: Waiting for app: TestDeleteBeforeUndeploy to get deployed
2021/12/03 22:21:03 clusterSetup.go:346: Waiting for app: TestDeleteBeforeUndeploy to get deployed
2021/12/03 22:21:03 clusterSetup.go:354: App: TestDeleteBeforeUndeploy got deployed
2021/12/03 22:21:03 eventingSetup.go:418: Function: TestDeleteBeforeUndeploy delete from main store request response code: 422 dump: {
 "name": "ERR_APP_NOT_UNDEPLOYED",
 "code": 22,
 "description": "Function hasn't undeployed",
 "attributes": null,
 "runtime_info": {
  "code": 22,
  "info": "Function: TestDeleteBeforeUndeploy skipping delete request from primary store, as it hasn't been undeployed"
 }
}
2021/12/03 22:21:03 eventingSetup.go:270: Function: TestDeleteBeforeUndeploy update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/03 22:21:03 eventingSetup.go:418: Function: TestDeleteBeforeUndeploy delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDeleteBeforeUndeploy deleting in the background"
}
2021/12/03 22:21:08 clusterSetup.go:397: Waiting for app: TestDeleteBeforeUndeploy to get un-deployed
2021/12/03 22:21:08 clusterSetup.go:405: App: TestDeleteBeforeUndeploy got un-deployed
2021/12/03 22:21:08 eventingSetup.go:46: No eventing-con process running
2021/12/03 22:21:08 eventingSetup.go:418: Function: TestDeleteBeforeUndeploy delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDeleteBeforeUndeploy not found"
 }
}
2021/12/03 22:21:09 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/03 22:21:14 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 22:21:16 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/03 22:21:21 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDeleteBeforeUndeploy (44.65s)
=== RUN   TestUndeployWhenTimersAreFired
2021/12/03 22:21:31 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/03 22:21:31 clusterSetup.go:405: App: TestUndeployWhenTimersAreFired got un-deployed
2021/12/03 22:21:31 eventingSetup.go:46: No eventing-con process running
2021/12/03 22:21:31 eventingSetup.go:418: Function: TestUndeployWhenTimersAreFired delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestUndeployWhenTimersAreFired not found"
 }
}
2021/12/03 22:21:33 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/03 22:21:38 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 22:21:39 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/03 22:21:44 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 22:21:44 eventingSetup.go:125: Deploying app: TestUndeployWhenTimersAreFired. Item count src bucket: 0 dst bucket: 0
2021/12/03 22:21:46 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestUndeployWhenTimersAreFired' in metakv",
  "warnings": null
 }
}
2021/12/03 22:21:51 clusterSetup.go:346: Waiting for app: TestUndeployWhenTimersAreFired to get deployed
2021/12/03 22:21:56 clusterSetup.go:346: Waiting for app: TestUndeployWhenTimersAreFired to get deployed
2021/12/03 22:22:01 clusterSetup.go:346: Waiting for app: TestUndeployWhenTimersAreFired to get deployed
2021/12/03 22:22:06 clusterSetup.go:346: Waiting for app: TestUndeployWhenTimersAreFired to get deployed
2021/12/03 22:22:06 clusterSetup.go:354: App: TestUndeployWhenTimersAreFired got deployed
2021/12/03 22:22:06 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/03 22:22:36 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 22:22:41 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/03 22:22:46 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/03 22:22:51 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/03 22:22:56 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/03 22:22:56 clusterSetup.go:405: App: TestUndeployWhenTimersAreFired got un-deployed
2021/12/03 22:22:56 eventingSetup.go:46: No eventing-con process running
2021/12/03 22:24:41 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/03 22:24:41 clusterSetup.go:405: App: TestUndeployWhenTimersAreFired got un-deployed
2021/12/03 22:24:41 eventingSetup.go:46: No eventing-con process running
2021/12/03 22:24:41 eventingSetup.go:418: Function: TestUndeployWhenTimersAreFired delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestUndeployWhenTimersAreFired deleting in the background"
}
2021/12/03 22:24:43 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/03 22:24:53 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/03 22:24:58 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/03 22:24:58 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 22:25:00 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/03 22:25:10 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/03 22:25:15 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/03 22:25:15 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestUndeployWhenTimersAreFired (233.40s)
=== RUN   TestUndeployWithKVFailover
2021/12/03 22:25:25 clusterSetup.go:397: Waiting for app: TestUndeployWithKVFailover to get un-deployed
2021/12/03 22:25:25 clusterSetup.go:405: App: TestUndeployWithKVFailover got un-deployed
2021/12/03 22:25:25 eventingSetup.go:46: No eventing-con process running
2021/12/03 22:25:25 eventingSetup.go:418: Function: TestUndeployWithKVFailover delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestUndeployWithKVFailover not found"
 }
}
2021/12/03 22:25:26 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/03 22:25:31 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 22:25:33 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/03 22:25:38 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 22:25:38 eventingSetup.go:125: Deploying app: TestUndeployWithKVFailover. Item count src bucket: 0 dst bucket: 0
2021/12/03 22:25:39 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestUndeployWithKVFailover' in metakv",
  "warnings": null
 }
}
2021/12/03 22:25:39 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/03 22:25:46 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 22:25:51 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 22:25:56 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 22:26:01 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 22:26:06 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 22:26:11 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 64
2021/12/03 22:26:16 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 2003
2021/12/03 22:26:21 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 4439
2021/12/03 22:26:21 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/03 22:26:21 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: kv to the cluster
2021/12/03 22:26:22 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/03 22:26:27 Rebalance progress: 29.16666666666666
2021/12/03 22:26:32 Rebalance progress: 29.16666666666666
2021/12/03 22:26:37 Rebalance progress: 100
2021/12/03 22:26:37 Failing over: [127.0.0.1:19003]
2021/12/03 22:26:47 Function: TestUndeployWithKVFailover update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/03 22:27:47 Removing node(s): [] from the cluster
2021/12/03 22:27:52 Rebalance progress: 29.16666666666666
2021/12/03 22:27:57 Rebalance progress: 29.16666666666666
2021/12/03 22:28:02 Rebalance progress: 100
2021/12/03 22:28:07 Waiting for app: TestUndeployWithKVFailover to get un-deployed
2021/12/03 22:28:07 App: TestUndeployWithKVFailover got un-deployed
2021/12/03 22:28:07 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 22:28:12 Waiting for app: TestUndeployWithKVFailover to get un-deployed
2021/12/03 22:28:12 App: TestUndeployWithKVFailover got un-deployed
2021/12/03 22:28:12 No eventing-con process running
2021/12/03 22:28:12 Function: TestUndeployWithKVFailover delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestUndeployWithKVFailover deleting in the background"
}
2021/12/03 22:28:14 Context: Bucket flush, response code: 200 dump: 
2021/12/03 22:28:24 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/03 22:28:29 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/03 22:28:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 22:28:30 Context: Bucket flush, response code: 200 dump: 
2021/12/03 22:28:40 Waiting for dst bucket item count to get to: 0 curr count: 3958
2021/12/03 22:28:45 Waiting for dst bucket item count to get to: 0 curr count: 3958
2021/12/03 22:28:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestUndeployWithKVFailover (210.78s)
=== RUN   TestBucketFlushWhileFnDeployed
2021/12/03 22:28:55 Waiting for app: TestBucketFlushWhileFnDeployed to get un-deployed
2021/12/03 22:28:55 App: TestBucketFlushWhileFnDeployed got un-deployed
2021/12/03 22:28:55 No eventing-con process running
2021/12/03 22:28:55 Function: TestBucketFlushWhileFnDeployed delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestBucketFlushWhileFnDeployed not found"
 }
}
2021/12/03 22:28:57 Context: Bucket flush, response code: 200 dump: 
2021/12/03 22:29:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 22:29:04 Context: Bucket flush, response code: 200 dump: 
2021/12/03 22:29:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 22:29:09 Deploying app: TestBucketFlushWhileFnDeployed. Item count src bucket: 0 dst bucket: 0
2021/12/03 22:29:10 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestBucketFlushWhileFnDeployed' in metakv",
  "warnings": null
 }
}
2021/12/03 22:29:15 Waiting for app: TestBucketFlushWhileFnDeployed to get deployed
2021/12/03 22:29:20 Waiting for app: TestBucketFlushWhileFnDeployed to get deployed
2021/12/03 22:29:25 Waiting for app: TestBucketFlushWhileFnDeployed to get deployed
2021/12/03 22:29:30 Waiting for app: TestBucketFlushWhileFnDeployed to get deployed
2021/12/03 22:29:30 App: TestBucketFlushWhileFnDeployed got deployed
2021/12/03 22:29:30 Starting bucket ops to source bucket
2021/12/03 22:29:50 Waiting for dst bucket item count to get to: 20000 curr count: 10731
2021/12/03 22:29:55 Waiting for dst bucket item count to get to: 20000 curr count: 16035
2021/12/03 22:29:55 src & dst bucket item count matched up. src bucket count: 20000 dst bucket count: 20000
2021/12/03 22:29:55 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 22:29:58 Context: Bucket flush, response code: 200 dump: 
2021/12/03 22:30:08 Waiting for dst bucket item count to get to: 0 curr count: 11104
2021/12/03 22:30:13 Waiting for dst bucket item count to get to: 0 curr count: 1104
2021/12/03 22:30:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 22:30:15 Context: Bucket flush, response code: 200 dump: 
2021/12/03 22:30:25 Waiting for dst bucket item count to get to: 0 curr count: 20000
2021/12/03 22:30:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 22:30:25 Starting bucket ops to source bucket
2021/12/03 22:30:38 Waiting for dst bucket item count to get to: 10000 curr count: 3989
2021/12/03 22:30:43 Waiting for dst bucket item count to get to: 10000 curr count: 3989
2021/12/03 22:30:43 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/03 22:30:43 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 22:30:43 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 22:30:48 Waiting for app: TestBucketFlushWhileFnDeployed to get un-deployed
2021/12/03 22:30:53 Waiting for app: TestBucketFlushWhileFnDeployed to get un-deployed
2021/12/03 22:30:58 Waiting for app: TestBucketFlushWhileFnDeployed to get un-deployed
2021/12/03 22:31:03 Waiting for app: TestBucketFlushWhileFnDeployed to get un-deployed
2021/12/03 22:31:03 App: TestBucketFlushWhileFnDeployed got un-deployed
2021/12/03 22:31:03 No eventing-con process running
2021/12/03 22:31:03 Function: TestBucketFlushWhileFnDeployed delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestBucketFlushWhileFnDeployed deleting in the background"
}
2021/12/03 22:31:04 Context: Bucket flush, response code: 200 dump: 
2021/12/03 22:31:14 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/03 22:31:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 22:31:16 Context: Bucket flush, response code: 200 dump: 
2021/12/03 22:31:26 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/03 22:31:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestBucketFlushWhileFnDeployed (160.93s)
=== RUN   TestTimerOverwrite
2021/12/03 22:31:36 Waiting for app: TestTimerOverwrite to get un-deployed
2021/12/03 22:31:36 App: TestTimerOverwrite got un-deployed
2021/12/03 22:31:36 No eventing-con process running
2021/12/03 22:31:36 Function: TestTimerOverwrite delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestTimerOverwrite not found"
 }
}
2021/12/03 22:31:38 Context: Bucket flush, response code: 200 dump: 
2021/12/03 22:31:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 22:31:44 Context: Bucket flush, response code: 200 dump: 
2021/12/03 22:31:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 22:31:50 Deploying app: TestTimerOverwrite. Item count src bucket: 0 dst bucket: 0
2021/12/03 22:31:51 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestTimerOverwrite' in metakv",
  "warnings": null
 }
}
2021/12/03 22:31:56 Waiting for app: TestTimerOverwrite to get deployed
2021/12/03 22:32:01 Waiting for app: TestTimerOverwrite to get deployed
2021/12/03 22:32:06 Waiting for app: TestTimerOverwrite to get deployed
2021/12/03 22:32:11 Waiting for app: TestTimerOverwrite to get deployed
2021/12/03 22:32:11 App: TestTimerOverwrite got deployed
2021/12/03 22:32:21 Starting bucket ops to source bucket
2021/12/03 22:32:30 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 22:32:35 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 22:32:40 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 22:32:45 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 22:32:50 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 22:32:55 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 22:33:00 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 22:33:05 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 22:33:10 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 22:33:15 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 22:33:20 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 22:33:25 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 22:33:25 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/03 22:33:35 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 22:33:35 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 22:33:40 Waiting for app: TestTimerOverwrite to get un-deployed
2021/12/03 22:33:45 Waiting for app: TestTimerOverwrite to get un-deployed
2021/12/03 22:33:50 Waiting for app: TestTimerOverwrite to get un-deployed
2021/12/03 22:33:55 Waiting for app: TestTimerOverwrite to get un-deployed
2021/12/03 22:33:55 App: TestTimerOverwrite got un-deployed
2021/12/03 22:33:55 No eventing-con process running
2021/12/03 22:33:55 Function: TestTimerOverwrite delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestTimerOverwrite deleting in the background"
}
2021/12/03 22:33:57 Context: Bucket flush, response code: 200 dump: 
2021/12/03 22:34:07 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/03 22:34:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 22:34:09 Context: Bucket flush, response code: 200 dump: 
2021/12/03 22:34:19 Waiting for dst bucket item count to get to: 0 curr count: 4403
2021/12/03 22:34:24 Waiting for dst bucket item count to get to: 0 curr count: 1945
2021/12/03 22:34:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestTimerOverwrite (177.22s)
=== RUN   TestUndeployBackdoorDuringBootstrap
2021/12/03 22:34:29 Adding node: https://127.0.0.1:19003 with role: kv to the cluster
2021/12/03 22:34:29 Removing node(s): [] from the cluster
2021/12/03 22:34:34 Rebalance progress: 100
2021/12/03 22:34:34 Failing over: [127.0.0.1:19003]
2021/12/03 22:34:39 Waiting for app: TestUndeployBackdoorDuringBootstrap to get un-deployed
2021/12/03 22:34:39 App: TestUndeployBackdoorDuringBootstrap got un-deployed
2021/12/03 22:34:39 No eventing-con process running
2021/12/03 22:34:39 Function: TestUndeployBackdoorDuringBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestUndeployBackdoorDuringBootstrap not found"
 }
}
2021/12/03 22:34:40 Context: Bucket flush, response code: 200 dump: 
2021/12/03 22:34:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 22:34:47 Context: Bucket flush, response code: 200 dump: 
2021/12/03 22:34:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 22:34:52 Deploying app: TestUndeployBackdoorDuringBootstrap. Item count src bucket: 0 dst bucket: 0
2021/12/03 22:34:54 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestUndeployBackdoorDuringBootstrap' in metakv",
  "warnings": null
 }
}
2021/12/03 22:34:54 Starting bucket ops to source bucket
2021/12/03 22:35:04 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 22:35:04 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 22:35:04 Context: Post to set retry counter, response code: 200 dump: 
2021/12/03 22:35:04 set retry counter, response &{[] }
2021/12/03 22:36:09 Waiting for app: TestUndeployBackdoorDuringBootstrap to get un-deployed
2021/12/03 22:36:09 App: TestUndeployBackdoorDuringBootstrap got un-deployed
2021/12/03 22:36:09 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 22:36:09 Function: TestUndeployBackdoorDuringBootstrap delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestUndeployBackdoorDuringBootstrap deleting in the background"
}
2021/12/03 22:36:14 Waiting for app: TestUndeployBackdoorDuringBootstrap to get un-deployed
2021/12/03 22:36:14 App: TestUndeployBackdoorDuringBootstrap got un-deployed
2021/12/03 22:36:14 No eventing-con process running
2021/12/03 22:36:14 Function: TestUndeployBackdoorDuringBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestUndeployBackdoorDuringBootstrap not found"
 }
}
2021/12/03 22:36:15 Context: Bucket flush, response code: 200 dump: 
2021/12/03 22:36:25 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/03 22:36:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 22:36:27 Context: Bucket flush, response code: 200 dump: 
2021/12/03 22:36:37 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/03 22:36:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 22:36:37 Removing node(s): [] from the cluster
2021/12/03 22:36:42 Rebalance progress: 100
--- PASS: TestUndeployBackdoorDuringBootstrap (138.28s)
=== RUN   TestOnUpdateSrcMutation
2021/12/03 22:36:52 Waiting for app: TestOnUpdateSrcMutation to get un-deployed
2021/12/03 22:36:52 App: TestOnUpdateSrcMutation got un-deployed
2021/12/03 22:36:52 No eventing-con process running
2021/12/03 22:36:52 Function: TestOnUpdateSrcMutation delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateSrcMutation not found"
 }
}
2021/12/03 22:36:53 Context: Bucket flush, response code: 200 dump: 
2021/12/03 22:36:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 22:37:00 Context: Bucket flush, response code: 200 dump: 
2021/12/03 22:37:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 22:37:05 Deploying app: TestOnUpdateSrcMutation. Item count src bucket: 0 dst bucket: 0
2021/12/03 22:37:06 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateSrcMutation' in metakv",
  "warnings": null
 }
}
2021/12/03 22:37:06 Starting bucket ops to source bucket
2021/12/03 22:37:33 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/03 22:37:33 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 22:37:33 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 22:37:38 Waiting for app: TestOnUpdateSrcMutation to get un-deployed
2021/12/03 22:37:43 Waiting for app: TestOnUpdateSrcMutation to get un-deployed
2021/12/03 22:37:48 Waiting for app: TestOnUpdateSrcMutation to get un-deployed
2021/12/03 22:37:53 Waiting for app: TestOnUpdateSrcMutation to get un-deployed
2021/12/03 22:37:53 App: TestOnUpdateSrcMutation got un-deployed
2021/12/03 22:37:53 No eventing-con process running
2021/12/03 22:37:53 Function: TestOnUpdateSrcMutation delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateSrcMutation deleting in the background"
}
2021/12/03 22:37:55 Context: Bucket flush, response code: 200 dump: 
2021/12/03 22:38:05 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/03 22:38:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 22:38:07 Context: Bucket flush, response code: 200 dump: 
2021/12/03 22:38:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateSrcMutation (89.68s)
=== RUN   TestOnDeleteSrcMutation
2021/12/03 22:38:22 Waiting for app: TestOnDeleteSrcMutation to get un-deployed
2021/12/03 22:38:22 App: TestOnDeleteSrcMutation got un-deployed
2021/12/03 22:38:22 No eventing-con process running
2021/12/03 22:38:22 Function: TestOnDeleteSrcMutation delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnDeleteSrcMutation not found"
 }
}
2021/12/03 22:38:23 Context: Bucket flush, response code: 200 dump: 
2021/12/03 22:38:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 22:38:30 Context: Bucket flush, response code: 200 dump: 
2021/12/03 22:38:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 22:38:35 Deploying app: TestOnDeleteSrcMutation. Item count src bucket: 0 dst bucket: 0
2021/12/03 22:38:36 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnDeleteSrcMutation' in metakv",
  "warnings": null
 }
}
2021/12/03 22:38:36 Starting bucket ops to source bucket
2021/12/03 22:39:05 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/03 22:39:05 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 22:39:05 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 22:39:10 Waiting for app: TestOnDeleteSrcMutation to get un-deployed
2021/12/03 22:39:15 Waiting for app: TestOnDeleteSrcMutation to get un-deployed
2021/12/03 22:39:20 Waiting for app: TestOnDeleteSrcMutation to get un-deployed
2021/12/03 22:39:25 Waiting for app: TestOnDeleteSrcMutation to get un-deployed
2021/12/03 22:39:25 App: TestOnDeleteSrcMutation got un-deployed
2021/12/03 22:39:25 No eventing-con process running
2021/12/03 22:39:25 Function: TestOnDeleteSrcMutation delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnDeleteSrcMutation deleting in the background"
}
2021/12/03 22:39:26 Context: Bucket flush, response code: 200 dump: 
2021/12/03 22:39:36 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/03 22:39:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 22:39:38 Context: Bucket flush, response code: 200 dump: 
2021/12/03 22:39:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnDeleteSrcMutation (91.50s)
=== RUN   TestOnUpdateSrcMutationWithTimer
2021/12/03 22:39:53 Waiting for app: TestOnUpdateSrcMutationWithTimer to get un-deployed
2021/12/03 22:39:53 App: TestOnUpdateSrcMutationWithTimer got un-deployed
2021/12/03 22:39:53 No eventing-con process running
2021/12/03 22:39:53 Function: TestOnUpdateSrcMutationWithTimer delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateSrcMutationWithTimer not found"
 }
}
2021/12/03 22:39:54 Context: Bucket flush, response code: 200 dump: 
2021/12/03 22:39:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 22:40:01 Context: Bucket flush, response code: 200 dump: 
2021/12/03 22:40:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 22:40:06 Deploying app: TestOnUpdateSrcMutationWithTimer. Item count src bucket: 0 dst bucket: 0
2021/12/03 22:40:07 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateSrcMutationWithTimer' in metakv",
  "warnings": null
 }
}
2021/12/03 22:40:07 Starting bucket ops to source bucket
2021/12/03 22:40:44 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/03 22:40:44 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 22:40:45 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 22:40:50 Waiting for app: TestOnUpdateSrcMutationWithTimer to get un-deployed
2021/12/03 22:40:55 Waiting for app: TestOnUpdateSrcMutationWithTimer to get un-deployed
2021/12/03 22:41:00 Waiting for app: TestOnUpdateSrcMutationWithTimer to get un-deployed
2021/12/03 22:41:05 Waiting for app: TestOnUpdateSrcMutationWithTimer to get un-deployed
2021/12/03 22:41:05 App: TestOnUpdateSrcMutationWithTimer got un-deployed
2021/12/03 22:41:05 No eventing-con process running
2021/12/03 22:41:05 Function: TestOnUpdateSrcMutationWithTimer delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateSrcMutationWithTimer deleting in the background"
}
2021/12/03 22:41:06 Context: Bucket flush, response code: 200 dump: 
2021/12/03 22:41:16 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/03 22:41:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 22:41:18 Context: Bucket flush, response code: 200 dump: 
2021/12/03 22:41:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateSrcMutationWithTimer (99.66s)
=== RUN   TestOnDeleteSrcMutationsWithTimer
2021/12/03 22:41:33 Waiting for app: TestOnDeleteSrcMutationsWithTimer to get un-deployed
2021/12/03 22:41:33 App: TestOnDeleteSrcMutationsWithTimer got un-deployed
2021/12/03 22:41:33 No eventing-con process running
2021/12/03 22:41:33 Function: TestOnDeleteSrcMutationsWithTimer delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnDeleteSrcMutationsWithTimer not found"
 }
}
2021/12/03 22:41:34 Context: Bucket flush, response code: 200 dump: 
2021/12/03 22:41:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 22:41:41 Context: Bucket flush, response code: 200 dump: 
2021/12/03 22:41:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 22:41:46 Deploying app: TestOnDeleteSrcMutationsWithTimer. Item count src bucket: 0 dst bucket: 0
2021/12/03 22:41:47 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnDeleteSrcMutationsWithTimer' in metakv",
  "warnings": null
 }
}
2021/12/03 22:41:47 Starting bucket ops to source bucket
2021/12/03 22:42:26 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/03 22:42:26 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 22:42:26 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 22:42:31 Waiting for app: TestOnDeleteSrcMutationsWithTimer to get un-deployed
2021/12/03 22:42:36 Waiting for app: TestOnDeleteSrcMutationsWithTimer to get un-deployed
2021/12/03 22:42:41 Waiting for app: TestOnDeleteSrcMutationsWithTimer to get un-deployed
2021/12/03 22:42:46 Waiting for app: TestOnDeleteSrcMutationsWithTimer to get un-deployed
2021/12/03 22:42:46 App: TestOnDeleteSrcMutationsWithTimer got un-deployed
2021/12/03 22:42:46 No eventing-con process running
2021/12/03 22:42:46 Function: TestOnDeleteSrcMutationsWithTimer delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnDeleteSrcMutationsWithTimer deleting in the background"
}
2021/12/03 22:42:47 Context: Bucket flush, response code: 200 dump: 
2021/12/03 22:42:57 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/03 22:42:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 22:42:59 Context: Bucket flush, response code: 200 dump: 
2021/12/03 22:43:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnDeleteSrcMutationsWithTimer (101.39s)
=== RUN   TestInterHandlerRecursion
2021/12/03 22:43:14 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/03 22:43:14 App: TestInterHandlerRecursion_source_bucket_update_op got un-deployed
2021/12/03 22:43:14 No eventing-con process running
2021/12/03 22:43:14 Function: TestInterHandlerRecursion_source_bucket_update_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestInterHandlerRecursion_source_bucket_update_op not found"
 }
}
2021/12/03 22:43:15 Context: Bucket flush, response code: 200 dump: 
2021/12/03 22:43:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 22:43:22 Context: Bucket flush, response code: 200 dump: 
2021/12/03 22:43:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 22:43:32 Waiting for app: TestInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/03 22:43:32 App: TestInterHandlerRecursion_src_bucket_op_on_delete got un-deployed
2021/12/03 22:43:32 No eventing-con process running
2021/12/03 22:43:32 Function: TestInterHandlerRecursion_src_bucket_op_on_delete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestInterHandlerRecursion_src_bucket_op_on_delete not found"
 }
}
2021/12/03 22:43:34 Context: Bucket flush, response code: 200 dump: 
2021/12/03 22:43:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 22:43:40 Context: Bucket flush, response code: 200 dump: 
2021/12/03 22:43:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 22:43:45 Deploying app: TestInterHandlerRecursion_source_bucket_update_op. Item count src bucket: 0 dst bucket: 0
2021/12/03 22:43:47 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterHandlerRecursion_source_bucket_update_op' in metakv",
  "warnings": null
 }
}
2021/12/03 22:43:47 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterHandlerRecursion_source_bucket_update_op' in metakv",
  "warnings": null
 }
} err 
2021/12/03 22:43:52 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/03 22:43:57 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/03 22:44:02 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/03 22:44:07 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/03 22:44:07 App: TestInterHandlerRecursion_source_bucket_update_op got deployed
2021/12/03 22:44:07 Deploying app: TestInterHandlerRecursion_src_bucket_op_on_delete. Item count src bucket: 0 dst bucket: 0
2021/12/03 22:44:07 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_INTER_FUNCTION_RECURSION",
 "code": 50,
 "description": "Inter function recursion error, only one function is allowed to do source bucket mutation/delete on a bucket",
 "attributes": null,
 "runtime_info": {
  "code": 50,
  "info": "Inter handler recursion error"
 }
}
2021/12/03 22:44:07 response body {
 "name": "ERR_INTER_FUNCTION_RECURSION",
 "code": 50,
 "description": "Inter function recursion error, only one function is allowed to do source bucket mutation/delete on a bucket",
 "attributes": null,
 "runtime_info": {
  "code": 50,
  "info": "Inter handler recursion error"
 }
} err 
2021/12/03 22:44:07 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 22:44:12 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/03 22:44:17 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/03 22:44:22 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/03 22:44:27 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/03 22:44:27 App: TestInterHandlerRecursion_source_bucket_update_op got un-deployed
2021/12/03 22:44:27 No eventing-con process running
2021/12/03 22:44:27 Function: TestInterHandlerRecursion_source_bucket_update_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestInterHandlerRecursion_source_bucket_update_op deleting in the background"
}
2021/12/03 22:44:28 Context: Bucket flush, response code: 200 dump: 
2021/12/03 22:44:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 22:44:35 Context: Bucket flush, response code: 200 dump: 
2021/12/03 22:44:40 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 22:44:45 Waiting for app: TestInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/03 22:44:45 App: TestInterHandlerRecursion_src_bucket_op_on_delete got un-deployed
2021/12/03 22:44:45 No eventing-con process running
2021/12/03 22:44:45 Function: TestInterHandlerRecursion_src_bucket_op_on_delete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestInterHandlerRecursion_src_bucket_op_on_delete not found"
 }
}
2021/12/03 22:44:46 Context: Bucket flush, response code: 200 dump: 
2021/12/03 22:44:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 22:44:53 Context: Bucket flush, response code: 200 dump: 
2021/12/03 22:44:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestInterHandlerRecursion (114.13s)
=== RUN   TestInterBucketRecursion
2021/12/03 22:45:03 Deploying app: TestInterBucketRecursion_function1. Item count src bucket: 0 dst bucket: 0
2021/12/03 22:45:04 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterBucketRecursion_function1' in metakv",
  "warnings": null
 }
}
2021/12/03 22:45:04 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterBucketRecursion_function1' in metakv",
  "warnings": null
 }
} err 
2021/12/03 22:45:04 Deploying app: TestInterBucketRecursion_function2. Item count src bucket: 0 dst bucket: 0
2021/12/03 22:45:04 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_INTER_BUCKET_RECURSION",
 "code": 51,
 "description": "Inter bucket recursion error, deployment of current handler will cause inter bucket recursion",
 "attributes": null,
 "runtime_info": {
  "code": 51,
  "info": "Inter bucket recursion error; function: TestInterBucketRecursion_function2 causes a cycle involving functions: [TestInterBucketRecursion_function1], hence deployment is disallowed"
 }
}
2021/12/03 22:45:09 Waiting for app: TestInterBucketRecursion_function1 to get deployed
2021/12/03 22:45:14 Waiting for app: TestInterBucketRecursion_function1 to get deployed
2021/12/03 22:45:19 Waiting for app: TestInterBucketRecursion_function1 to get deployed
2021/12/03 22:45:24 Waiting for app: TestInterBucketRecursion_function1 to get deployed
2021/12/03 22:45:24 App: TestInterBucketRecursion_function1 got deployed
2021/12/03 22:45:25 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 22:45:30 Waiting for app: TestInterBucketRecursion_function1 to get un-deployed
2021/12/03 22:45:35 Waiting for app: TestInterBucketRecursion_function1 to get un-deployed
2021/12/03 22:45:40 Waiting for app: TestInterBucketRecursion_function1 to get un-deployed
2021/12/03 22:45:45 Waiting for app: TestInterBucketRecursion_function1 to get un-deployed
2021/12/03 22:45:45 App: TestInterBucketRecursion_function1 got un-deployed
2021/12/03 22:45:45 No eventing-con process running
2021/12/03 22:45:45 Function: TestInterBucketRecursion_function1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestInterBucketRecursion_function1 deleting in the background"
}
2021/12/03 22:45:46 Context: Bucket flush, response code: 200 dump: 
2021/12/03 22:45:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 22:45:53 Context: Bucket flush, response code: 200 dump: 
2021/12/03 22:45:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 22:46:03 Waiting for app: TestInterBucketRecursion_function2 to get un-deployed
2021/12/03 22:46:03 App: TestInterBucketRecursion_function2 got un-deployed
2021/12/03 22:46:03 No eventing-con process running
2021/12/03 22:46:03 Function: TestInterBucketRecursion_function2 delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestInterBucketRecursion_function2 not found"
 }
}
2021/12/03 22:46:04 Context: Bucket flush, response code: 200 dump: 
2021/12/03 22:46:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 22:46:11 Context: Bucket flush, response code: 200 dump: 
2021/12/03 22:46:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestInterBucketRecursion (77.67s)
=== RUN   TestLargeHandler
2021/12/03 22:46:16 Changed force_compress value to false
2021/12/03 22:46:16 Deploying app: TestLargeHandler_function. Item count src bucket: 0 dst bucket: 0
2021/12/03 22:46:16 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_APPCODE_SIZE",
 "code": 39,
 "description": "Function Code size is more than the configured limit",
 "attributes": null,
 "runtime_info": {
  "code": 39,
  "info": "Function: TestLargeHandler_function handler Code size is more than 131072. Code Size: 131768"
 }
}
2021/12/03 22:46:16 Changed force_compress value to true
2021/12/03 22:46:16 Deploying app: TestLargeHandler_function. Item count src bucket: 0 dst bucket: 0
2021/12/03 22:46:18 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestLargeHandler_function' in metakv",
  "warnings": null
 }
}
2021/12/03 22:46:23 Waiting for app: TestLargeHandler_function to get deployed
2021/12/03 22:46:28 Waiting for app: TestLargeHandler_function to get deployed
2021/12/03 22:46:33 Waiting for app: TestLargeHandler_function to get deployed
2021/12/03 22:46:38 Waiting for app: TestLargeHandler_function to get deployed
2021/12/03 22:46:38 App: TestLargeHandler_function got deployed
2021/12/03 22:46:38 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 22:46:43 Waiting for app: TestLargeHandler_function to get un-deployed
2021/12/03 22:46:48 Waiting for app: TestLargeHandler_function to get un-deployed
2021/12/03 22:46:53 Waiting for app: TestLargeHandler_function to get un-deployed
2021/12/03 22:46:58 Waiting for app: TestLargeHandler_function to get un-deployed
2021/12/03 22:46:58 App: TestLargeHandler_function got un-deployed
2021/12/03 22:46:58 No eventing-con process running
2021/12/03 22:46:58 Function: TestLargeHandler_function delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestLargeHandler_function deleting in the background"
}
2021/12/03 22:46:59 Context: Bucket flush, response code: 200 dump: 
2021/12/03 22:47:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 22:47:06 Context: Bucket flush, response code: 200 dump: 
2021/12/03 22:47:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestLargeHandler (55.38s)
=== RUN   TestAllowInterHandlerRecursion
2021/12/03 22:47:21 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/03 22:47:21 App: TestAllowInterHandlerRecursion_source_bucket_update_op got un-deployed
2021/12/03 22:47:21 No eventing-con process running
2021/12/03 22:47:21 Function: TestAllowInterHandlerRecursion_source_bucket_update_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestAllowInterHandlerRecursion_source_bucket_update_op not found"
 }
}
2021/12/03 22:47:23 Context: Bucket flush, response code: 200 dump: 
2021/12/03 22:47:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 22:47:29 Context: Bucket flush, response code: 200 dump: 
2021/12/03 22:47:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 22:47:39 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/03 22:47:39 App: TestAllowInterHandlerRecursion_src_bucket_op_on_delete got un-deployed
2021/12/03 22:47:39 No eventing-con process running
2021/12/03 22:47:39 Function: TestAllowInterHandlerRecursion_src_bucket_op_on_delete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestAllowInterHandlerRecursion_src_bucket_op_on_delete not found"
 }
}
2021/12/03 22:47:41 Context: Bucket flush, response code: 200 dump: 
2021/12/03 22:47:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 22:47:48 Context: Bucket flush, response code: 200 dump: 
2021/12/03 22:47:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 22:47:53 Deploying app: TestAllowInterHandlerRecursion_source_bucket_update_op. Item count src bucket: 0 dst bucket: 0
2021/12/03 22:47:54 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterHandlerRecursion_source_bucket_update_op' in metakv",
  "warnings": null
 }
}
2021/12/03 22:47:54 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterHandlerRecursion_source_bucket_update_op' in metakv",
  "warnings": null
 }
} err 
2021/12/03 22:47:59 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/03 22:48:04 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/03 22:48:09 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/03 22:48:14 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/03 22:48:14 App: TestAllowInterHandlerRecursion_source_bucket_update_op got deployed
2021/12/03 22:48:14 Deploying app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete. Item count src bucket: 0 dst bucket: 0
2021/12/03 22:48:15 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterHandlerRecursion_src_bucket_op_on_delete' in metakv",
  "warnings": null
 }
}
2021/12/03 22:48:15 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterHandlerRecursion_src_bucket_op_on_delete' in metakv",
  "warnings": null
 }
} err 
2021/12/03 22:48:20 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get deployed
2021/12/03 22:48:25 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get deployed
2021/12/03 22:48:30 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get deployed
2021/12/03 22:48:35 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get deployed
2021/12/03 22:48:35 App: TestAllowInterHandlerRecursion_src_bucket_op_on_delete got deployed
2021/12/03 22:48:35 TestAllowInterHandlerRecursion_src_bucket_op_on_delete is deployed
2021/12/03 22:48:35 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 22:48:40 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/03 22:48:45 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/03 22:48:50 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/03 22:48:55 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/03 22:48:55 App: TestAllowInterHandlerRecursion_source_bucket_update_op got un-deployed
2021/12/03 22:48:55 No eventing-con process running
2021/12/03 22:48:55 Function: TestAllowInterHandlerRecursion_source_bucket_update_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAllowInterHandlerRecursion_source_bucket_update_op deleting in the background"
}
2021/12/03 22:48:58 Context: Bucket flush, response code: 200 dump: 
2021/12/03 22:49:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 22:49:04 Context: Bucket flush, response code: 200 dump: 
2021/12/03 22:49:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 22:49:10 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 22:49:15 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/03 22:49:20 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/03 22:49:25 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/03 22:49:30 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/03 22:49:30 App: TestAllowInterHandlerRecursion_src_bucket_op_on_delete got un-deployed
2021/12/03 22:49:30 No eventing-con process running
2021/12/03 22:49:30 Function: TestAllowInterHandlerRecursion_src_bucket_op_on_delete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAllowInterHandlerRecursion_src_bucket_op_on_delete deleting in the background"
}
2021/12/03 22:49:31 Context: Bucket flush, response code: 200 dump: 
2021/12/03 22:49:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 22:49:38 Context: Bucket flush, response code: 200 dump: 
2021/12/03 22:49:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAllowInterHandlerRecursion (151.52s)
=== RUN   TestAllowInterBucketRecursion
2021/12/03 22:49:48 Deploying app: TestAllowInterBucketRecursion_function1. Item count src bucket: 0 dst bucket: 0
2021/12/03 22:49:49 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterBucketRecursion_function1' in metakv",
  "warnings": null
 }
}
2021/12/03 22:49:49 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterBucketRecursion_function1' in metakv",
  "warnings": null
 }
} err 
2021/12/03 22:49:54 Waiting for app: TestAllowInterBucketRecursion_function1 to get deployed
2021/12/03 22:49:59 Waiting for app: TestAllowInterBucketRecursion_function1 to get deployed
2021/12/03 22:50:04 Waiting for app: TestAllowInterBucketRecursion_function1 to get deployed
2021/12/03 22:50:09 Waiting for app: TestAllowInterBucketRecursion_function1 to get deployed
2021/12/03 22:50:09 App: TestAllowInterBucketRecursion_function1 got deployed
2021/12/03 22:50:09 Deploying app: TestAllowInterBucketRecursion_function2. Item count src bucket: 0 dst bucket: 0
2021/12/03 22:50:10 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterBucketRecursion_function2' in metakv",
  "warnings": null
 }
}
2021/12/03 22:50:10 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterBucketRecursion_function2' in metakv",
  "warnings": null
 }
} err 
2021/12/03 22:50:15 Waiting for app: TestAllowInterBucketRecursion_function2 to get deployed
2021/12/03 22:50:20 Waiting for app: TestAllowInterBucketRecursion_function2 to get deployed
2021/12/03 22:50:25 Waiting for app: TestAllowInterBucketRecursion_function2 to get deployed
2021/12/03 22:50:30 Waiting for app: TestAllowInterBucketRecursion_function2 to get deployed
2021/12/03 22:50:30 App: TestAllowInterBucketRecursion_function2 got deployed
2021/12/03 22:50:30 TestAllowInterBucketRecursion_function2 is deployed
2021/12/03 22:50:30 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 22:50:35 Waiting for app: TestAllowInterBucketRecursion_function1 to get un-deployed
2021/12/03 22:50:40 Waiting for app: TestAllowInterBucketRecursion_function1 to get un-deployed
2021/12/03 22:50:45 Waiting for app: TestAllowInterBucketRecursion_function1 to get un-deployed
2021/12/03 22:50:50 Waiting for app: TestAllowInterBucketRecursion_function1 to get un-deployed
2021/12/03 22:50:50 App: TestAllowInterBucketRecursion_function1 got un-deployed
2021/12/03 22:50:50 No eventing-con process running
2021/12/03 22:50:50 Function: TestAllowInterBucketRecursion_function1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAllowInterBucketRecursion_function1 deleting in the background"
}
2021/12/03 22:50:52 Context: Bucket flush, response code: 200 dump: 
2021/12/03 22:50:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 22:51:00 Context: Bucket flush, response code: 200 dump: 
2021/12/03 22:51:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 22:51:05 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 22:51:10 Waiting for app: TestAllowInterBucketRecursion_function2 to get un-deployed
2021/12/03 22:51:15 Waiting for app: TestAllowInterBucketRecursion_function2 to get un-deployed
2021/12/03 22:51:20 Waiting for app: TestAllowInterBucketRecursion_function2 to get un-deployed
2021/12/03 22:51:25 Waiting for app: TestAllowInterBucketRecursion_function2 to get un-deployed
2021/12/03 22:51:25 App: TestAllowInterBucketRecursion_function2 got un-deployed
2021/12/03 22:51:25 No eventing-con process running
2021/12/03 22:51:25 Function: TestAllowInterBucketRecursion_function2 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAllowInterBucketRecursion_function2 deleting in the background"
}
2021/12/03 22:51:27 Context: Bucket flush, response code: 200 dump: 
2021/12/03 22:51:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 22:51:33 Context: Bucket flush, response code: 200 dump: 
2021/12/03 22:51:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAllowInterBucketRecursion (115.58s)
=== RUN   TestInterBucketRecursion2
2021/12/03 22:51:45 Context: Bucket flush, response code: 200 dump: 
2021/12/03 22:51:51 Context: Bucket flush, response code: 200 dump: 
2021/12/03 22:51:57 Context: Bucket flush, response code: 200 dump: 
2021/12/03 22:52:03 Context: Bucket flush, response code: 200 dump: 
2021/12/03 22:52:08 Deploying app: TestInterBucketRecursion2_function1. Item count src bucket: 0 dst bucket: 0
2021/12/03 22:52:09 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterBucketRecursion2_function1' in metakv",
  "warnings": null
 }
}
2021/12/03 22:52:14 Waiting for app: TestInterBucketRecursion2_function1 to get deployed
2021/12/03 22:52:20 Waiting for app: TestInterBucketRecursion2_function1 to get deployed
2021/12/03 22:52:25 Waiting for app: TestInterBucketRecursion2_function1 to get deployed
2021/12/03 22:52:30 Waiting for app: TestInterBucketRecursion2_function1 to get deployed
2021/12/03 22:52:30 App: TestInterBucketRecursion2_function1 got deployed
2021/12/03 22:52:30 Deploying app: TestInterBucketRecursion2_function2. Item count src bucket: 0 dst bucket: 0
2021/12/03 22:52:31 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterBucketRecursion2_function2' in metakv",
  "warnings": null
 }
}
2021/12/03 22:52:36 Waiting for app: TestInterBucketRecursion2_function2 to get deployed
2021/12/03 22:52:41 Waiting for app: TestInterBucketRecursion2_function2 to get deployed
2021/12/03 22:52:46 Waiting for app: TestInterBucketRecursion2_function2 to get deployed
2021/12/03 22:52:51 Waiting for app: TestInterBucketRecursion2_function2 to get deployed
2021/12/03 22:52:51 App: TestInterBucketRecursion2_function2 got deployed
2021/12/03 22:52:51 Deploying app: TestInterBucketRecursion2_function3. Item count src bucket: 0 dst bucket: 0
2021/12/03 22:52:52 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterBucketRecursion2_function3' in metakv",
  "warnings": null
 }
}
2021/12/03 22:52:57 Waiting for app: TestInterBucketRecursion2_function3 to get deployed
2021/12/03 22:53:02 Waiting for app: TestInterBucketRecursion2_function3 to get deployed
2021/12/03 22:53:07 Waiting for app: TestInterBucketRecursion2_function3 to get deployed
2021/12/03 22:53:12 Waiting for app: TestInterBucketRecursion2_function3 to get deployed
2021/12/03 22:53:12 App: TestInterBucketRecursion2_function3 got deployed
2021/12/03 22:53:12 Deploying app: TestInterBucketRecursion2_function4. Item count src bucket: 0 dst bucket: 0
2021/12/03 22:53:12 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_INTER_BUCKET_RECURSION",
 "code": 51,
 "description": "Inter bucket recursion error, deployment of current handler will cause inter bucket recursion",
 "attributes": null,
 "runtime_info": {
  "code": 51,
  "info": "Inter bucket recursion error; function: TestInterBucketRecursion2_function4 causes a cycle involving functions: [TestInterBucketRecursion2_function1 TestInterBucketRecursion2_function2 TestInterBucketRecursion2_function3], hence deployment is disallowed"
 }
}
2021/12/03 22:53:12 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 22:53:17 Waiting for app: TestInterBucketRecursion2_function1 to get un-deployed
2021/12/03 22:53:22 Waiting for app: TestInterBucketRecursion2_function1 to get un-deployed
2021/12/03 22:53:27 Waiting for app: TestInterBucketRecursion2_function1 to get un-deployed
2021/12/03 22:53:32 Waiting for app: TestInterBucketRecursion2_function1 to get un-deployed
2021/12/03 22:53:32 App: TestInterBucketRecursion2_function1 got un-deployed
2021/12/03 22:53:32 No eventing-con process running
2021/12/03 22:53:32 Function: TestInterBucketRecursion2_function1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestInterBucketRecursion2_function1 deleting in the background"
}
2021/12/03 22:53:34 Context: Bucket flush, response code: 200 dump: 
2021/12/03 22:53:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 22:53:41 Context: Bucket flush, response code: 200 dump: 
2021/12/03 22:53:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 22:53:46 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 22:53:51 Waiting for app: TestInterBucketRecursion2_function2 to get un-deployed
2021/12/03 22:53:56 Waiting for app: TestInterBucketRecursion2_function2 to get un-deployed
2021/12/03 22:54:01 Waiting for app: TestInterBucketRecursion2_function2 to get un-deployed
2021/12/03 22:54:06 Waiting for app: TestInterBucketRecursion2_function2 to get un-deployed
2021/12/03 22:54:06 App: TestInterBucketRecursion2_function2 got un-deployed
2021/12/03 22:54:06 No eventing-con process running
2021/12/03 22:54:06 Function: TestInterBucketRecursion2_function2 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestInterBucketRecursion2_function2 deleting in the background"
}
2021/12/03 22:54:07 Context: Bucket flush, response code: 200 dump: 
2021/12/03 22:54:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 22:54:14 Context: Bucket flush, response code: 200 dump: 
2021/12/03 22:54:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 22:54:19 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 22:54:24 Waiting for app: TestInterBucketRecursion2_function3 to get un-deployed
2021/12/03 22:54:29 Waiting for app: TestInterBucketRecursion2_function3 to get un-deployed
2021/12/03 22:54:34 Waiting for app: TestInterBucketRecursion2_function3 to get un-deployed
2021/12/03 22:54:39 Waiting for app: TestInterBucketRecursion2_function3 to get un-deployed
2021/12/03 22:54:39 App: TestInterBucketRecursion2_function3 got un-deployed
2021/12/03 22:54:39 No eventing-con process running
2021/12/03 22:54:39 Function: TestInterBucketRecursion2_function3 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestInterBucketRecursion2_function3 deleting in the background"
}
2021/12/03 22:54:41 Context: Bucket flush, response code: 200 dump: 
2021/12/03 22:54:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 22:54:48 Context: Bucket flush, response code: 200 dump: 
2021/12/03 22:54:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 22:54:58 Waiting for app: TestInterBucketRecursion2_function4 to get un-deployed
2021/12/03 22:54:58 App: TestInterBucketRecursion2_function4 got un-deployed
2021/12/03 22:54:58 No eventing-con process running
2021/12/03 22:54:58 Function: TestInterBucketRecursion2_function4 delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestInterBucketRecursion2_function4 not found"
 }
}
2021/12/03 22:54:59 Context: Bucket flush, response code: 200 dump: 
2021/12/03 22:55:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 22:55:06 Context: Bucket flush, response code: 200 dump: 
2021/12/03 22:55:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestInterBucketRecursion2 (214.57s)
=== RUN   TestBucketDeleteWithRebOut
2021/12/03 22:55:18 Removing node(s): [127.0.0.1:19001 127.0.0.1:19002 127.0.0.1:19003] from the cluster
2021/12/03 22:55:23 Rebalance progress: 100

VB distribution across eventing nodes and their workers::

2021/12/03 22:55:26 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:52} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/03 22:55:41 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:52} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/03 22:55:56 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:52} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/03 22:56:11 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:52} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/03 22:56:26 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:52} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/03 22:56:41 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:52} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/03 22:56:56 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:52} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/03 22:57:11 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:52} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/03 22:57:26 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:52} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/03 22:57:41 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:52} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/03 22:57:56 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:52} {Name:#primary Bucket:hello-world Completion:100}]

DCP Stream statuses:

vbucket curr owner:


2021/12/03 22:58:11 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 22:58:11 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/03 22:58:11 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/03 22:58:16 Rebalance progress: 100

VB distribution across eventing nodes and their workers::


DCP Stream statuses:

vbucket curr owner:


2021/12/03 22:58:16 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 22:58:16 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/03 22:58:17 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/03 22:58:22 Rebalance progress: 100

VB distribution across eventing nodes and their workers::


DCP Stream statuses:

vbucket curr owner:


2021/12/03 22:58:22 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 22:58:27 clusterSetup.go:397: Waiting for app: TestBucketDeleteWithRebOut to get un-deployed
2021/12/03 22:58:27 clusterSetup.go:405: App: TestBucketDeleteWithRebOut got un-deployed
2021/12/03 22:58:27 eventingSetup.go:46: No eventing-con process running
2021/12/03 22:58:27 eventingSetup.go:418: Function: TestBucketDeleteWithRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestBucketDeleteWithRebOut not found"
 }
}
2021/12/03 22:58:28 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/03 22:58:33 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 22:58:35 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/03 22:58:40 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 22:58:45 eventingSetup.go:125: Deploying app: TestBucketDeleteWithRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/03 22:58:46 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestBucketDeleteWithRebOut' in metakv",
  "warnings": null
 }
}
2021/12/03 22:58:51 clusterSetup.go:346: Waiting for app: TestBucketDeleteWithRebOut to get deployed
2021/12/03 22:58:56 clusterSetup.go:346: Waiting for app: TestBucketDeleteWithRebOut to get deployed
2021/12/03 22:59:01 clusterSetup.go:346: Waiting for app: TestBucketDeleteWithRebOut to get deployed
2021/12/03 22:59:06 clusterSetup.go:346: Waiting for app: TestBucketDeleteWithRebOut to get deployed
2021/12/03 22:59:06 clusterSetup.go:354: App: TestBucketDeleteWithRebOut got deployed

VB distribution across eventing nodes and their workers::


DCP Stream statuses:
	stream status: running
	len: 1024
	vb list dump: [0-1023]

vbucket curr owner:
Producer node: 127.0.0.1:9302	Node UUID: 8f4040b40e22991ccc6417cab9eed230
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [0-113]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [114-227]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 114
	v: [228-341]
Producer node: 172.31.2.134:9300	Node UUID: 17ef4d05f9a1c14fa6d6f1f8b465e112
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 113
	v: [911-1023]
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [683-796]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [797-910]
Producer node: 127.0.0.1:9303	Node UUID: 3bdd07c6c33df1dbf7c4f258364e903b
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [342-455]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [456-569]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 113
	v: [570-682]


2021/12/03 22:59:06 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 22:59:06 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/03 22:59:11 Rebalance progress: 29.16666666666666
2021/12/03 22:59:16 Rebalance progress: 29.16666666666666
2021/12/03 22:59:21 Rebalance progress: 100

VB distribution across eventing nodes and their workers::


DCP Stream statuses:
	stream status: running
	len: 1024
	vb list dump: [0-1023]

vbucket curr owner:
Producer node: 172.31.2.134:9300	Node UUID: 17ef4d05f9a1c14fa6d6f1f8b465e112
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 113
	v: [911-1023]
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [683-796]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [797-910]
Producer node: 127.0.0.1:9303	Node UUID: 3bdd07c6c33df1dbf7c4f258364e903b
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 113
	v: [570-682]
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [342-455]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [456-569]
Producer node: 127.0.0.1:9302	Node UUID: 8f4040b40e22991ccc6417cab9eed230
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [0-113]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [114-227]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 114
	v: [228-341]


2021/12/03 22:59:21 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 22:59:21 handler_test.go:1674: Deleting source bucket: default
2021/12/03 22:59:23 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/03 22:59:24 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/03 22:59:29 Rebalance progress: 29.16666666666666
2021/12/03 22:59:34 Rebalance progress: 29.16666666666666
2021/12/03 22:59:39 Rebalance progress: 100

VB distribution across eventing nodes and their workers::


DCP Stream statuses:
	stream status: running
	len: 898
	vb list dump: [0-12, 15-33, 35, 37-45, 48-50, 52-74, 76-81, 83-93, 95-96, 98-103, 106-123, 125-131, 134-138, 140-155, 157-169, 171-174, 176-181, 183-191, 193-195, 197-208, 210-217, 219-220, 222, 224-226, 228-234, 236-245, 247, 249-280, 283, 285, 287-296, 298-308, 311-312, 314-320, 322-327, 329-330, 332-343, 346-377, 380-383, 385, 387-393, 395, 397-402, 404-405, 407-413, 415-423, 425-431, 434-450, 452-456, 458-459, 461-474, 476-477, 480-484, 486-491, 493, 496-500, 504-510, 512-584, 586-588, 590-592, 594-601, 603-607, 609-617, 619-621, 623-631, 633-635, 637, 639-640, 642-652, 654-671, 673-678, 680-690, 692-693, 695-719, 721-726, 728-733, 735-738, 740-743, 745-748, 750, 752-755, 757-766, 768-776, 778-787, 789-796, 798, 800-805, 807-813, 815-824, 826-829, 831-835, 837-839, 841-848, 850-854, 856-863, 865-867, 869-870, 872-875, 878-887, 889-902, 904-926, 928-929, 932-933, 935-939, 941-945, 947-961, 963-968, 970-982, 984-989, 991-1011, 1013-1019, 1021-1023]
	stream status: stopped
	len: 126
	vb list dump: [13-14, 34, 36, 46-47, 51, 75, 82, 94, 97, 104-105, 124, 132-133, 139, 156, 170, 175, 182, 192, 196, 209, 218, 221, 223, 227, 235, 246, 248, 281-282, 284, 286, 297, 309-310, 313, 321, 328, 331, 344-345, 378-379, 384, 386, 394, 396, 403, 406, 414, 424, 432-433, 451, 457, 460, 475, 478-479, 485, 492, 494-495, 501-503, 511, 585, 589, 593, 602, 608, 618, 622, 632, 636, 638, 641, 653, 672, 679, 691, 694, 720, 727, 734, 739, 744, 749, 751, 756, 767, 777, 788, 797, 799, 806, 814, 825, 830, 836, 840, 849, 855, 864, 868, 871, 876-877, 888, 903, 927, 930-931, 934, 940, 946, 962, 969, 983, 990, 1012, 1020]

vbucket curr owner:
Producer node: 127.0.0.1:9303	Node UUID: 3bdd07c6c33df1dbf7c4f258364e903b
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 99
	v: [342-343, 346-377, 380-383, 385, 387-393, 395, 397-402, 404-405, 407-413, 415-423, 425-431, 434-450, 452-455]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 101
	v: [456, 458-459, 461-474, 476-477, 480-484, 486-491, 493, 496-500, 504-510, 512-569]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 99
	v: [570-584, 586-588, 590-592, 594-601, 603-607, 609-617, 619-621, 623-631, 633-635, 637, 639-640, 642-652, 654-671, 673-678, 680-682]
Producer node: 127.0.0.1:9302	Node UUID: 8f4040b40e22991ccc6417cab9eed230
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 101
	v: [0-12, 15-33, 35, 37-45, 48-50, 52-74, 76-81, 83-93, 95-96, 98-103, 106-113]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 99
	v: [114-123, 125-131, 134-138, 140-155, 157-169, 171-174, 176-181, 183-191, 193-195, 197-208, 210-217, 219-220, 222, 224-226]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 100
	v: [228-234, 236-245, 247, 249-280, 283, 285, 287-296, 298-308, 311-312, 314-320, 322-327, 329-330, 332-341]
Producer node: 172.31.2.134:9300	Node UUID: 17ef4d05f9a1c14fa6d6f1f8b465e112
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 101
	v: [911-926, 928-929, 932-933, 935-939, 941-945, 947-961, 963-968, 970-982, 984-989, 991-1011, 1013-1019, 1021-1023]
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 101
	v: [683-690, 692-693, 695-719, 721-726, 728-733, 735-738, 740-743, 745-748, 750, 752-755, 757-766, 768-776, 778-787, 789-796]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 97
	v: [798, 800-805, 807-813, 815-824, 826-829, 831-835, 837-839, 841-848, 850-854, 856-863, 865-867, 869-870, 872-875, 878-887, 889-902, 904-910]


2021/12/03 22:59:39 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 22:59:44 eventingSetup.go:719: Waiting for function: TestBucketDeleteWithRebOut status to change to undeployed
2021/12/03 22:59:44 eventingSetup.go:727: Function: TestBucketDeleteWithRebOut status changed to undeployed
2021/12/03 22:59:49 clusterSetup.go:397: Waiting for app: TestBucketDeleteWithRebOut to get un-deployed
2021/12/03 22:59:49 clusterSetup.go:405: App: TestBucketDeleteWithRebOut got un-deployed
2021/12/03 22:59:49 eventingSetup.go:46: No eventing-con process running
2021/12/03 22:59:49 eventingSetup.go:418: Function: TestBucketDeleteWithRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestBucketDeleteWithRebOut deleting in the background"
}
2021/12/03 22:59:51 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/03 22:59:56 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 22:59:57 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/03 23:00:02 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 23:00:02 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv,index,n1ql to the cluster
2021/12/03 23:00:03 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/03 23:00:08 Rebalance progress: 100
--- PASS: TestBucketDeleteWithRebOut (297.55s)
=== RUN   TestN1QLRecursion
2021/12/03 23:00:10 Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/03 23:00:11 Error in adding nodes : , response: ["Prepare join failed. Node is already part of cluster."]
2021/12/03 23:00:11 Removing node(s): [] from the cluster
2021/12/03 23:00:17 Rebalance progress: 100
2021/12/03 23:00:23 Context: Bucket flush, response code: 200 dump: 
2021/12/03 23:00:29 Context: Bucket flush, response code: 200 dump: 
2021/12/03 23:00:35 Context: Bucket flush, response code: 200 dump: 
2021/12/03 23:00:40 Deploying app: TestN1QLRecursion_function1. Item count src bucket: 0 dst bucket: 0
2021/12/03 23:00:41 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLRecursion_function1' in metakv",
  "warnings": null
 }
}
2021/12/03 23:00:41 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLRecursion_function1' in metakv",
  "warnings": null
 }
} err 
2021/12/03 23:00:46 Waiting for app: TestN1QLRecursion_function1 to get deployed
2021/12/03 23:00:51 Waiting for app: TestN1QLRecursion_function1 to get deployed
2021/12/03 23:00:56 Waiting for app: TestN1QLRecursion_function1 to get deployed
2021/12/03 23:01:01 Waiting for app: TestN1QLRecursion_function1 to get deployed
2021/12/03 23:01:01 App: TestN1QLRecursion_function1 got deployed
2021/12/03 23:01:01 TestN1QLRecursion_function1 is deployed
2021/12/03 23:01:01 Deploying app: TestN1QLRecursion_function3. Item count src bucket: 0 dst bucket: 0
2021/12/03 23:01:02 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLRecursion_function3' in metakv",
  "warnings": null
 }
}
2021/12/03 23:01:02 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLRecursion_function3' in metakv",
  "warnings": null
 }
} err 
2021/12/03 23:01:07 Waiting for app: TestN1QLRecursion_function3 to get deployed
2021/12/03 23:01:12 Waiting for app: TestN1QLRecursion_function3 to get deployed
2021/12/03 23:01:17 Waiting for app: TestN1QLRecursion_function3 to get deployed
2021/12/03 23:01:22 Waiting for app: TestN1QLRecursion_function3 to get deployed
2021/12/03 23:01:22 App: TestN1QLRecursion_function3 got deployed
2021/12/03 23:01:22 TestN1QLRecursion_function3 is deployed
2021/12/03 23:01:22 Deploying app: TestN1QLRecursion_function2. Item count src bucket: 0 dst bucket: 0
2021/12/03 23:01:22 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_INTER_BUCKET_RECURSION",
 "code": 51,
 "description": "Inter bucket recursion error, deployment of current handler will cause inter bucket recursion",
 "attributes": null,
 "runtime_info": {
  "code": 51,
  "info": "Inter bucket recursion error; function: TestN1QLRecursion_function2 causes a cycle involving functions: [TestN1QLRecursion_function3 TestN1QLRecursion_function1], hence deployment is disallowed"
 }
}
2021/12/03 23:01:22 response body {
 "name": "ERR_INTER_BUCKET_RECURSION",
 "code": 51,
 "description": "Inter bucket recursion error, deployment of current handler will cause inter bucket recursion",
 "attributes": null,
 "runtime_info": {
  "code": 51,
  "info": "Inter bucket recursion error; function: TestN1QLRecursion_function2 causes a cycle involving functions: [TestN1QLRecursion_function3 TestN1QLRecursion_function1], hence deployment is disallowed"
 }
} err 
2021/12/03 23:01:22 Success: TestN1QLRecursion_function2 is not deployed
2021/12/03 23:01:23 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 23:01:28 Waiting for app: TestN1QLRecursion_function1 to get un-deployed
2021/12/03 23:01:33 Waiting for app: TestN1QLRecursion_function1 to get un-deployed
2021/12/03 23:01:38 Waiting for app: TestN1QLRecursion_function1 to get un-deployed
2021/12/03 23:01:43 Waiting for app: TestN1QLRecursion_function1 to get un-deployed
2021/12/03 23:01:43 App: TestN1QLRecursion_function1 got un-deployed
2021/12/03 23:01:43 No eventing-con process running
2021/12/03 23:01:43 Function: TestN1QLRecursion_function1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLRecursion_function1 deleting in the background"
}
2021/12/03 23:01:44 Context: Bucket flush, response code: 200 dump: 
2021/12/03 23:01:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 23:01:51 Context: Bucket flush, response code: 200 dump: 
2021/12/03 23:01:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 23:01:57 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 23:02:02 Waiting for app: TestN1QLRecursion_function3 to get un-deployed
2021/12/03 23:02:07 Waiting for app: TestN1QLRecursion_function3 to get un-deployed
2021/12/03 23:02:12 Waiting for app: TestN1QLRecursion_function3 to get un-deployed
2021/12/03 23:02:17 Waiting for app: TestN1QLRecursion_function3 to get un-deployed
2021/12/03 23:02:17 App: TestN1QLRecursion_function3 got un-deployed
2021/12/03 23:02:17 No eventing-con process running
2021/12/03 23:02:17 Function: TestN1QLRecursion_function3 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLRecursion_function3 deleting in the background"
}
2021/12/03 23:02:19 Context: Bucket flush, response code: 200 dump: 
2021/12/03 23:02:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 23:02:26 Context: Bucket flush, response code: 200 dump: 
2021/12/03 23:02:31 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 23:02:32 Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/03 23:02:37 Rebalance progress: 100
--- PASS: TestN1QLRecursion (146.50s)
=== RUN   TestN1QLAllowRecursion
2021/12/03 23:02:37 Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/03 23:02:37 Error in adding nodes : , response: ["Prepare join failed. Node is already part of cluster."]
2021/12/03 23:02:37 Removing node(s): [] from the cluster
2021/12/03 23:02:42 Rebalance progress: 100
2021/12/03 23:02:48 Context: Bucket flush, response code: 200 dump: 
2021/12/03 23:02:54 Context: Bucket flush, response code: 200 dump: 
2021/12/03 23:03:00 Context: Bucket flush, response code: 200 dump: 
2021/12/03 23:03:05 Deploying app: TestN1QLAllowRecursion_function1. Item count src bucket: 0 dst bucket: 0
2021/12/03 23:03:06 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function1' in metakv",
  "warnings": null
 }
}
2021/12/03 23:03:06 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function1' in metakv",
  "warnings": null
 }
} err 
2021/12/03 23:03:11 Waiting for app: TestN1QLAllowRecursion_function1 to get deployed
2021/12/03 23:03:16 Waiting for app: TestN1QLAllowRecursion_function1 to get deployed
2021/12/03 23:03:21 Waiting for app: TestN1QLAllowRecursion_function1 to get deployed
2021/12/03 23:03:26 Waiting for app: TestN1QLAllowRecursion_function1 to get deployed
2021/12/03 23:03:26 App: TestN1QLAllowRecursion_function1 got deployed
2021/12/03 23:03:26 TestN1QLAllowRecursion_function1 is deployed
2021/12/03 23:03:26 Deploying app: TestN1QLAllowRecursion_function3. Item count src bucket: 0 dst bucket: 0
2021/12/03 23:03:28 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function3' in metakv",
  "warnings": null
 }
}
2021/12/03 23:03:28 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function3' in metakv",
  "warnings": null
 }
} err 
2021/12/03 23:03:33 Waiting for app: TestN1QLAllowRecursion_function3 to get deployed
2021/12/03 23:03:38 Waiting for app: TestN1QLAllowRecursion_function3 to get deployed
2021/12/03 23:03:43 Waiting for app: TestN1QLAllowRecursion_function3 to get deployed
2021/12/03 23:03:48 Waiting for app: TestN1QLAllowRecursion_function3 to get deployed
2021/12/03 23:03:48 App: TestN1QLAllowRecursion_function3 got deployed
2021/12/03 23:03:48 TestN1QLAllowRecursion_function3 is deployed
2021/12/03 23:03:48 Deploying app: TestN1QLAllowRecursion_function2. Item count src bucket: 0 dst bucket: 0
2021/12/03 23:03:49 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function2' in metakv",
  "warnings": null
 }
}
2021/12/03 23:03:49 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function2' in metakv",
  "warnings": null
 }
} err 
2021/12/03 23:03:54 Waiting for app: TestN1QLAllowRecursion_function2 to get deployed
2021/12/03 23:03:59 Waiting for app: TestN1QLAllowRecursion_function2 to get deployed
2021/12/03 23:04:04 Waiting for app: TestN1QLAllowRecursion_function2 to get deployed
2021/12/03 23:04:09 Waiting for app: TestN1QLAllowRecursion_function2 to get deployed
2021/12/03 23:04:09 App: TestN1QLAllowRecursion_function2 got deployed
2021/12/03 23:04:09 Success: TestN1QLAllowRecursion_function2 is deployed after allow_interbucket_recursion
2021/12/03 23:04:09 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 23:04:14 Waiting for app: TestN1QLAllowRecursion_function1 to get un-deployed
2021/12/03 23:04:19 Waiting for app: TestN1QLAllowRecursion_function1 to get un-deployed
2021/12/03 23:04:24 Waiting for app: TestN1QLAllowRecursion_function1 to get un-deployed
2021/12/03 23:04:29 Waiting for app: TestN1QLAllowRecursion_function1 to get un-deployed
2021/12/03 23:04:29 App: TestN1QLAllowRecursion_function1 got un-deployed
2021/12/03 23:04:29 No eventing-con process running
2021/12/03 23:04:29 Function: TestN1QLAllowRecursion_function1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLAllowRecursion_function1 deleting in the background"
}
2021/12/03 23:04:31 Context: Bucket flush, response code: 200 dump: 
2021/12/03 23:04:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 23:04:38 Context: Bucket flush, response code: 200 dump: 
2021/12/03 23:04:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 23:04:43 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 23:04:48 Waiting for app: TestN1QLAllowRecursion_function2 to get un-deployed
2021/12/03 23:04:53 Waiting for app: TestN1QLAllowRecursion_function2 to get un-deployed
2021/12/03 23:04:58 Waiting for app: TestN1QLAllowRecursion_function2 to get un-deployed
2021/12/03 23:05:03 Waiting for app: TestN1QLAllowRecursion_function2 to get un-deployed
2021/12/03 23:05:03 App: TestN1QLAllowRecursion_function2 got un-deployed
2021/12/03 23:05:03 No eventing-con process running
2021/12/03 23:05:03 Function: TestN1QLAllowRecursion_function2 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLAllowRecursion_function2 deleting in the background"
}
2021/12/03 23:05:05 Context: Bucket flush, response code: 200 dump: 
2021/12/03 23:05:10 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 23:05:12 Context: Bucket flush, response code: 200 dump: 
2021/12/03 23:05:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 23:05:17 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 23:05:22 Waiting for app: TestN1QLAllowRecursion_function3 to get un-deployed
2021/12/03 23:05:28 Waiting for app: TestN1QLAllowRecursion_function3 to get un-deployed
2021/12/03 23:05:33 Waiting for app: TestN1QLAllowRecursion_function3 to get un-deployed
2021/12/03 23:05:38 Waiting for app: TestN1QLAllowRecursion_function3 to get un-deployed
2021/12/03 23:05:38 App: TestN1QLAllowRecursion_function3 got un-deployed
2021/12/03 23:05:38 No eventing-con process running
2021/12/03 23:05:38 Function: TestN1QLAllowRecursion_function3 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLAllowRecursion_function3 deleting in the background"
}
2021/12/03 23:05:40 Context: Bucket flush, response code: 200 dump: 
2021/12/03 23:05:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 23:05:47 Context: Bucket flush, response code: 200 dump: 
2021/12/03 23:05:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 23:05:53 Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/03 23:05:58 Rebalance progress: 100
--- PASS: TestN1QLAllowRecursion (201.23s)
=== RUN   TestOnDeleteExpiryBucketOp
2021/12/03 23:05:58 Sleeping for some time
2021/12/03 23:06:08 Waiting for app: TestOnDeleteExpiryBucketOp to get un-deployed
2021/12/03 23:06:08 App: TestOnDeleteExpiryBucketOp got un-deployed
2021/12/03 23:06:08 No eventing-con process running
2021/12/03 23:06:08 Function: TestOnDeleteExpiryBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnDeleteExpiryBucketOp not found"
 }
}
2021/12/03 23:06:10 Context: Bucket flush, response code: 200 dump: 
2021/12/03 23:06:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 23:06:17 Context: Bucket flush, response code: 200 dump: 
2021/12/03 23:06:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 23:06:22 Deploying app: TestOnDeleteExpiryBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/03 23:06:23 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnDeleteExpiryBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/03 23:06:23 Starting bucket ops to source bucket
2021/12/03 23:07:12 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 23:07:17 Waiting for dst bucket item count to get to: 5000 curr count: 1473
2021/12/03 23:07:22 Waiting for dst bucket item count to get to: 5000 curr count: 3973
2021/12/03 23:07:22 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/03 23:07:22 Starting bucket ops to source bucket
2021/12/03 23:07:23 Starting bucket ops to source bucket
2021/12/03 23:08:29 Waiting for dst bucket item count to get to: 7000 curr count: 5000
2021/12/03 23:08:34 Waiting for dst bucket item count to get to: 7000 curr count: 6000
2021/12/03 23:08:34 src & dst bucket item count matched up. src bucket count: 7000 dst bucket count: 7000
2021/12/03 23:08:34 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 23:08:34 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 23:08:39 Waiting for app: TestOnDeleteExpiryBucketOp to get un-deployed
2021/12/03 23:08:44 Waiting for app: TestOnDeleteExpiryBucketOp to get un-deployed
2021/12/03 23:08:49 Waiting for app: TestOnDeleteExpiryBucketOp to get un-deployed
2021/12/03 23:08:54 Waiting for app: TestOnDeleteExpiryBucketOp to get un-deployed
2021/12/03 23:08:54 App: TestOnDeleteExpiryBucketOp got un-deployed
2021/12/03 23:08:54 No eventing-con process running
2021/12/03 23:08:54 Function: TestOnDeleteExpiryBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnDeleteExpiryBucketOp deleting in the background"
}
2021/12/03 23:08:56 Context: Bucket flush, response code: 200 dump: 
2021/12/03 23:09:06 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/03 23:09:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 23:09:08 Context: Bucket flush, response code: 200 dump: 
2021/12/03 23:09:18 Waiting for dst bucket item count to get to: 0 curr count: 5094
2021/12/03 23:09:23 Waiting for dst bucket item count to get to: 0 curr count: 1594
2021/12/03 23:09:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnDeleteExpiryBucketOp (204.87s)
=== RUN   TestTimerOverWriteSameReference
2021/12/03 23:09:28 Waiting for app: TestTimerOverWriteSameReference to get un-deployed
2021/12/03 23:09:28 App: TestTimerOverWriteSameReference got un-deployed
2021/12/03 23:09:28 No eventing-con process running
2021/12/03 23:09:28 Function: TestTimerOverWriteSameReference delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestTimerOverWriteSameReference not found"
 }
}
2021/12/03 23:09:30 Context: Bucket flush, response code: 200 dump: 
2021/12/03 23:09:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 23:09:37 Context: Bucket flush, response code: 200 dump: 
2021/12/03 23:09:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 23:09:42 Deploying app: TestTimerOverWriteSameReference. Item count src bucket: 0 dst bucket: 0
2021/12/03 23:09:43 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestTimerOverWriteSameReference' in metakv",
  "warnings": null
 }
}
2021/12/03 23:09:48 Waiting for app: TestTimerOverWriteSameReference to get deployed
2021/12/03 23:09:53 Waiting for app: TestTimerOverWriteSameReference to get deployed
2021/12/03 23:09:58 Waiting for app: TestTimerOverWriteSameReference to get deployed
2021/12/03 23:10:03 Waiting for app: TestTimerOverWriteSameReference to get deployed
2021/12/03 23:10:03 App: TestTimerOverWriteSameReference got deployed
2021/12/03 23:10:13 Starting bucket ops to source bucket
2021/12/03 23:11:14 src & dst bucket item count matched up. src bucket count: 1 dst bucket count: 1
2021/12/03 23:11:24 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 23:11:24 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 23:11:29 Waiting for app: TestTimerOverWriteSameReference to get un-deployed
2021/12/03 23:11:34 Waiting for app: TestTimerOverWriteSameReference to get un-deployed
2021/12/03 23:11:39 Waiting for app: TestTimerOverWriteSameReference to get un-deployed
2021/12/03 23:11:44 Waiting for app: TestTimerOverWriteSameReference to get un-deployed
2021/12/03 23:11:44 App: TestTimerOverWriteSameReference got un-deployed
2021/12/03 23:11:44 No eventing-con process running
2021/12/03 23:11:44 Function: TestTimerOverWriteSameReference delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestTimerOverWriteSameReference deleting in the background"
}
2021/12/03 23:11:46 Context: Bucket flush, response code: 200 dump: 
2021/12/03 23:11:56 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/03 23:11:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 23:11:58 Context: Bucket flush, response code: 200 dump: 
2021/12/03 23:12:08 Waiting for dst bucket item count to get to: 0 curr count: 1
2021/12/03 23:12:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestTimerOverWriteSameReference (165.05s)
=== RUN   TestCancelTimerBucketop
2021/12/03 23:12:13 Waiting for app: TestCancelTimerBucketop to get un-deployed
2021/12/03 23:12:13 App: TestCancelTimerBucketop got un-deployed
2021/12/03 23:12:13 No eventing-con process running
2021/12/03 23:12:13 Function: TestCancelTimerBucketop delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestCancelTimerBucketop not found"
 }
}
2021/12/03 23:12:15 Context: Bucket flush, response code: 200 dump: 
2021/12/03 23:12:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 23:12:22 Context: Bucket flush, response code: 200 dump: 
2021/12/03 23:12:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 23:12:27 Deploying app: TestCancelTimerBucketop. Item count src bucket: 0 dst bucket: 0
2021/12/03 23:12:28 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCancelTimerBucketop' in metakv",
  "warnings": null
 }
}
2021/12/03 23:12:33 Waiting for app: TestCancelTimerBucketop to get deployed
2021/12/03 23:12:38 Waiting for app: TestCancelTimerBucketop to get deployed
2021/12/03 23:12:43 Waiting for app: TestCancelTimerBucketop to get deployed
2021/12/03 23:12:48 Waiting for app: TestCancelTimerBucketop to get deployed
2021/12/03 23:12:48 App: TestCancelTimerBucketop got deployed
2021/12/03 23:12:48 Starting bucket ops to source bucket
2021/12/03 23:12:55 Starting bucket ops to source bucket
2021/12/03 23:13:57 src & dst bucket item count matched up. src bucket count: 1000 dst bucket count: 1000
2021/12/03 23:13:57 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 23:13:58 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 23:14:03 Waiting for app: TestCancelTimerBucketop to get un-deployed
2021/12/03 23:14:08 Waiting for app: TestCancelTimerBucketop to get un-deployed
2021/12/03 23:14:13 Waiting for app: TestCancelTimerBucketop to get un-deployed
2021/12/03 23:14:18 Waiting for app: TestCancelTimerBucketop to get un-deployed
2021/12/03 23:14:18 App: TestCancelTimerBucketop got un-deployed
2021/12/03 23:14:18 No eventing-con process running
2021/12/03 23:14:18 Function: TestCancelTimerBucketop delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCancelTimerBucketop deleting in the background"
}
2021/12/03 23:14:20 Context: Bucket flush, response code: 200 dump: 
2021/12/03 23:14:30 Waiting for dst bucket item count to get to: 0 curr count: 500
2021/12/03 23:14:35 Waiting for dst bucket item count to get to: 0 curr count: 500
2021/12/03 23:14:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 23:14:36 Context: Bucket flush, response code: 200 dump: 
2021/12/03 23:14:46 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/03 23:14:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCancelTimerBucketop (158.34s)
=== RUN   TestJSExpiryDate
2021/12/03 23:14:51 Starting bucket ops to source bucket
2021/12/03 23:15:04 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/03 23:15:04 Deploying app: TestJSExpiryDate. Item count src bucket: 5000 dst bucket: 0
2021/12/03 23:15:06 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestJSExpiryDate' in metakv",
  "warnings": null
 }
}
2021/12/03 23:15:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 23:15:36 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 23:15:36 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 23:15:41 Waiting for app: TestJSExpiryDate to get un-deployed
2021/12/03 23:15:46 Waiting for app: TestJSExpiryDate to get un-deployed
2021/12/03 23:15:51 Waiting for app: TestJSExpiryDate to get un-deployed
2021/12/03 23:15:56 Waiting for app: TestJSExpiryDate to get un-deployed
2021/12/03 23:15:56 App: TestJSExpiryDate got un-deployed
2021/12/03 23:15:56 No eventing-con process running
2021/12/03 23:15:56 Function: TestJSExpiryDate delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestJSExpiryDate deleting in the background"
}
2021/12/03 23:15:58 Context: Bucket flush, response code: 200 dump: 
2021/12/03 23:16:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 23:16:05 Context: Bucket flush, response code: 200 dump: 
2021/12/03 23:16:10 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestJSExpiryDate (83.86s)
=== RUN   TestBinaryDoc
2021/12/03 23:16:15 Starting bucket ops to source bucket
2021/12/03 23:16:15 Deploying app: TestBinaryDoc. Item count src bucket: 0 dst bucket: 0
2021/12/03 23:16:17 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestBinaryDoc' in metakv",
  "warnings": null
 }
}
2021/12/03 23:16:17 Starting bucket ops to source bucket
2021/12/03 23:16:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 23:16:17 Deleting source bucket docs now
2021/12/03 23:16:17 Starting bucket ops to source bucket
2021/12/03 23:16:54 Waiting for dst bucket item count to get to: 32 curr count: 21
2021/12/03 23:16:59 Waiting for dst bucket item count to get to: 32 curr count: 31
2021/12/03 23:17:04 Waiting for dst bucket item count to get to: 32 curr count: 31
2021/12/03 23:17:04 src & dst bucket item count matched up. src bucket count: 32 dst bucket count: 32
2021/12/03 23:17:04 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 23:17:04 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 23:17:09 Waiting for app: TestBinaryDoc to get un-deployed
2021/12/03 23:17:14 Waiting for app: TestBinaryDoc to get un-deployed
2021/12/03 23:17:19 Waiting for app: TestBinaryDoc to get un-deployed
2021/12/03 23:17:24 Waiting for app: TestBinaryDoc to get un-deployed
2021/12/03 23:17:24 App: TestBinaryDoc got un-deployed
2021/12/03 23:17:24 No eventing-con process running
2021/12/03 23:17:24 Function: TestBinaryDoc delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestBinaryDoc deleting in the background"
}
2021/12/03 23:17:26 Context: Bucket flush, response code: 200 dump: 
2021/12/03 23:17:36 Waiting for dst bucket item count to get to: 0 curr count: 32
2021/12/03 23:17:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 23:17:38 Context: Bucket flush, response code: 200 dump: 
2021/12/03 23:17:48 Waiting for dst bucket item count to get to: 0 curr count: 17
2021/12/03 23:17:53 Waiting for dst bucket item count to get to: 0 curr count: 1
2021/12/03 23:17:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestBinaryDoc (102.41s)
=== RUN   TestConstantBindings
2021/12/03 23:17:53 Deploying app: TestConstantBindings. Item count src bucket: 0 dst bucket: 0
2021/12/03 23:17:54 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestConstantBindings' in metakv",
  "warnings": null
 }
}
2021/12/03 23:17:54 Starting bucket ops to source bucket
2021/12/03 23:18:00 Waiting for dst bucket item count to get to: 6000 curr count: 0
2021/12/03 23:18:05 Waiting for dst bucket item count to get to: 6000 curr count: 0
2021/12/03 23:18:10 Waiting for dst bucket item count to get to: 6000 curr count: 0
2021/12/03 23:18:15 Waiting for dst bucket item count to get to: 6000 curr count: 1922
2021/12/03 23:18:20 Waiting for dst bucket item count to get to: 6000 curr count: 4922
2021/12/03 23:18:20 src & dst bucket item count matched up. src bucket count: 6000 dst bucket count: 6000
2021/12/03 23:18:20 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 23:18:20 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 23:18:25 Waiting for app: TestConstantBindings to get un-deployed
2021/12/03 23:18:30 Waiting for app: TestConstantBindings to get un-deployed
2021/12/03 23:18:35 Waiting for app: TestConstantBindings to get un-deployed
2021/12/03 23:18:40 Waiting for app: TestConstantBindings to get un-deployed
2021/12/03 23:18:40 App: TestConstantBindings got un-deployed
2021/12/03 23:18:40 No eventing-con process running
2021/12/03 23:18:40 Function: TestConstantBindings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestConstantBindings deleting in the background"
}
2021/12/03 23:18:42 Context: Bucket flush, response code: 200 dump: 
2021/12/03 23:18:52 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/03 23:18:57 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/03 23:18:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 23:18:59 Context: Bucket flush, response code: 200 dump: 
2021/12/03 23:19:09 Waiting for dst bucket item count to get to: 0 curr count: 5080
2021/12/03 23:19:14 Waiting for dst bucket item count to get to: 0 curr count: 2080
2021/12/03 23:19:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestConstantBindings (80.78s)
=== RUN   TestCodeInsight
2021/12/03 23:19:19 Waiting for app: TestCodeInsight to get un-deployed
2021/12/03 23:19:19 App: TestCodeInsight got un-deployed
2021/12/03 23:19:19 No eventing-con process running
2021/12/03 23:19:19 Function: TestCodeInsight delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestCodeInsight not found"
 }
}
2021/12/03 23:19:20 Context: Bucket flush, response code: 200 dump: 
2021/12/03 23:19:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 23:19:27 Context: Bucket flush, response code: 200 dump: 
2021/12/03 23:19:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 23:19:32 Starting bucket ops to source bucket
2021/12/03 23:19:33 Deploying app: TestCodeInsight. Item count src bucket: 0 dst bucket: 0
2021/12/03 23:19:34 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCodeInsight' in metakv",
  "warnings": null
 }
}
2021/12/03 23:19:39 Waiting for app: TestCodeInsight to get deployed
2021/12/03 23:19:44 Waiting for app: TestCodeInsight to get deployed
2021/12/03 23:19:49 Waiting for app: TestCodeInsight to get deployed
2021/12/03 23:19:54 Waiting for app: TestCodeInsight to get deployed
2021/12/03 23:19:54 App: TestCodeInsight got deployed
2021/12/03 23:19:59 Waiting for dst bucket item count to get to: 1000 curr count: 856
2021/12/03 23:19:59 src & dst bucket item count matched up. src bucket count: 1000 dst bucket count: 1000
2021/12/03 23:19:59 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 23:20:04 Waiting for app: TestCodeInsight to get un-deployed
2021/12/03 23:20:09 Waiting for app: TestCodeInsight to get un-deployed
2021/12/03 23:20:14 Waiting for app: TestCodeInsight to get un-deployed
2021/12/03 23:20:19 Waiting for app: TestCodeInsight to get un-deployed
2021/12/03 23:20:19 App: TestCodeInsight got un-deployed
2021/12/03 23:20:19 No eventing-con process running
2021/12/03 23:20:19 Function: TestCodeInsight delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCodeInsight deleting in the background"
}
2021/12/03 23:20:21 Context: Bucket flush, response code: 200 dump: 
2021/12/03 23:20:31 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/03 23:20:36 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/03 23:20:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 23:20:38 Context: Bucket flush, response code: 200 dump: 
2021/12/03 23:20:48 Waiting for dst bucket item count to get to: 0 curr count: 716
2021/12/03 23:20:53 Waiting for dst bucket item count to get to: 0 curr count: 216
2021/12/03 23:20:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCodeInsight (99.52s)
=== RUN   TestParserTransform
--- PASS: TestParserTransform (0.05s)
=== RUN   TestParserDetect
--- PASS: TestParserDetect (0.00s)
PASS
ok  	github.com/couchbase/eventing/tests/functional_tests	12632.987s

Functional tests took 3 hours, 30 minutes and 36 seconds.

N1ql tests

2021/12/03 23:21:01 Started n1ql
2021/12/03 23:21:10 Adding node: https://127.0.0.1:19001 with role: kv,index,n1ql to the cluster
2021/12/03 23:21:11 Error in adding nodes : , response: ["Prepare join failed. Node is already part of cluster."]
2021/12/03 23:21:11 Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/03 23:21:11 Removing node(s): [] from the cluster
2021/12/03 23:21:16 Rebalance progress: 100
=== RUN   TestRecursiveMutationN1QL
2021/12/03 23:21:21 Waiting for app: TestRecursiveMutationN1QL to get un-deployed
2021/12/03 23:21:21 App: TestRecursiveMutationN1QL got un-deployed
2021/12/03 23:21:21 No eventing-con process running
2021/12/03 23:21:21 Function: TestRecursiveMutationN1QL delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestRecursiveMutationN1QL not found"
 }
}
2021/12/03 23:21:23 Context: Bucket flush, response code: 200 dump: 
2021/12/03 23:21:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 23:21:29 Context: Bucket flush, response code: 200 dump: 
2021/12/03 23:21:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 23:21:35 Deploying app: TestRecursiveMutationN1QL. Item count src bucket: 0 dst bucket: 0
2021/12/03 23:21:35 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_INTER_BUCKET_RECURSION",
 "code": 51,
 "description": "Inter bucket recursion error, deployment of current handler will cause inter bucket recursion",
 "attributes": null,
 "runtime_info": {
  "code": 51,
  "info": "Function: TestRecursiveMutationN1QL N1QL DML to source keyspace default:default"
 }
}
--- PASS: TestRecursiveMutationN1QL (18.63s)
=== RUN   TestFlexReset1
2021/12/03 23:21:40 Waiting for app: TestFlexReset1 to get un-deployed
2021/12/03 23:21:40 App: TestFlexReset1 got un-deployed
2021/12/03 23:21:40 No eventing-con process running
2021/12/03 23:21:40 Function: TestFlexReset1 delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestFlexReset1 not found"
 }
}
2021/12/03 23:21:41 Context: Bucket flush, response code: 200 dump: 
2021/12/03 23:21:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 23:21:48 Context: Bucket flush, response code: 200 dump: 
2021/12/03 23:21:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 23:21:53 Starting bucket ops to source bucket
2021/12/03 23:21:54 Deploying app: TestFlexReset1. Item count src bucket: 0 dst bucket: 0
2021/12/03 23:21:55 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestFlexReset1' in metakv",
  "warnings": null
 }
}
2021/12/03 23:22:00 Waiting for app: TestFlexReset1 to get deployed
2021/12/03 23:22:05 Waiting for app: TestFlexReset1 to get deployed
2021/12/03 23:22:10 Waiting for app: TestFlexReset1 to get deployed
2021/12/03 23:22:15 Waiting for app: TestFlexReset1 to get deployed
2021/12/03 23:22:15 App: TestFlexReset1 got deployed
2021/12/03 23:22:20 Waiting for dst bucket item count to get to: 1000 curr count: 425
2021/12/03 23:22:25 Waiting for dst bucket item count to get to: 1000 curr count: 900
2021/12/03 23:22:25 src & dst bucket item count matched up. src bucket count: 1000 dst bucket count: 1000
2021/12/03 23:22:25 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 23:22:25 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 23:22:30 Waiting for app: TestFlexReset1 to get un-deployed
2021/12/03 23:22:35 Waiting for app: TestFlexReset1 to get un-deployed
2021/12/03 23:22:40 Waiting for app: TestFlexReset1 to get un-deployed
2021/12/03 23:22:45 Waiting for app: TestFlexReset1 to get un-deployed
2021/12/03 23:22:45 App: TestFlexReset1 got un-deployed
2021/12/03 23:22:45 No eventing-con process running
2021/12/03 23:22:45 Function: TestFlexReset1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestFlexReset1 deleting in the background"
}
2021/12/03 23:22:47 Context: Bucket flush, response code: 200 dump: 
2021/12/03 23:22:57 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/03 23:22:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 23:22:59 Context: Bucket flush, response code: 200 dump: 
2021/12/03 23:23:09 Waiting for dst bucket item count to get to: 0 curr count: 964
2021/12/03 23:23:14 Waiting for dst bucket item count to get to: 0 curr count: 464
2021/12/03 23:23:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestFlexReset1 (99.32s)
=== RUN   TestFlexReset2
2021/12/03 23:23:19 Waiting for app: TestFlexReset2 to get un-deployed
2021/12/03 23:23:19 App: TestFlexReset2 got un-deployed
2021/12/03 23:23:19 No eventing-con process running
2021/12/03 23:23:19 Function: TestFlexReset2 delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestFlexReset2 not found"
 }
}
2021/12/03 23:23:21 Context: Bucket flush, response code: 200 dump: 
2021/12/03 23:23:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 23:23:27 Context: Bucket flush, response code: 200 dump: 
2021/12/03 23:23:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 23:23:32 Starting bucket ops to source bucket
2021/12/03 23:23:33 Deploying app: TestFlexReset2. Item count src bucket: 0 dst bucket: 0
2021/12/03 23:23:34 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestFlexReset2' in metakv",
  "warnings": null
 }
}
2021/12/03 23:23:39 Waiting for app: TestFlexReset2 to get deployed
2021/12/03 23:23:44 Waiting for app: TestFlexReset2 to get deployed
2021/12/03 23:23:49 Waiting for app: TestFlexReset2 to get deployed
2021/12/03 23:23:54 Waiting for app: TestFlexReset2 to get deployed
2021/12/03 23:23:54 App: TestFlexReset2 got deployed
2021/12/03 23:23:59 Waiting for dst bucket item count to get to: 1000 curr count: 600
2021/12/03 23:23:59 src & dst bucket item count matched up. src bucket count: 1000 dst bucket count: 1000
2021/12/03 23:23:59 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 23:23:59 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 23:24:04 Waiting for app: TestFlexReset2 to get un-deployed
2021/12/03 23:24:10 Waiting for app: TestFlexReset2 to get un-deployed
2021/12/03 23:24:15 Waiting for app: TestFlexReset2 to get un-deployed
2021/12/03 23:24:20 Waiting for app: TestFlexReset2 to get un-deployed
2021/12/03 23:24:20 App: TestFlexReset2 got un-deployed
2021/12/03 23:24:20 No eventing-con process running
2021/12/03 23:24:20 Function: TestFlexReset2 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestFlexReset2 deleting in the background"
}
2021/12/03 23:24:21 Context: Bucket flush, response code: 200 dump: 
2021/12/03 23:24:31 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/03 23:24:36 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/03 23:24:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 23:24:38 Context: Bucket flush, response code: 200 dump: 
2021/12/03 23:24:48 Waiting for dst bucket item count to get to: 0 curr count: 769
2021/12/03 23:24:53 Waiting for dst bucket item count to get to: 0 curr count: 269
2021/12/03 23:24:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestFlexReset2 (99.35s)
=== RUN   TestN1QLLabelledBreak
2021/12/03 23:24:58 Waiting for app: TestN1QLLabelledBreak to get un-deployed
2021/12/03 23:24:58 App: TestN1QLLabelledBreak got un-deployed
2021/12/03 23:24:58 No eventing-con process running
2021/12/03 23:24:58 Function: TestN1QLLabelledBreak delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLLabelledBreak not found"
 }
}
2021/12/03 23:25:00 Context: Bucket flush, response code: 200 dump: 
2021/12/03 23:25:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 23:25:07 Context: Bucket flush, response code: 200 dump: 
2021/12/03 23:25:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 23:25:12 Deploying app: TestN1QLLabelledBreak. Item count src bucket: 0 dst bucket: 0
2021/12/03 23:25:13 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLLabelledBreak' in metakv",
  "warnings": null
 }
}
2021/12/03 23:25:18 Waiting for app: TestN1QLLabelledBreak to get deployed
2021/12/03 23:25:23 Waiting for app: TestN1QLLabelledBreak to get deployed
2021/12/03 23:25:28 Waiting for app: TestN1QLLabelledBreak to get deployed
2021/12/03 23:25:33 Waiting for app: TestN1QLLabelledBreak to get deployed
2021/12/03 23:25:33 App: TestN1QLLabelledBreak got deployed
2021/12/03 23:25:33 Starting bucket ops to source bucket
2021/12/03 23:25:43 Waiting for dst bucket item count to get to: 10000 curr count: 0
2021/12/03 23:25:48 Waiting for dst bucket item count to get to: 10000 curr count: 584
2021/12/03 23:25:53 Waiting for dst bucket item count to get to: 10000 curr count: 870
2021/12/03 23:25:58 Waiting for dst bucket item count to get to: 10000 curr count: 2778
2021/12/03 23:26:03 Waiting for dst bucket item count to get to: 10000 curr count: 4831
2021/12/03 23:26:08 Waiting for dst bucket item count to get to: 10000 curr count: 6863
2021/12/03 23:26:13 Waiting for dst bucket item count to get to: 10000 curr count: 8875
2021/12/03 23:26:18 Waiting for dst bucket item count to get to: 10000 curr count: 9649
2021/12/03 23:26:18 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/03 23:26:18 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 23:26:18 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 23:26:23 Waiting for app: TestN1QLLabelledBreak to get un-deployed
2021/12/03 23:26:28 Waiting for app: TestN1QLLabelledBreak to get un-deployed
2021/12/03 23:26:33 Waiting for app: TestN1QLLabelledBreak to get un-deployed
2021/12/03 23:26:38 Waiting for app: TestN1QLLabelledBreak to get un-deployed
2021/12/03 23:26:38 App: TestN1QLLabelledBreak got un-deployed
2021/12/03 23:26:38 No eventing-con process running
2021/12/03 23:26:38 Function: TestN1QLLabelledBreak delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLLabelledBreak deleting in the background"
}
2021/12/03 23:26:40 Context: Bucket flush, response code: 200 dump: 
2021/12/03 23:26:50 Waiting for dst bucket item count to get to: 0 curr count: 2699
2021/12/03 23:26:55 Waiting for dst bucket item count to get to: 0 curr count: 2699
2021/12/03 23:26:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 23:26:57 Context: Bucket flush, response code: 200 dump: 
2021/12/03 23:27:07 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/03 23:27:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLLabelledBreak (133.64s)
=== RUN   TestN1QLUnlabelledBreak
2021/12/03 23:27:12 Waiting for app: TestN1QLUnlabelledBreak to get un-deployed
2021/12/03 23:27:12 App: TestN1QLUnlabelledBreak got un-deployed
2021/12/03 23:27:12 No eventing-con process running
2021/12/03 23:27:12 Function: TestN1QLUnlabelledBreak delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLUnlabelledBreak not found"
 }
}
2021/12/03 23:27:14 Context: Bucket flush, response code: 200 dump: 
2021/12/03 23:27:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 23:27:20 Context: Bucket flush, response code: 200 dump: 
2021/12/03 23:27:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 23:27:25 Deploying app: TestN1QLUnlabelledBreak. Item count src bucket: 0 dst bucket: 0
2021/12/03 23:27:27 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLUnlabelledBreak' in metakv",
  "warnings": null
 }
}
2021/12/03 23:27:32 Waiting for app: TestN1QLUnlabelledBreak to get deployed
2021/12/03 23:27:37 Waiting for app: TestN1QLUnlabelledBreak to get deployed
2021/12/03 23:27:42 Waiting for app: TestN1QLUnlabelledBreak to get deployed
2021/12/03 23:27:47 Waiting for app: TestN1QLUnlabelledBreak to get deployed
2021/12/03 23:27:47 App: TestN1QLUnlabelledBreak got deployed
2021/12/03 23:27:47 Starting bucket ops to source bucket
2021/12/03 23:27:56 Waiting for dst bucket item count to get to: 10000 curr count: 298
2021/12/03 23:28:01 Waiting for dst bucket item count to get to: 10000 curr count: 298
2021/12/03 23:28:06 Waiting for dst bucket item count to get to: 10000 curr count: 298
2021/12/03 23:28:11 Waiting for dst bucket item count to get to: 10000 curr count: 4939
2021/12/03 23:28:16 Waiting for dst bucket item count to get to: 10000 curr count: 4939
2021/12/03 23:28:21 Waiting for dst bucket item count to get to: 10000 curr count: 9994
2021/12/03 23:28:26 Waiting for dst bucket item count to get to: 10000 curr count: 9994
2021/12/03 23:28:26 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/03 23:28:26 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 23:28:26 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 23:28:31 Waiting for app: TestN1QLUnlabelledBreak to get un-deployed
2021/12/03 23:28:36 Waiting for app: TestN1QLUnlabelledBreak to get un-deployed
2021/12/03 23:28:41 Waiting for app: TestN1QLUnlabelledBreak to get un-deployed
2021/12/03 23:28:46 Waiting for app: TestN1QLUnlabelledBreak to get un-deployed
2021/12/03 23:28:46 App: TestN1QLUnlabelledBreak got un-deployed
2021/12/03 23:28:46 No eventing-con process running
2021/12/03 23:28:46 Function: TestN1QLUnlabelledBreak delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLUnlabelledBreak deleting in the background"
}
2021/12/03 23:28:48 Context: Bucket flush, response code: 200 dump: 
2021/12/03 23:28:58 Waiting for dst bucket item count to get to: 0 curr count: 3930
2021/12/03 23:29:04 Waiting for dst bucket item count to get to: 0 curr count: 1430
2021/12/03 23:29:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 23:29:05 Context: Bucket flush, response code: 200 dump: 
2021/12/03 23:29:15 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/03 23:29:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLUnlabelledBreak (128.51s)
=== RUN   TestN1QLThrowStatement
2021/12/03 23:29:20 Waiting for app: TestN1QLThrowStatement to get un-deployed
2021/12/03 23:29:20 App: TestN1QLThrowStatement got un-deployed
2021/12/03 23:29:20 No eventing-con process running
2021/12/03 23:29:20 Function: TestN1QLThrowStatement delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLThrowStatement not found"
 }
}
2021/12/03 23:29:22 Context: Bucket flush, response code: 200 dump: 
2021/12/03 23:29:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 23:29:29 Context: Bucket flush, response code: 200 dump: 
2021/12/03 23:29:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 23:29:34 Deploying app: TestN1QLThrowStatement. Item count src bucket: 0 dst bucket: 0
2021/12/03 23:29:35 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLThrowStatement' in metakv",
  "warnings": null
 }
}
2021/12/03 23:29:40 Waiting for app: TestN1QLThrowStatement to get deployed
2021/12/03 23:29:45 Waiting for app: TestN1QLThrowStatement to get deployed
2021/12/03 23:29:50 Waiting for app: TestN1QLThrowStatement to get deployed
2021/12/03 23:29:55 Waiting for app: TestN1QLThrowStatement to get deployed
2021/12/03 23:29:55 App: TestN1QLThrowStatement got deployed
2021/12/03 23:29:55 Starting bucket ops to source bucket
2021/12/03 23:30:05 Waiting for dst bucket item count to get to: 10000 curr count: 0
2021/12/03 23:30:10 Waiting for dst bucket item count to get to: 10000 curr count: 406
2021/12/03 23:30:15 Waiting for dst bucket item count to get to: 10000 curr count: 406
2021/12/03 23:30:20 Waiting for dst bucket item count to get to: 10000 curr count: 4073
2021/12/03 23:30:25 Waiting for dst bucket item count to get to: 10000 curr count: 4073
2021/12/03 23:30:30 Waiting for dst bucket item count to get to: 10000 curr count: 8126
2021/12/03 23:30:35 Waiting for dst bucket item count to get to: 10000 curr count: 8126
2021/12/03 23:30:35 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/03 23:30:35 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 23:30:35 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 23:30:40 Waiting for app: TestN1QLThrowStatement to get un-deployed
2021/12/03 23:30:45 Waiting for app: TestN1QLThrowStatement to get un-deployed
2021/12/03 23:30:50 Waiting for app: TestN1QLThrowStatement to get un-deployed
2021/12/03 23:30:55 Waiting for app: TestN1QLThrowStatement to get un-deployed
2021/12/03 23:30:55 App: TestN1QLThrowStatement got un-deployed
2021/12/03 23:30:55 No eventing-con process running
2021/12/03 23:30:55 Function: TestN1QLThrowStatement delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLThrowStatement deleting in the background"
}
2021/12/03 23:30:57 Context: Bucket flush, response code: 200 dump: 
2021/12/03 23:31:07 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/03 23:31:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 23:31:09 Context: Bucket flush, response code: 200 dump: 
2021/12/03 23:31:19 Waiting for dst bucket item count to get to: 0 curr count: 9227
2021/12/03 23:31:24 Waiting for dst bucket item count to get to: 0 curr count: 4227
2021/12/03 23:31:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLThrowStatement (128.34s)
=== RUN   TestN1QLNestedForLoop
2021/12/03 23:31:29 Waiting for app: TestN1QLNestedForLoop to get un-deployed
2021/12/03 23:31:29 App: TestN1QLNestedForLoop got un-deployed
2021/12/03 23:31:29 No eventing-con process running
2021/12/03 23:31:29 Function: TestN1QLNestedForLoop delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLNestedForLoop not found"
 }
}
2021/12/03 23:31:31 Context: Bucket flush, response code: 200 dump: 
2021/12/03 23:31:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 23:31:37 Context: Bucket flush, response code: 200 dump: 
2021/12/03 23:31:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 23:31:42 Deploying app: TestN1QLNestedForLoop. Item count src bucket: 0 dst bucket: 0
2021/12/03 23:31:44 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLNestedForLoop' in metakv",
  "warnings": null
 }
}
2021/12/03 23:31:49 Waiting for app: TestN1QLNestedForLoop to get deployed
2021/12/03 23:31:54 Waiting for app: TestN1QLNestedForLoop to get deployed
2021/12/03 23:31:59 Waiting for app: TestN1QLNestedForLoop to get deployed
2021/12/03 23:32:04 Waiting for app: TestN1QLNestedForLoop to get deployed
2021/12/03 23:32:04 App: TestN1QLNestedForLoop got deployed
2021/12/03 23:32:04 Starting bucket ops to source bucket
2021/12/03 23:32:13 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 23:32:18 Waiting for dst bucket item count to get to: 5000 curr count: 126
2021/12/03 23:32:23 Waiting for dst bucket item count to get to: 5000 curr count: 423
2021/12/03 23:32:28 Waiting for dst bucket item count to get to: 5000 curr count: 1546
2021/12/03 23:32:33 Waiting for dst bucket item count to get to: 5000 curr count: 2759
2021/12/03 23:32:38 Waiting for dst bucket item count to get to: 5000 curr count: 3958
2021/12/03 23:32:43 Waiting for dst bucket item count to get to: 5000 curr count: 4948
2021/12/03 23:32:43 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/03 23:32:43 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 23:32:43 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 23:32:48 Waiting for app: TestN1QLNestedForLoop to get un-deployed
2021/12/03 23:32:53 Waiting for app: TestN1QLNestedForLoop to get un-deployed
2021/12/03 23:32:58 Waiting for app: TestN1QLNestedForLoop to get un-deployed
2021/12/03 23:33:03 Waiting for app: TestN1QLNestedForLoop to get un-deployed
2021/12/03 23:33:03 App: TestN1QLNestedForLoop got un-deployed
2021/12/03 23:33:03 No eventing-con process running
2021/12/03 23:33:03 Function: TestN1QLNestedForLoop delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLNestedForLoop deleting in the background"
}
2021/12/03 23:33:05 Context: Bucket flush, response code: 200 dump: 
2021/12/03 23:33:15 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/03 23:33:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 23:33:17 Context: Bucket flush, response code: 200 dump: 
2021/12/03 23:33:27 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/03 23:33:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLNestedForLoop (123.15s)
=== RUN   TestN1QLAutoClose
2021/12/03 23:33:32 Waiting for app: TestN1QLAutoClose to get un-deployed
2021/12/03 23:33:32 App: TestN1QLAutoClose got un-deployed
2021/12/03 23:33:32 No eventing-con process running
2021/12/03 23:33:32 Function: TestN1QLAutoClose delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLAutoClose not found"
 }
}
2021/12/03 23:33:34 Context: Bucket flush, response code: 200 dump: 
2021/12/03 23:33:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 23:33:41 Context: Bucket flush, response code: 200 dump: 
2021/12/03 23:33:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 23:33:46 Deploying app: TestN1QLAutoClose. Item count src bucket: 0 dst bucket: 0
2021/12/03 23:33:47 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAutoClose' in metakv",
  "warnings": null
 }
}
2021/12/03 23:33:52 Waiting for app: TestN1QLAutoClose to get deployed
2021/12/03 23:33:57 Waiting for app: TestN1QLAutoClose to get deployed
2021/12/03 23:34:02 Waiting for app: TestN1QLAutoClose to get deployed
2021/12/03 23:34:07 Waiting for app: TestN1QLAutoClose to get deployed
2021/12/03 23:34:07 App: TestN1QLAutoClose got deployed
2021/12/03 23:34:07 Starting bucket ops to source bucket
2021/12/03 23:34:12 Waiting for dst bucket item count to get to: 500 curr count: 0
2021/12/03 23:34:17 Waiting for dst bucket item count to get to: 500 curr count: 1
2021/12/03 23:34:22 Waiting for dst bucket item count to get to: 500 curr count: 1
2021/12/03 23:34:27 Waiting for dst bucket item count to get to: 500 curr count: 24
2021/12/03 23:34:32 Waiting for dst bucket item count to get to: 500 curr count: 24
2021/12/03 23:34:37 Waiting for dst bucket item count to get to: 500 curr count: 46
2021/12/03 23:34:42 Waiting for dst bucket item count to get to: 500 curr count: 46
2021/12/03 23:34:47 Waiting for dst bucket item count to get to: 500 curr count: 68
2021/12/03 23:34:52 Waiting for dst bucket item count to get to: 500 curr count: 68
2021/12/03 23:34:57 Waiting for dst bucket item count to get to: 500 curr count: 91
2021/12/03 23:35:02 Waiting for dst bucket item count to get to: 500 curr count: 91
2021/12/03 23:35:07 Waiting for dst bucket item count to get to: 500 curr count: 111
2021/12/03 23:35:12 Waiting for dst bucket item count to get to: 500 curr count: 111
2021/12/03 23:35:17 Waiting for dst bucket item count to get to: 500 curr count: 123
2021/12/03 23:35:22 Waiting for dst bucket item count to get to: 500 curr count: 133
2021/12/03 23:35:27 Waiting for dst bucket item count to get to: 500 curr count: 144
2021/12/03 23:35:33 Waiting for dst bucket item count to get to: 500 curr count: 155
2021/12/03 23:35:38 Waiting for dst bucket item count to get to: 500 curr count: 165
2021/12/03 23:35:43 Waiting for dst bucket item count to get to: 500 curr count: 178
2021/12/03 23:35:48 Waiting for dst bucket item count to get to: 500 curr count: 190
2021/12/03 23:35:53 Waiting for dst bucket item count to get to: 500 curr count: 200
2021/12/03 23:35:58 Waiting for dst bucket item count to get to: 500 curr count: 210
2021/12/03 23:36:03 Waiting for dst bucket item count to get to: 500 curr count: 221
2021/12/03 23:36:08 Waiting for dst bucket item count to get to: 500 curr count: 231
2021/12/03 23:36:13 Waiting for dst bucket item count to get to: 500 curr count: 242
2021/12/03 23:36:18 Waiting for dst bucket item count to get to: 500 curr count: 251
2021/12/03 23:36:23 Waiting for dst bucket item count to get to: 500 curr count: 262
2021/12/03 23:36:28 Waiting for dst bucket item count to get to: 500 curr count: 271
2021/12/03 23:36:33 Waiting for dst bucket item count to get to: 500 curr count: 283
2021/12/03 23:36:38 Waiting for dst bucket item count to get to: 500 curr count: 294
2021/12/03 23:36:43 Waiting for dst bucket item count to get to: 500 curr count: 303
2021/12/03 23:36:48 Waiting for dst bucket item count to get to: 500 curr count: 316
2021/12/03 23:36:53 Waiting for dst bucket item count to get to: 500 curr count: 324
2021/12/03 23:36:58 Waiting for dst bucket item count to get to: 500 curr count: 335
2021/12/03 23:37:03 Waiting for dst bucket item count to get to: 500 curr count: 345
2021/12/03 23:37:08 Waiting for dst bucket item count to get to: 500 curr count: 366
2021/12/03 23:37:13 Waiting for dst bucket item count to get to: 500 curr count: 366
2021/12/03 23:37:18 Waiting for dst bucket item count to get to: 500 curr count: 386
2021/12/03 23:37:23 Waiting for dst bucket item count to get to: 500 curr count: 386
2021/12/03 23:37:28 Waiting for dst bucket item count to get to: 500 curr count: 407
2021/12/03 23:37:33 Waiting for dst bucket item count to get to: 500 curr count: 407
2021/12/03 23:37:38 Waiting for dst bucket item count to get to: 500 curr count: 429
2021/12/03 23:37:43 Waiting for dst bucket item count to get to: 500 curr count: 429
2021/12/03 23:37:48 Waiting for dst bucket item count to get to: 500 curr count: 452
2021/12/03 23:37:53 Waiting for dst bucket item count to get to: 500 curr count: 452
2021/12/03 23:37:58 Waiting for dst bucket item count to get to: 500 curr count: 473
2021/12/03 23:38:03 Waiting for dst bucket item count to get to: 500 curr count: 473
2021/12/03 23:38:08 Waiting for dst bucket item count to get to: 500 curr count: 494
2021/12/03 23:38:13 Waiting for dst bucket item count to get to: 500 curr count: 494
2021/12/03 23:38:13 src & dst bucket item count matched up. src bucket count: 500 dst bucket count: 500
2021/12/03 23:38:13 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 23:38:13 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 23:38:18 Waiting for app: TestN1QLAutoClose to get un-deployed
2021/12/03 23:38:23 Waiting for app: TestN1QLAutoClose to get un-deployed
2021/12/03 23:38:28 Waiting for app: TestN1QLAutoClose to get un-deployed
2021/12/03 23:38:33 Waiting for app: TestN1QLAutoClose to get un-deployed
2021/12/03 23:38:33 App: TestN1QLAutoClose got un-deployed
2021/12/03 23:38:33 No eventing-con process running
2021/12/03 23:38:33 Function: TestN1QLAutoClose delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLAutoClose deleting in the background"
}
2021/12/03 23:38:35 Context: Bucket flush, response code: 200 dump: 
2021/12/03 23:38:45 Waiting for dst bucket item count to get to: 0 curr count: 500
2021/12/03 23:38:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 23:38:47 Context: Bucket flush, response code: 200 dump: 
2021/12/03 23:38:57 Waiting for dst bucket item count to get to: 0 curr count: 500
2021/12/03 23:38:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLAutoClose (329.81s)
=== RUN   TestN1QLPosParams
2021/12/03 23:39:02 Waiting for app: TestN1QLPosParams to get un-deployed
2021/12/03 23:39:02 App: TestN1QLPosParams got un-deployed
2021/12/03 23:39:02 No eventing-con process running
2021/12/03 23:39:02 Function: TestN1QLPosParams delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLPosParams not found"
 }
}
2021/12/03 23:39:04 Context: Bucket flush, response code: 200 dump: 
2021/12/03 23:39:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 23:39:10 Context: Bucket flush, response code: 200 dump: 
2021/12/03 23:39:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 23:39:15 Deploying app: TestN1QLPosParams. Item count src bucket: 0 dst bucket: 0
2021/12/03 23:39:16 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLPosParams' in metakv",
  "warnings": null
 }
}
2021/12/03 23:39:21 Waiting for app: TestN1QLPosParams to get deployed
2021/12/03 23:39:26 Waiting for app: TestN1QLPosParams to get deployed
2021/12/03 23:39:31 Waiting for app: TestN1QLPosParams to get deployed
2021/12/03 23:39:36 Waiting for app: TestN1QLPosParams to get deployed
2021/12/03 23:39:36 App: TestN1QLPosParams got deployed
2021/12/03 23:39:36 Starting bucket ops to source bucket
2021/12/03 23:39:46 Waiting for dst bucket item count to get to: 5000 curr count: 543
2021/12/03 23:39:51 Waiting for dst bucket item count to get to: 5000 curr count: 543
2021/12/03 23:39:51 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/03 23:39:51 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 23:39:51 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 23:39:56 Waiting for app: TestN1QLPosParams to get un-deployed
2021/12/03 23:40:01 Waiting for app: TestN1QLPosParams to get un-deployed
2021/12/03 23:40:06 Waiting for app: TestN1QLPosParams to get un-deployed
2021/12/03 23:40:11 Waiting for app: TestN1QLPosParams to get un-deployed
2021/12/03 23:40:11 App: TestN1QLPosParams got un-deployed
2021/12/03 23:40:11 No eventing-con process running
2021/12/03 23:40:11 Function: TestN1QLPosParams delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLPosParams deleting in the background"
}
2021/12/03 23:40:13 Context: Bucket flush, response code: 200 dump: 
2021/12/03 23:40:23 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/03 23:40:28 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/03 23:40:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 23:40:30 Context: Bucket flush, response code: 200 dump: 
2021/12/03 23:40:40 Waiting for dst bucket item count to get to: 0 curr count: 3088
2021/12/03 23:40:45 Waiting for dst bucket item count to get to: 0 curr count: 3088
2021/12/03 23:40:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLPosParams (108.32s)
=== RUN   TestN1QLExhaustConnPool
2021/12/03 23:40:50 Waiting for app: TestN1QLExhaustConnPool to get un-deployed
2021/12/03 23:40:50 App: TestN1QLExhaustConnPool got un-deployed
2021/12/03 23:40:50 No eventing-con process running
2021/12/03 23:40:50 Function: TestN1QLExhaustConnPool delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLExhaustConnPool not found"
 }
}
2021/12/03 23:40:52 Context: Bucket flush, response code: 200 dump: 
2021/12/03 23:40:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 23:40:59 Context: Bucket flush, response code: 200 dump: 
2021/12/03 23:41:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 23:41:04 Deploying app: TestN1QLExhaustConnPool. Item count src bucket: 0 dst bucket: 0
2021/12/03 23:41:05 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLExhaustConnPool' in metakv",
  "warnings": null
 }
}
2021/12/03 23:41:10 Waiting for app: TestN1QLExhaustConnPool to get deployed
2021/12/03 23:41:15 Waiting for app: TestN1QLExhaustConnPool to get deployed
2021/12/03 23:41:20 Waiting for app: TestN1QLExhaustConnPool to get deployed
2021/12/03 23:41:25 Waiting for app: TestN1QLExhaustConnPool to get deployed
2021/12/03 23:41:25 App: TestN1QLExhaustConnPool got deployed
2021/12/03 23:41:25 Starting bucket ops to source bucket
2021/12/03 23:41:30 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/03 23:41:35 Waiting for dst bucket item count to get to: 100 curr count: 29
2021/12/03 23:41:40 Waiting for dst bucket item count to get to: 100 curr count: 79
2021/12/03 23:41:40 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/03 23:41:40 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 23:41:40 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 23:41:45 Waiting for app: TestN1QLExhaustConnPool to get un-deployed
2021/12/03 23:41:50 Waiting for app: TestN1QLExhaustConnPool to get un-deployed
2021/12/03 23:41:55 Waiting for app: TestN1QLExhaustConnPool to get un-deployed
2021/12/03 23:42:00 Waiting for app: TestN1QLExhaustConnPool to get un-deployed
2021/12/03 23:42:00 App: TestN1QLExhaustConnPool got un-deployed
2021/12/03 23:42:00 No eventing-con process running
2021/12/03 23:42:00 Function: TestN1QLExhaustConnPool delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLExhaustConnPool deleting in the background"
}
2021/12/03 23:42:02 Context: Bucket flush, response code: 200 dump: 
2021/12/03 23:42:12 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/03 23:42:17 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/03 23:42:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 23:42:19 Context: Bucket flush, response code: 200 dump: 
2021/12/03 23:42:29 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/03 23:42:34 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/03 23:42:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLExhaustConnPool (109.02s)
=== RUN   TestN1QLGraceSufficientTimeOut
2021/12/03 23:42:39 Waiting for app: TestN1QLGraceSufficientTimeOut to get un-deployed
2021/12/03 23:42:39 App: TestN1QLGraceSufficientTimeOut got un-deployed
2021/12/03 23:42:39 No eventing-con process running
2021/12/03 23:42:39 Function: TestN1QLGraceSufficientTimeOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLGraceSufficientTimeOut not found"
 }
}
2021/12/03 23:42:41 Context: Bucket flush, response code: 200 dump: 
2021/12/03 23:42:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 23:42:48 Context: Bucket flush, response code: 200 dump: 
2021/12/03 23:42:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 23:42:53 Starting bucket ops to source bucket
2021/12/03 23:43:10 Starting bucket ops to source bucket
2021/12/03 23:43:28 Deploying app: TestN1QLGraceSufficientTimeOut. Item count src bucket: 30000 dst bucket: 13537
2021/12/03 23:43:29 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLGraceSufficientTimeOut' in metakv",
  "warnings": null
 }
}
2021/12/03 23:43:34 Waiting for app: TestN1QLGraceSufficientTimeOut to get deployed
2021/12/03 23:43:39 Waiting for app: TestN1QLGraceSufficientTimeOut to get deployed
2021/12/03 23:43:44 Waiting for app: TestN1QLGraceSufficientTimeOut to get deployed
2021/12/03 23:43:49 Waiting for app: TestN1QLGraceSufficientTimeOut to get deployed
2021/12/03 23:43:49 App: TestN1QLGraceSufficientTimeOut got deployed
2021/12/03 23:43:49 Starting bucket ops to source bucket
2021/12/03 23:43:54 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/03 23:43:59 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/03 23:44:04 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/03 23:44:09 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/03 23:44:14 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/03 23:44:19 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/03 23:44:24 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/03 23:44:29 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/03 23:44:34 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/03 23:44:39 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/03 23:44:44 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/03 23:44:44 src & dst bucket item count matched up. src bucket count: 30001 dst bucket count: 30001
2021/12/03 23:44:44 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 23:44:45 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 23:44:50 Waiting for app: TestN1QLGraceSufficientTimeOut to get un-deployed
2021/12/03 23:44:55 Waiting for app: TestN1QLGraceSufficientTimeOut to get un-deployed
2021/12/03 23:45:00 Waiting for app: TestN1QLGraceSufficientTimeOut to get un-deployed
2021/12/03 23:45:05 Waiting for app: TestN1QLGraceSufficientTimeOut to get un-deployed
2021/12/03 23:45:05 App: TestN1QLGraceSufficientTimeOut got un-deployed
2021/12/03 23:45:05 No eventing-con process running
2021/12/03 23:45:05 Function: TestN1QLGraceSufficientTimeOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLGraceSufficientTimeOut deleting in the background"
}
2021/12/03 23:45:06 Context: Bucket flush, response code: 200 dump: 
2021/12/03 23:45:16 Waiting for dst bucket item count to get to: 0 curr count: 30000
2021/12/03 23:45:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 23:45:18 Context: Bucket flush, response code: 200 dump: 
2021/12/03 23:45:28 Waiting for dst bucket item count to get to: 0 curr count: 25315
2021/12/03 23:45:33 Waiting for dst bucket item count to get to: 0 curr count: 10315
2021/12/03 23:45:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLGraceSufficientTimeOut (179.36s)
=== RUN   TestN1QLQueryConsistency
2021/12/03 23:45:38 Waiting for app: n1qlquery_consistency to get un-deployed
2021/12/03 23:45:38 App: n1qlquery_consistency got un-deployed
2021/12/03 23:45:38 No eventing-con process running
2021/12/03 23:45:38 Function: n1qlquery_consistency delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: n1qlquery_consistency not found"
 }
}
2021/12/03 23:45:40 Context: Bucket flush, response code: 200 dump: 
2021/12/03 23:45:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 23:45:47 Context: Bucket flush, response code: 200 dump: 
2021/12/03 23:45:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 23:45:52 Starting bucket ops to source bucket
2021/12/03 23:45:55 Deploying app: n1qlquery_consistency. Item count src bucket: 0 dst bucket: 0
2021/12/03 23:45:56 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'n1qlquery_consistency' in metakv",
  "warnings": null
 }
}
2021/12/03 23:46:01 Waiting for app: n1qlquery_consistency to get deployed
2021/12/03 23:46:06 Waiting for app: n1qlquery_consistency to get deployed
2021/12/03 23:46:11 Waiting for app: n1qlquery_consistency to get deployed
2021/12/03 23:46:16 Waiting for app: n1qlquery_consistency to get deployed
2021/12/03 23:46:16 App: n1qlquery_consistency got deployed
2021/12/03 23:46:21 Waiting for dst bucket item count to get to: 5000 curr count: 147
2021/12/03 23:46:26 Waiting for dst bucket item count to get to: 5000 curr count: 2647
2021/12/03 23:46:26 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/03 23:46:26 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 23:46:26 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 23:46:31 Waiting for app: n1qlquery_consistency to get un-deployed
2021/12/03 23:46:36 Waiting for app: n1qlquery_consistency to get un-deployed
2021/12/03 23:46:41 Waiting for app: n1qlquery_consistency to get un-deployed
2021/12/03 23:46:46 Waiting for app: n1qlquery_consistency to get un-deployed
2021/12/03 23:46:47 App: n1qlquery_consistency got un-deployed
2021/12/03 23:46:47 No eventing-con process running
2021/12/03 23:46:47 Function: n1qlquery_consistency delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: n1qlquery_consistency deleting in the background"
}
2021/12/03 23:46:48 Context: Bucket flush, response code: 200 dump: 
2021/12/03 23:46:58 Waiting for dst bucket item count to get to: 0 curr count: 3888
2021/12/03 23:47:03 Waiting for dst bucket item count to get to: 0 curr count: 1388
2021/12/03 23:47:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 23:47:05 Context: Bucket flush, response code: 200 dump: 
2021/12/03 23:47:15 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/03 23:47:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLQueryConsistency (101.83s)
PASS
ok  	github.com/couchbase/eventing/tests/functional_tests	1570.597s

N1ql tests took 0 hours, 26 minutes and 14 seconds.

Starting server
Starting server: attempt 1

Rebalance tests

2021/12/03 23:48:50 Started rebalance
Post "http://127.0.0.1:9000/pools/default": read tcp 127.0.0.1:50694->127.0.0.1:9000: read: connection reset by peer
Quota setup Post "http://127.0.0.1:9000/pools/default": read tcp 127.0.0.1:50694->127.0.0.1:9000: read: connection reset by peer
=== RUN   TestDuplicateWithEventingRebIn
2021/12/03 23:49:09 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
2021/12/03 23:49:09 App: TestDuplicateWithEventingRebIn got un-deployed
2021/12/03 23:49:09 No eventing-con process running
2021/12/03 23:49:09 Function: TestDuplicateWithEventingRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithEventingRebIn not found"
 }
}
2021/12/03 23:49:10 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 23:49:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 23:49:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 23:49:17 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 23:49:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 23:49:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 23:49:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 23:49:27 Deploying app: TestDuplicateWithEventingRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/03 23:49:28 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithEventingRebIn' in metakv",
  "warnings": null
 }
}
2021/12/03 23:49:33 Waiting for app: TestDuplicateWithEventingRebIn to get deployed
2021/12/03 23:49:38 Waiting for app: TestDuplicateWithEventingRebIn to get deployed
2021/12/03 23:49:43 Waiting for app: TestDuplicateWithEventingRebIn to get deployed
2021/12/03 23:49:48 Waiting for app: TestDuplicateWithEventingRebIn to get deployed
2021/12/03 23:49:48 App: TestDuplicateWithEventingRebIn got deployed
2021/12/03 23:49:48 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/03 23:49:48 Starting bucket ops to source bucket
2021/12/03 23:50:03 Removing node(s): [] from the cluster
2021/12/03 23:50:08 Rebalance progress: 25
2021/12/03 23:50:13 Rebalance progress: 25
2021/12/03 23:50:18 Rebalance progress: 25
2021/12/03 23:50:23 Rebalance progress: 25
2021/12/03 23:50:28 Rebalance progress: 75
2021/12/03 23:50:33 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 23:50:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 23:50:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 23:50:33 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9302/api/v1/stats": dial tcp 127.0.0.1:9302: connect: connection refused
2021/12/03 23:50:33 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/03 23:50:33 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/03 23:50:38 Rebalance progress: 25
2021/12/03 23:50:43 Rebalance progress: 25
2021/12/03 23:50:48 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 23:50:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 23:50:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 23:50:48 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9302/api/v1/stats": dial tcp 127.0.0.1:9302: connect: connection refused
2021/12/03 23:50:48 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/03 23:50:48 Waiting for src bucket item count to get to: 200000 curr count: 77797
2021/12/03 23:50:53 Waiting for src bucket item count to get to: 200000 curr count: 77797
2021/12/03 23:50:58 Waiting for src bucket item count to get to: 200000 curr count: 94073
2021/12/03 23:51:03 Waiting for src bucket item count to get to: 200000 curr count: 94073
2021/12/03 23:51:08 Waiting for src bucket item count to get to: 200000 curr count: 110520
2021/12/03 23:51:13 Waiting for src bucket item count to get to: 200000 curr count: 110520
2021/12/03 23:51:18 Waiting for src bucket item count to get to: 200000 curr count: 126751
2021/12/03 23:51:23 Waiting for src bucket item count to get to: 200000 curr count: 126751
2021/12/03 23:51:28 Waiting for src bucket item count to get to: 200000 curr count: 142977
2021/12/03 23:51:33 Waiting for src bucket item count to get to: 200000 curr count: 142977
2021/12/03 23:51:38 Waiting for src bucket item count to get to: 200000 curr count: 159199
2021/12/03 23:51:43 Waiting for src bucket item count to get to: 200000 curr count: 159199
2021/12/03 23:51:48 Waiting for src bucket item count to get to: 200000 curr count: 175211
2021/12/03 23:51:53 Waiting for src bucket item count to get to: 200000 curr count: 191462
2021/12/03 23:51:58 Waiting for src bucket item count to get to: 200000 curr count: 191462
2021/12/03 23:52:08 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/03 23:52:13 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/03 23:52:18 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/03 23:52:23 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/03 23:52:28 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/03 23:52:33 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/03 23:52:38 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/03 23:52:43 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/03 23:52:48 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/03 23:52:53 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/03 23:52:53 Post SET ops. Expected item count: 200000 got 200001
2021/12/03 23:52:53 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 23:52:58 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
2021/12/03 23:53:03 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
2021/12/03 23:53:08 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
2021/12/03 23:53:13 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
2021/12/03 23:53:13 App: TestDuplicateWithEventingRebIn got un-deployed
2021/12/03 23:53:13 No eventing-con process running
2021/12/03 23:53:13 Function: TestDuplicateWithEventingRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithEventingRebIn deleting in the background"
}
2021/12/03 23:53:15 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 23:53:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 23:53:25 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/03 23:53:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 23:53:27 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 23:53:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 23:53:37 Waiting for dst bucket item count to get to: 0 curr count: 200001
2021/12/03 23:53:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithEventingRebIn (277.95s)
=== RUN   TestDuplicateWithEventingRebOut
2021/12/03 23:53:42 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/03 23:53:42 Removing node(s): [] from the cluster
2021/12/03 23:53:47 Rebalance progress: 100
2021/12/03 23:53:52 Waiting for app: TestDuplicateWithEventingRebOut to get un-deployed
2021/12/03 23:53:52 App: TestDuplicateWithEventingRebOut got un-deployed
2021/12/03 23:53:52 No eventing-con process running
2021/12/03 23:53:52 Function: TestDuplicateWithEventingRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithEventingRebOut not found"
 }
}
2021/12/03 23:53:54 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 23:53:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 23:53:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 23:54:01 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 23:54:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 23:54:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 23:54:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 23:54:11 Deploying app: TestDuplicateWithEventingRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/03 23:54:12 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithEventingRebOut' in metakv",
  "warnings": null
 }
}
2021/12/03 23:54:17 Waiting for app: TestDuplicateWithEventingRebOut to get deployed
2021/12/03 23:54:22 Waiting for app: TestDuplicateWithEventingRebOut to get deployed
2021/12/03 23:54:27 Waiting for app: TestDuplicateWithEventingRebOut to get deployed
2021/12/03 23:54:32 Waiting for app: TestDuplicateWithEventingRebOut to get deployed
2021/12/03 23:54:32 App: TestDuplicateWithEventingRebOut got deployed
2021/12/03 23:54:32 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/03 23:54:32 Starting bucket ops to source bucket
2021/12/03 23:54:37 Rebalance progress: 25
2021/12/03 23:54:42 Rebalance progress: 25
2021/12/03 23:54:47 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 23:54:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 23:54:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 23:54:47 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9302/api/v1/stats": dial tcp 127.0.0.1:9302: connect: connection refused
2021/12/03 23:54:47 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/03 23:54:47 Waiting for src bucket item count to get to: 200000 curr count: 11743
2021/12/03 23:54:52 Waiting for src bucket item count to get to: 200000 curr count: 28114
2021/12/03 23:54:57 Waiting for src bucket item count to get to: 200000 curr count: 28114
2021/12/03 23:55:02 Waiting for src bucket item count to get to: 200000 curr count: 44573
2021/12/03 23:55:07 Waiting for src bucket item count to get to: 200000 curr count: 44573
2021/12/03 23:55:12 Waiting for src bucket item count to get to: 200000 curr count: 60954
2021/12/03 23:55:17 Waiting for src bucket item count to get to: 200000 curr count: 60954
2021/12/03 23:55:22 Waiting for src bucket item count to get to: 200000 curr count: 60954
2021/12/03 23:55:27 Waiting for src bucket item count to get to: 200000 curr count: 76999
2021/12/03 23:55:32 Waiting for src bucket item count to get to: 200000 curr count: 76999
2021/12/03 23:55:37 Waiting for src bucket item count to get to: 200000 curr count: 93057
2021/12/03 23:55:42 Waiting for src bucket item count to get to: 200000 curr count: 93057
2021/12/03 23:55:47 Waiting for src bucket item count to get to: 200000 curr count: 109591
2021/12/03 23:55:52 Waiting for src bucket item count to get to: 200000 curr count: 109591
2021/12/03 23:55:57 Waiting for src bucket item count to get to: 200000 curr count: 126068
2021/12/03 23:56:02 Waiting for src bucket item count to get to: 200000 curr count: 126068
2021/12/03 23:56:07 Waiting for src bucket item count to get to: 200000 curr count: 142531
2021/12/03 23:56:12 Waiting for src bucket item count to get to: 200000 curr count: 142531
2021/12/03 23:56:17 Waiting for src bucket item count to get to: 200000 curr count: 158908
2021/12/03 23:56:22 Waiting for src bucket item count to get to: 200000 curr count: 158908
2021/12/03 23:56:27 Waiting for src bucket item count to get to: 200000 curr count: 174939
2021/12/03 23:56:32 Waiting for src bucket item count to get to: 200000 curr count: 174939
2021/12/03 23:56:37 Waiting for src bucket item count to get to: 200000 curr count: 191044
2021/12/03 23:56:42 Waiting for src bucket item count to get to: 200000 curr count: 191044
2021/12/03 23:56:47 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/03 23:56:47 Post SET ops. Expected item count: 200000 got 200000
2021/12/03 23:56:47 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 23:56:52 Waiting for app: TestDuplicateWithEventingRebOut to get un-deployed
2021/12/03 23:56:57 Waiting for app: TestDuplicateWithEventingRebOut to get un-deployed
2021/12/03 23:57:02 Waiting for app: TestDuplicateWithEventingRebOut to get un-deployed
2021/12/03 23:57:07 Waiting for app: TestDuplicateWithEventingRebOut to get un-deployed
2021/12/03 23:57:07 App: TestDuplicateWithEventingRebOut got un-deployed
2021/12/03 23:57:07 No eventing-con process running
2021/12/03 23:57:07 Function: TestDuplicateWithEventingRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithEventingRebOut deleting in the background"
}
2021/12/03 23:57:09 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 23:57:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 23:57:19 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/03 23:57:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 23:57:21 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 23:57:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 23:57:31 Waiting for dst bucket item count to get to: 0 curr count: 120177
2021/12/03 23:57:36 Waiting for dst bucket item count to get to: 0 curr count: 120177
2021/12/03 23:57:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithEventingRebOut (238.87s)
=== RUN   TestDuplicateWithEventingSwapReb
2021/12/03 23:57:46 Waiting for app: TestDuplicateWithEventingSwapReb to get un-deployed
2021/12/03 23:57:46 App: TestDuplicateWithEventingSwapReb got un-deployed
2021/12/03 23:57:46 No eventing-con process running
2021/12/03 23:57:46 Function: TestDuplicateWithEventingSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithEventingSwapReb not found"
 }
}
2021/12/03 23:57:47 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 23:57:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 23:57:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 23:57:54 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 23:57:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 23:57:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 23:58:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 23:58:04 Deploying app: TestDuplicateWithEventingSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/03 23:58:05 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithEventingSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/03 23:58:10 Waiting for app: TestDuplicateWithEventingSwapReb to get deployed
2021/12/03 23:58:15 Waiting for app: TestDuplicateWithEventingSwapReb to get deployed
2021/12/03 23:58:20 Waiting for app: TestDuplicateWithEventingSwapReb to get deployed
2021/12/03 23:58:25 Waiting for app: TestDuplicateWithEventingSwapReb to get deployed
2021/12/03 23:58:25 App: TestDuplicateWithEventingSwapReb got deployed
2021/12/03 23:58:25 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/03 23:58:25 Starting bucket ops to source bucket
2021/12/03 23:58:25 Removing node(s): [] from the cluster
2021/12/03 23:58:31 Rebalance progress: 25
2021/12/03 23:58:36 Rebalance progress: 25
2021/12/03 23:58:41 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 23:58:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 23:58:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 23:58:41 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9302/api/v1/stats": dial tcp 127.0.0.1:9302: connect: connection refused
2021/12/03 23:58:41 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/03 23:58:41 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/03 23:58:48 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/03 23:58:53 Rebalance progress: 16.66666666666667
2021/12/03 23:58:58 Rebalance progress: 16.66666666666667
2021/12/03 23:59:03 Rebalance progress: 16.66666666666667
2021/12/03 23:59:08 Rebalance progress: 72.22222222222223
2021/12/03 23:59:13 Rebalance progress: 72.22222222222223
2021/12/03 23:59:18 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 23:59:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 23:59:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 23:59:18 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/03 23:59:18 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/03 23:59:23 Rebalance progress: 16.66666666666667
2021/12/03 23:59:28 Rebalance progress: 16.66666666666667
2021/12/03 23:59:33 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 23:59:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 23:59:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 23:59:33 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/03 23:59:33 Waiting for src bucket item count to get to: 200000 curr count: 96268
2021/12/03 23:59:38 Waiting for src bucket item count to get to: 200000 curr count: 96268
2021/12/03 23:59:43 Waiting for src bucket item count to get to: 200000 curr count: 112329
2021/12/03 23:59:48 Waiting for src bucket item count to get to: 200000 curr count: 112329
2021/12/03 23:59:53 Waiting for src bucket item count to get to: 200000 curr count: 128347
2021/12/03 23:59:58 Waiting for src bucket item count to get to: 200000 curr count: 128347
2021/12/04 00:00:03 Waiting for src bucket item count to get to: 200000 curr count: 144184
2021/12/04 00:00:08 Waiting for src bucket item count to get to: 200000 curr count: 144184
2021/12/04 00:00:13 Waiting for src bucket item count to get to: 200000 curr count: 160198
2021/12/04 00:00:18 Waiting for src bucket item count to get to: 200000 curr count: 160198
2021/12/04 00:00:23 Waiting for src bucket item count to get to: 200000 curr count: 176299
2021/12/04 00:00:28 Waiting for src bucket item count to get to: 200000 curr count: 176299
2021/12/04 00:00:33 Waiting for src bucket item count to get to: 200000 curr count: 192355
2021/12/04 00:00:38 Waiting for src bucket item count to get to: 200000 curr count: 192355
2021/12/04 00:00:43 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/04 00:00:43 Post SET ops. Expected item count: 200000 got 200000
2021/12/04 00:00:43 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 00:00:48 Waiting for app: TestDuplicateWithEventingSwapReb to get un-deployed
2021/12/04 00:00:53 Waiting for app: TestDuplicateWithEventingSwapReb to get un-deployed
2021/12/04 00:00:58 Waiting for app: TestDuplicateWithEventingSwapReb to get un-deployed
2021/12/04 00:01:03 Waiting for app: TestDuplicateWithEventingSwapReb to get un-deployed
2021/12/04 00:01:03 App: TestDuplicateWithEventingSwapReb got un-deployed
2021/12/04 00:01:03 No eventing-con process running
2021/12/04 00:01:03 Function: TestDuplicateWithEventingSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithEventingSwapReb deleting in the background"
}
2021/12/04 00:01:05 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:01:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 00:01:15 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/04 00:01:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 00:01:17 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:01:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 00:01:27 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/04 00:01:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithEventingSwapReb (231.09s)
=== RUN   TestDuplicateWithKVRebIn
2021/12/04 00:01:37 Waiting for app: TestDuplicateWithKVRebIn to get un-deployed
2021/12/04 00:01:37 App: TestDuplicateWithKVRebIn got un-deployed
2021/12/04 00:01:37 No eventing-con process running
2021/12/04 00:01:37 Function: TestDuplicateWithKVRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithKVRebIn not found"
 }
}
2021/12/04 00:01:38 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:01:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 00:01:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 00:01:45 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:01:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 00:01:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:01:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 00:01:55 Deploying app: TestDuplicateWithKVRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/04 00:01:56 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithKVRebIn' in metakv",
  "warnings": null
 }
}
2021/12/04 00:02:01 Waiting for app: TestDuplicateWithKVRebIn to get deployed
2021/12/04 00:02:06 Waiting for app: TestDuplicateWithKVRebIn to get deployed
2021/12/04 00:02:11 Waiting for app: TestDuplicateWithKVRebIn to get deployed
2021/12/04 00:02:16 Waiting for app: TestDuplicateWithKVRebIn to get deployed
2021/12/04 00:02:16 App: TestDuplicateWithKVRebIn got deployed
2021/12/04 00:02:16 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/04 00:02:16 Starting bucket ops to source bucket
2021/12/04 00:02:17 Removing node(s): [] from the cluster
2021/12/04 00:02:22 Rebalance progress: 16.66666666666667
2021/12/04 00:02:27 Rebalance progress: 16.66666666666667
2021/12/04 00:02:32 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:02:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:02:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 00:02:32 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/04 00:02:32 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/04 00:02:37 Rebalance progress: 16.66666666666667
2021/12/04 00:02:42 Rebalance progress: 16.66666666666667
2021/12/04 00:02:47 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:02:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:02:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 00:02:47 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/04 00:02:47 Waiting for src bucket item count to get to: 200000 curr count: 35083
2021/12/04 00:02:52 Waiting for src bucket item count to get to: 200000 curr count: 50801
2021/12/04 00:02:57 Waiting for src bucket item count to get to: 200000 curr count: 50801
2021/12/04 00:03:02 Waiting for src bucket item count to get to: 200000 curr count: 66666
2021/12/04 00:03:07 Waiting for src bucket item count to get to: 200000 curr count: 66666
2021/12/04 00:03:12 Waiting for src bucket item count to get to: 200000 curr count: 81771
2021/12/04 00:03:17 Waiting for src bucket item count to get to: 200000 curr count: 81771
2021/12/04 00:03:22 Waiting for src bucket item count to get to: 200000 curr count: 81771
2021/12/04 00:03:27 Waiting for src bucket item count to get to: 200000 curr count: 97320
2021/12/04 00:03:32 Waiting for src bucket item count to get to: 200000 curr count: 97320
2021/12/04 00:03:37 Waiting for src bucket item count to get to: 200000 curr count: 113214
2021/12/04 00:03:42 Waiting for src bucket item count to get to: 200000 curr count: 113214
2021/12/04 00:03:47 Waiting for src bucket item count to get to: 200000 curr count: 129489
2021/12/04 00:03:52 Waiting for src bucket item count to get to: 200000 curr count: 129489
2021/12/04 00:03:57 Waiting for src bucket item count to get to: 200000 curr count: 145331
2021/12/04 00:04:02 Waiting for src bucket item count to get to: 200000 curr count: 145331
2021/12/04 00:04:07 Waiting for src bucket item count to get to: 200000 curr count: 161208
2021/12/04 00:04:12 Waiting for src bucket item count to get to: 200000 curr count: 161208
2021/12/04 00:04:17 Waiting for src bucket item count to get to: 200000 curr count: 176339
2021/12/04 00:04:22 Waiting for src bucket item count to get to: 200000 curr count: 176339
2021/12/04 00:04:27 Waiting for src bucket item count to get to: 200000 curr count: 191925
2021/12/04 00:04:32 Waiting for src bucket item count to get to: 200000 curr count: 191925
2021/12/04 00:04:37 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/04 00:04:37 Post SET ops. Expected item count: 200000 got 200000
2021/12/04 00:04:37 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 00:04:42 Waiting for app: TestDuplicateWithKVRebIn to get un-deployed
2021/12/04 00:04:47 Waiting for app: TestDuplicateWithKVRebIn to get un-deployed
2021/12/04 00:04:52 Waiting for app: TestDuplicateWithKVRebIn to get un-deployed
2021/12/04 00:04:57 Waiting for app: TestDuplicateWithKVRebIn to get un-deployed
2021/12/04 00:04:57 App: TestDuplicateWithKVRebIn got un-deployed
2021/12/04 00:04:57 No eventing-con process running
2021/12/04 00:04:57 Function: TestDuplicateWithKVRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithKVRebIn deleting in the background"
}
2021/12/04 00:04:59 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:05:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 00:05:09 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/04 00:05:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 00:05:11 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:05:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 00:05:21 Waiting for dst bucket item count to get to: 0 curr count: 123627
2021/12/04 00:05:26 Waiting for dst bucket item count to get to: 0 curr count: 123627
2021/12/04 00:05:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithKVRebIn (238.94s)
=== RUN   TestDuplicateWithKVRebOut
2021/12/04 00:05:31 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/04 00:05:31 Removing node(s): [] from the cluster
2021/12/04 00:05:36 Rebalance progress: 100
2021/12/04 00:05:41 Waiting for app: TestDuplicateWithKVRebOut to get un-deployed
2021/12/04 00:05:41 App: TestDuplicateWithKVRebOut got un-deployed
2021/12/04 00:05:41 No eventing-con process running
2021/12/04 00:05:41 Function: TestDuplicateWithKVRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithKVRebOut not found"
 }
}
2021/12/04 00:05:43 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:05:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 00:05:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 00:05:50 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:05:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 00:05:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:06:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 00:06:00 Deploying app: TestDuplicateWithKVRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/04 00:06:01 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithKVRebOut' in metakv",
  "warnings": null
 }
}
2021/12/04 00:06:06 Waiting for app: TestDuplicateWithKVRebOut to get deployed
2021/12/04 00:06:11 Waiting for app: TestDuplicateWithKVRebOut to get deployed
2021/12/04 00:06:16 Waiting for app: TestDuplicateWithKVRebOut to get deployed
2021/12/04 00:06:21 Waiting for app: TestDuplicateWithKVRebOut to get deployed
2021/12/04 00:06:21 App: TestDuplicateWithKVRebOut got deployed
2021/12/04 00:06:21 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/04 00:06:21 Starting bucket ops to source bucket
2021/12/04 00:06:26 Rebalance progress: 16.66666666666667
2021/12/04 00:06:31 Rebalance progress: 16.66666666666667
2021/12/04 00:06:36 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:06:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:06:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 00:06:36 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/04 00:06:36 Waiting for src bucket item count to get to: 200000 curr count: 12738
2021/12/04 00:06:41 Waiting for src bucket item count to get to: 200000 curr count: 29123
2021/12/04 00:06:46 Waiting for src bucket item count to get to: 200000 curr count: 29123
2021/12/04 00:06:51 Waiting for src bucket item count to get to: 200000 curr count: 45605
2021/12/04 00:06:56 Waiting for src bucket item count to get to: 200000 curr count: 45605
2021/12/04 00:07:01 Waiting for src bucket item count to get to: 200000 curr count: 61735
2021/12/04 00:07:06 Waiting for src bucket item count to get to: 200000 curr count: 61735
2021/12/04 00:07:11 Waiting for src bucket item count to get to: 200000 curr count: 61735
2021/12/04 00:07:16 Waiting for src bucket item count to get to: 200000 curr count: 77214
2021/12/04 00:07:21 Waiting for src bucket item count to get to: 200000 curr count: 77214
2021/12/04 00:07:26 Waiting for src bucket item count to get to: 200000 curr count: 93135
2021/12/04 00:07:31 Waiting for src bucket item count to get to: 200000 curr count: 93135
2021/12/04 00:07:36 Waiting for src bucket item count to get to: 200000 curr count: 109357
2021/12/04 00:07:41 Waiting for src bucket item count to get to: 200000 curr count: 109357
2021/12/04 00:07:46 Waiting for src bucket item count to get to: 200000 curr count: 125639
2021/12/04 00:07:51 Waiting for src bucket item count to get to: 200000 curr count: 125639
2021/12/04 00:07:56 Waiting for src bucket item count to get to: 200000 curr count: 142069
2021/12/04 00:08:01 Waiting for src bucket item count to get to: 200000 curr count: 142069
2021/12/04 00:08:06 Waiting for src bucket item count to get to: 200000 curr count: 158391
2021/12/04 00:08:11 Waiting for src bucket item count to get to: 200000 curr count: 158391
2021/12/04 00:08:16 Waiting for src bucket item count to get to: 200000 curr count: 174002
2021/12/04 00:08:21 Waiting for src bucket item count to get to: 200000 curr count: 174002
2021/12/04 00:08:26 Waiting for src bucket item count to get to: 200000 curr count: 190037
2021/12/04 00:08:31 Waiting for src bucket item count to get to: 200000 curr count: 190037
2021/12/04 00:08:36 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/04 00:08:36 Post SET ops. Expected item count: 200000 got 200000
2021/12/04 00:08:36 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 00:08:41 Waiting for app: TestDuplicateWithKVRebOut to get un-deployed
2021/12/04 00:08:46 Waiting for app: TestDuplicateWithKVRebOut to get un-deployed
2021/12/04 00:08:51 Waiting for app: TestDuplicateWithKVRebOut to get un-deployed
2021/12/04 00:08:56 Waiting for app: TestDuplicateWithKVRebOut to get un-deployed
2021/12/04 00:08:56 App: TestDuplicateWithKVRebOut got un-deployed
2021/12/04 00:08:56 No eventing-con process running
2021/12/04 00:08:57 Function: TestDuplicateWithKVRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithKVRebOut deleting in the background"
}
2021/12/04 00:08:58 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:09:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 00:09:08 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/04 00:09:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 00:09:10 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:09:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 00:09:20 Waiting for dst bucket item count to get to: 0 curr count: 35156
2021/12/04 00:09:25 Waiting for dst bucket item count to get to: 0 curr count: 35156
2021/12/04 00:09:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithKVRebOut (239.50s)
=== RUN   TestDuplicateWithKVSwapReb
2021/12/04 00:09:35 Waiting for app: TestDuplicateWithKVSwapReb to get un-deployed
2021/12/04 00:09:35 App: TestDuplicateWithKVSwapReb got un-deployed
2021/12/04 00:09:35 No eventing-con process running
2021/12/04 00:09:35 Function: TestDuplicateWithKVSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithKVSwapReb not found"
 }
}
2021/12/04 00:09:37 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:09:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 00:09:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 00:09:44 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:09:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 00:09:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:09:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 00:09:54 Deploying app: TestDuplicateWithKVSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/04 00:09:55 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithKVSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/04 00:10:00 Waiting for app: TestDuplicateWithKVSwapReb to get deployed
2021/12/04 00:10:05 Waiting for app: TestDuplicateWithKVSwapReb to get deployed
2021/12/04 00:10:10 Waiting for app: TestDuplicateWithKVSwapReb to get deployed
2021/12/04 00:10:15 Waiting for app: TestDuplicateWithKVSwapReb to get deployed
2021/12/04 00:10:15 App: TestDuplicateWithKVSwapReb got deployed
2021/12/04 00:10:15 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/04 00:10:15 Starting bucket ops to source bucket
2021/12/04 00:10:15 Removing node(s): [] from the cluster
2021/12/04 00:10:20 Rebalance progress: 16.66666666666667
2021/12/04 00:10:25 Rebalance progress: 16.66666666666667
2021/12/04 00:10:30 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:10:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:10:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 00:10:30 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/04 00:10:30 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/04 00:10:31 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/04 00:10:36 Rebalance progress: 16.66666666666667
2021/12/04 00:10:41 Rebalance progress: 16.66666666666667
2021/12/04 00:10:46 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:10:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:10:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 00:10:46 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/04 00:10:46 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/04 00:10:51 Rebalance progress: 16.66666666666667
2021/12/04 00:10:56 Rebalance progress: 16.66666666666667
2021/12/04 00:11:01 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:11:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:11:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 00:11:01 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/04 00:11:01 Waiting for src bucket item count to get to: 200000 curr count: 69759
2021/12/04 00:11:06 Waiting for src bucket item count to get to: 200000 curr count: 69759
2021/12/04 00:11:11 Waiting for src bucket item count to get to: 200000 curr count: 84916
2021/12/04 00:11:16 Waiting for src bucket item count to get to: 200000 curr count: 84916
2021/12/04 00:11:21 Waiting for src bucket item count to get to: 200000 curr count: 100555
2021/12/04 00:11:26 Waiting for src bucket item count to get to: 200000 curr count: 100555
2021/12/04 00:11:31 Waiting for src bucket item count to get to: 200000 curr count: 116457
2021/12/04 00:11:36 Waiting for src bucket item count to get to: 200000 curr count: 116457
2021/12/04 00:11:41 Waiting for src bucket item count to get to: 200000 curr count: 132585
2021/12/04 00:11:46 Waiting for src bucket item count to get to: 200000 curr count: 132585
2021/12/04 00:11:51 Waiting for src bucket item count to get to: 200000 curr count: 149059
2021/12/04 00:11:56 Waiting for src bucket item count to get to: 200000 curr count: 149059
2021/12/04 00:12:01 Waiting for src bucket item count to get to: 200000 curr count: 165378
2021/12/04 00:12:06 Waiting for src bucket item count to get to: 200000 curr count: 165378
2021/12/04 00:12:11 Waiting for src bucket item count to get to: 200000 curr count: 180961
2021/12/04 00:12:16 Waiting for src bucket item count to get to: 200000 curr count: 180961
2021/12/04 00:12:21 Waiting for src bucket item count to get to: 200000 curr count: 196897
2021/12/04 00:12:26 Waiting for src bucket item count to get to: 200000 curr count: 196897
2021/12/04 00:12:31 Waiting for src bucket item count to get to: 200000 curr count: 196897
2021/12/04 00:12:36 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/04 00:12:36 Post SET ops. Expected item count: 200000 got 200000
2021/12/04 00:12:36 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 00:12:41 Waiting for app: TestDuplicateWithKVSwapReb to get un-deployed
2021/12/04 00:12:46 Waiting for app: TestDuplicateWithKVSwapReb to get un-deployed
2021/12/04 00:12:51 Waiting for app: TestDuplicateWithKVSwapReb to get un-deployed
2021/12/04 00:12:56 Waiting for app: TestDuplicateWithKVSwapReb to get un-deployed
2021/12/04 00:12:56 App: TestDuplicateWithKVSwapReb got un-deployed
2021/12/04 00:12:56 No eventing-con process running
2021/12/04 00:12:56 Function: TestDuplicateWithKVSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithKVSwapReb deleting in the background"
}
2021/12/04 00:12:58 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:13:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 00:13:08 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/04 00:13:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 00:13:10 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:13:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 00:13:20 Waiting for dst bucket item count to get to: 0 curr count: 26669
2021/12/04 00:13:25 Waiting for dst bucket item count to get to: 0 curr count: 26669
2021/12/04 00:13:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithKVSwapReb (239.97s)
=== RUN   TestEventingRebKVOpsOnUpdateBucketOpOneByOne
2021/12/04 00:13:35 Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/04 00:13:35 App: TestEventingRebKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/04 00:13:35 No eventing-con process running
2021/12/04 00:13:35 Function: TestEventingRebKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpOneByOne not found"
 }
}
2021/12/04 00:13:37 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:13:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 00:13:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 00:13:44 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:13:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 00:13:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:13:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 00:13:54 Deploying app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne. Item count src bucket: 0 dst bucket: 0
2021/12/04 00:13:55 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebKVOpsOnUpdateBucketOpOneByOne' in metakv",
  "warnings": null
 }
}
2021/12/04 00:14:00 Starting bucket ops to source bucket
2021/12/04 00:14:05 Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/04 00:14:10 Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/04 00:14:15 Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/04 00:14:15 App: TestEventingRebKVOpsOnUpdateBucketOpOneByOne got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:14:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:14:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 00:14:15 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/04 00:14:15 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/04 00:14:15 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/04 00:14:20 Rebalance progress: 16.66666666666667
2021/12/04 00:14:25 Rebalance progress: 16.66666666666667
2021/12/04 00:14:30 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:14:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:14:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 00:14:30 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/04 00:14:30 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/04 00:14:31 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/04 00:14:36 Rebalance progress: 16.66666666666667
2021/12/04 00:14:41 Rebalance progress: 16.66666666666667
2021/12/04 00:14:46 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:14:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:14:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 00:14:46 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/04 00:14:46 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/04 00:14:51 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/04 00:14:57 Rebalance progress: 12.5
2021/12/04 00:15:02 Rebalance progress: 12.5
2021/12/04 00:15:07 Rebalance progress: 12.5
2021/12/04 00:15:12 Rebalance progress: 70.83333333333334
2021/12/04 00:15:17 Rebalance progress: 70.83333333333334
2021/12/04 00:15:22 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:15:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:15:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 00:15:22 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/04 00:15:27 Rebalance progress: 12.5
2021/12/04 00:15:32 Rebalance progress: 12.5
2021/12/04 00:15:37 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:15:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:15:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 00:15:37 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/04 00:15:42 Rebalance progress: 12.5
2021/12/04 00:15:47 Rebalance progress: 12.5
2021/12/04 00:15:52 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:15:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:15:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 00:15:52 clusterSetup.go:253: Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/04 00:15:57 Rebalance progress: 12.5
2021/12/04 00:16:02 Rebalance progress: 12.5
2021/12/04 00:16:07 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:16:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:16:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 00:16:07 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 00:16:12 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/04 00:16:17 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/04 00:16:22 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/04 00:16:27 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/04 00:16:27 clusterSetup.go:405: App: TestEventingRebKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/04 00:16:27 eventingSetup.go:46: No eventing-con process running
2021/12/04 00:16:27 eventingSetup.go:418: Function: TestEventingRebKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpOneByOne deleting in the background"
}
2021/12/04 00:16:29 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:16:34 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 00:16:39 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12703
2021/12/04 00:16:39 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 00:16:41 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:16:46 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 00:16:51 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 8637
2021/12/04 00:16:56 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 8637
2021/12/04 00:16:56 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebKVOpsOnUpdateBucketOpOneByOne (210.58s)
=== RUN   TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce
2021/12/04 00:17:06 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get un-deployed
2021/12/04 00:17:06 clusterSetup.go:405: App: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce got un-deployed
2021/12/04 00:17:06 eventingSetup.go:46: No eventing-con process running
2021/12/04 00:17:06 eventingSetup.go:418: Function: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce not found"
 }
}
2021/12/04 00:17:07 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:17:12 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 00:17:12 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 00:17:14 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:17:19 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 00:17:19 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:17:24 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 00:17:24 eventingSetup.go:125: Deploying app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce. Item count src bucket: 0 dst bucket: 0
2021/12/04 00:17:25 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce' in metakv",
  "warnings": null
 }
}
2021/12/04 00:17:30 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/04 00:17:35 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get deployed
2021/12/04 00:17:40 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get deployed
2021/12/04 00:17:45 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get deployed
2021/12/04 00:17:45 clusterSetup.go:354: App: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:17:45 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:17:45 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 00:17:45 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/04 00:17:46 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/04 00:17:46 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/04 00:17:46 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/04 00:17:51 Rebalance progress: 12.5
2021/12/04 00:17:56 Rebalance progress: 12.5
2021/12/04 00:18:01 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:18:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:18:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 00:18:01 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002 127.0.0.1:19003] from the cluster
2021/12/04 00:18:06 Rebalance progress: 12.5
2021/12/04 00:18:11 Rebalance progress: 12.5
2021/12/04 00:18:16 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:18:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:18:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 00:18:16 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 00:18:21 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get un-deployed
2021/12/04 00:18:27 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get un-deployed
2021/12/04 00:18:32 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get un-deployed
2021/12/04 00:18:37 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get un-deployed
2021/12/04 00:18:37 clusterSetup.go:405: App: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce got un-deployed
2021/12/04 00:18:37 eventingSetup.go:46: No eventing-con process running
2021/12/04 00:18:37 eventingSetup.go:418: Function: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce deleting in the background"
}
2021/12/04 00:18:38 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:18:43 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 00:18:48 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4580
2021/12/04 00:18:48 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 00:18:50 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:18:55 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 00:19:00 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1252
2021/12/04 00:19:05 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1252
2021/12/04 00:19:05 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce (129.54s)
=== RUN   TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings
2021/12/04 00:19:15 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/04 00:19:15 clusterSetup.go:405: App: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings got un-deployed
2021/12/04 00:19:15 eventingSetup.go:46: No eventing-con process running
2021/12/04 00:19:15 eventingSetup.go:418: Function: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings not found"
 }
}
2021/12/04 00:19:17 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:19:22 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 00:19:22 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 00:19:24 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:19:29 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 00:19:29 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:19:34 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 00:19:34 eventingSetup.go:125: Deploying app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/04 00:19:35 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/04 00:19:40 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/04 00:19:45 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get deployed
2021/12/04 00:19:50 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get deployed
2021/12/04 00:19:55 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get deployed
2021/12/04 00:19:55 clusterSetup.go:354: App: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:19:55 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:19:55 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 00:19:55 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/04 00:19:55 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/04 00:20:00 Rebalance progress: 12.5
2021/12/04 00:20:05 Rebalance progress: 12.5
2021/12/04 00:20:10 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:20:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:20:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 00:20:11 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/04 00:20:11 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/04 00:20:16 Rebalance progress: 12.5
2021/12/04 00:20:21 Rebalance progress: 12.5
2021/12/04 00:20:26 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:20:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:20:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 00:20:26 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/04 00:20:26 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/04 00:20:31 Rebalance progress: 12.5
2021/12/04 00:20:36 Rebalance progress: 12.5
2021/12/04 00:20:41 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:20:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:20:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 00:20:41 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/04 00:20:46 Rebalance progress: 12.5
2021/12/04 00:20:51 Rebalance progress: 12.5
2021/12/04 00:20:56 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:20:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:20:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 00:20:56 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/04 00:21:02 Rebalance progress: 12.5
2021/12/04 00:21:07 Rebalance progress: 12.5
2021/12/04 00:21:12 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:21:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:21:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 00:21:12 clusterSetup.go:253: Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/04 00:21:17 Rebalance progress: 12.5
2021/12/04 00:21:22 Rebalance progress: 12.5
2021/12/04 00:21:27 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:21:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:21:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 00:21:27 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 00:21:32 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/04 00:21:37 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/04 00:21:42 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/04 00:21:47 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/04 00:21:47 clusterSetup.go:405: App: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings got un-deployed
2021/12/04 00:21:47 eventingSetup.go:46: No eventing-con process running
2021/12/04 00:21:47 eventingSetup.go:418: Function: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings deleting in the background"
}
2021/12/04 00:21:49 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:21:54 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 00:21:59 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 10665
2021/12/04 00:21:59 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 00:22:00 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:22:05 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 00:22:10 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5338
2021/12/04 00:22:15 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5338
2021/12/04 00:22:15 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings (190.27s)
=== RUN   TestEventingSwapRebOnUpdateBucketOp
2021/12/04 00:22:25 clusterSetup.go:397: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get un-deployed
2021/12/04 00:22:25 clusterSetup.go:405: App: TestEventingSwapRebOnUpdateBucketOp got un-deployed
2021/12/04 00:22:25 eventingSetup.go:46: No eventing-con process running
2021/12/04 00:22:25 eventingSetup.go:418: Function: TestEventingSwapRebOnUpdateBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingSwapRebOnUpdateBucketOp not found"
 }
}
2021/12/04 00:22:27 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:22:32 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 00:22:32 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 00:22:34 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:22:39 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 00:22:39 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:22:39 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 00:22:39 eventingSetup.go:125: Deploying app: TestEventingSwapRebOnUpdateBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/04 00:22:40 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingSwapRebOnUpdateBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/04 00:22:45 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/04 00:22:50 clusterSetup.go:346: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get deployed
2021/12/04 00:22:55 clusterSetup.go:346: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get deployed
2021/12/04 00:23:00 clusterSetup.go:346: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get deployed
2021/12/04 00:23:00 clusterSetup.go:354: App: TestEventingSwapRebOnUpdateBucketOp got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:23:00 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:23:00 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 00:23:00 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/04 00:23:01 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/04 00:23:06 Rebalance progress: 12.5
2021/12/04 00:23:11 Rebalance progress: 12.5
2021/12/04 00:23:16 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:23:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:23:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 00:23:16 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/04 00:23:16 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/04 00:23:21 Rebalance progress: 12.5
2021/12/04 00:23:26 Rebalance progress: 12.5
2021/12/04 00:23:31 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:23:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:23:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 00:23:31 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/04 00:23:36 Rebalance progress: 12.5
2021/12/04 00:23:41 Rebalance progress: 12.5
2021/12/04 00:23:46 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:23:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:23:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 00:23:46 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 00:23:51 clusterSetup.go:397: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get un-deployed
2021/12/04 00:23:57 clusterSetup.go:397: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get un-deployed
2021/12/04 00:24:02 clusterSetup.go:397: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get un-deployed
2021/12/04 00:24:07 clusterSetup.go:397: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get un-deployed
2021/12/04 00:24:07 clusterSetup.go:405: App: TestEventingSwapRebOnUpdateBucketOp got un-deployed
2021/12/04 00:24:07 eventingSetup.go:46: No eventing-con process running
2021/12/04 00:24:07 eventingSetup.go:418: Function: TestEventingSwapRebOnUpdateBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingSwapRebOnUpdateBucketOp deleting in the background"
}
2021/12/04 00:24:08 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:24:13 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 00:24:18 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6112
2021/12/04 00:24:18 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 00:24:20 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:24:25 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 00:24:30 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1443
2021/12/04 00:24:35 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1443
2021/12/04 00:24:35 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingSwapRebOnUpdateBucketOp (139.70s)
=== RUN   TestMetaRollbackWithEventingReb
2021/12/04 00:24:40 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/04 00:24:40 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/04 00:24:45 Rebalance progress: 100
2021/12/04 00:24:50 Waiting for app: TestMetaRollbackWithEventingReb to get un-deployed
2021/12/04 00:24:50 App: TestMetaRollbackWithEventingReb got un-deployed
2021/12/04 00:24:50 No eventing-con process running
2021/12/04 00:24:50 Function: TestMetaRollbackWithEventingReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMetaRollbackWithEventingReb not found"
 }
}
2021/12/04 00:24:52 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:24:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 00:24:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 00:24:59 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:25:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 00:25:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:25:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 00:25:04 Deploying app: TestMetaRollbackWithEventingReb. Item count src bucket: 0 dst bucket: 0
2021/12/04 00:25:05 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMetaRollbackWithEventingReb' in metakv",
  "warnings": null
 }
}
2021/12/04 00:25:10 Starting bucket ops to source bucket
2021/12/04 00:25:15 Waiting for app: TestMetaRollbackWithEventingReb to get deployed
2021/12/04 00:25:20 Waiting for app: TestMetaRollbackWithEventingReb to get deployed
2021/12/04 00:25:25 Waiting for app: TestMetaRollbackWithEventingReb to get deployed
2021/12/04 00:25:25 App: TestMetaRollbackWithEventingReb got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:25:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:25:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 00:25:25 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/04 00:25:26 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/04 00:25:31 Rebalance progress: 12.5
2021/12/04 00:25:36 Rebalance progress: 12.5
2021/12/04 00:25:41 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:25:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:25:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 00:25:41 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002] from the cluster
2021/12/04 00:25:41 bucketOps.go:134: Metakv lookup failed, err: Get "/_metakv/eventing/tempApps/TestMetaRollbackWithEventingReb/0": Unable to initialize cbauth's revrpc: cbauth environment variable CBAUTH_REVRPC_URL is not set
2021/12/04 00:25:46 Rebalance progress: 12.5
2021/12/04 00:25:51 Rebalance progress: 12.5
2021/12/04 00:25:56 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:25:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:25:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 00:25:56 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 00:26:01 clusterSetup.go:397: Waiting for app: TestMetaRollbackWithEventingReb to get un-deployed
2021/12/04 00:26:06 clusterSetup.go:397: Waiting for app: TestMetaRollbackWithEventingReb to get un-deployed
2021/12/04 00:26:11 clusterSetup.go:397: Waiting for app: TestMetaRollbackWithEventingReb to get un-deployed
2021/12/04 00:26:16 clusterSetup.go:397: Waiting for app: TestMetaRollbackWithEventingReb to get un-deployed
2021/12/04 00:26:16 clusterSetup.go:405: App: TestMetaRollbackWithEventingReb got un-deployed
2021/12/04 00:26:16 eventingSetup.go:46: No eventing-con process running
2021/12/04 00:26:16 eventingSetup.go:418: Function: TestMetaRollbackWithEventingReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMetaRollbackWithEventingReb deleting in the background"
}
2021/12/04 00:26:18 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:26:23 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 00:26:28 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4587
2021/12/04 00:26:28 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 00:26:30 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:26:35 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 00:26:35 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestMetaRollbackWithEventingReb (119.74s)
=== RUN   TestMetaPartialRollbackWithEventingReb
2021/12/04 00:26:40 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/04 00:26:40 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/04 00:26:45 Rebalance progress: 100
2021/12/04 00:26:50 Waiting for app: TestMetaPartialRollbackWithEventingReb to get un-deployed
2021/12/04 00:26:50 App: TestMetaPartialRollbackWithEventingReb got un-deployed
2021/12/04 00:26:50 No eventing-con process running
2021/12/04 00:26:50 Function: TestMetaPartialRollbackWithEventingReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMetaPartialRollbackWithEventingReb not found"
 }
}
2021/12/04 00:26:52 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:26:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 00:26:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 00:26:59 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:27:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 00:27:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:27:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 00:27:04 Deploying app: TestMetaPartialRollbackWithEventingReb. Item count src bucket: 0 dst bucket: 0
2021/12/04 00:27:05 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMetaPartialRollbackWithEventingReb' in metakv",
  "warnings": null
 }
}
2021/12/04 00:27:10 Starting bucket ops to source bucket
2021/12/04 00:27:15 Waiting for app: TestMetaPartialRollbackWithEventingReb to get deployed
2021/12/04 00:27:20 Waiting for app: TestMetaPartialRollbackWithEventingReb to get deployed
2021/12/04 00:27:25 Waiting for app: TestMetaPartialRollbackWithEventingReb to get deployed
2021/12/04 00:27:25 App: TestMetaPartialRollbackWithEventingReb got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:27:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:27:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 00:27:25 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/04 00:27:26 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/04 00:27:31 Rebalance progress: 12.5
2021/12/04 00:27:36 Rebalance progress: 12.5
2021/12/04 00:27:41 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:27:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:27:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 00:27:41 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002] from the cluster
2021/12/04 00:27:41 bucketOps.go:65: Metakv lookup failed, err: Get "/_metakv/eventing/tempApps/TestMetaPartialRollbackWithEventingReb/0": Unable to initialize cbauth's revrpc: cbauth environment variable CBAUTH_REVRPC_URL is not set
2021/12/04 00:27:46 Rebalance progress: 12.5
2021/12/04 00:27:51 Rebalance progress: 12.5
2021/12/04 00:27:56 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:27:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:27:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 00:27:56 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 00:27:56 bucketOps.go:65: Metakv lookup failed, err: Get "/_metakv/eventing/tempApps/TestMetaPartialRollbackWithEventingReb/0": Unable to initialize cbauth's revrpc: cbauth environment variable CBAUTH_REVRPC_URL is not set
2021/12/04 00:28:01 clusterSetup.go:397: Waiting for app: TestMetaPartialRollbackWithEventingReb to get un-deployed
2021/12/04 00:28:06 clusterSetup.go:397: Waiting for app: TestMetaPartialRollbackWithEventingReb to get un-deployed
2021/12/04 00:28:11 clusterSetup.go:397: Waiting for app: TestMetaPartialRollbackWithEventingReb to get un-deployed
2021/12/04 00:28:16 clusterSetup.go:397: Waiting for app: TestMetaPartialRollbackWithEventingReb to get un-deployed
2021/12/04 00:28:16 clusterSetup.go:405: App: TestMetaPartialRollbackWithEventingReb got un-deployed
2021/12/04 00:28:16 eventingSetup.go:46: No eventing-con process running
2021/12/04 00:28:16 eventingSetup.go:418: Function: TestMetaPartialRollbackWithEventingReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMetaPartialRollbackWithEventingReb deleting in the background"
}
2021/12/04 00:28:18 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:28:23 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 00:28:28 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 37283
2021/12/04 00:28:28 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 00:28:30 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:28:35 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 00:28:35 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestMetaPartialRollbackWithEventingReb (119.68s)
=== RUN   TestEventingRebMultipleHandlersOneByOne
2021/12/04 00:28:45 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get un-deployed
2021/12/04 00:28:45 clusterSetup.go:405: App: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op got un-deployed
2021/12/04 00:28:45 eventingSetup.go:46: No eventing-con process running
2021/12/04 00:28:45 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op not found"
 }
}
2021/12/04 00:28:46 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:28:51 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 00:28:51 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 00:28:53 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:28:58 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 00:28:58 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:29:03 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 00:29:03 eventingSetup.go:125: Deploying app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op. Item count src bucket: 0 dst bucket: 0
2021/12/04 00:29:04 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op' in metakv",
  "warnings": null
 }
}
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:29:04 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 00:29:04 eventingSetup.go:125: Deploying app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/04 00:29:05 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebMultipleHandlersOneByOnebucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/04 00:29:10 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/04 00:29:15 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get deployed
2021/12/04 00:29:20 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get deployed
2021/12/04 00:29:26 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get deployed
2021/12/04 00:29:26 clusterSetup.go:354: App: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op got deployed
2021/12/04 00:29:31 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get deployed
2021/12/04 00:29:36 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get deployed
2021/12/04 00:29:41 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get deployed
2021/12/04 00:29:41 clusterSetup.go:354: App: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:29:41 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:29:41 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 00:29:41 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/04 00:29:41 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/04 00:29:46 Rebalance progress: 12.5
2021/12/04 00:29:51 Rebalance progress: 12.5
2021/12/04 00:29:56 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:29:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:29:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 00:29:56 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/04 00:29:56 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/04 00:30:01 Rebalance progress: 12.5
2021/12/04 00:30:06 Rebalance progress: 12.5
2021/12/04 00:30:11 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:30:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:30:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 00:30:11 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/04 00:30:12 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/04 00:30:17 Rebalance progress: 12.5
2021/12/04 00:30:22 Rebalance progress: 12.5
2021/12/04 00:30:27 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:30:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:30:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 00:30:27 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/04 00:30:32 Rebalance progress: 12.5
2021/12/04 00:30:37 Rebalance progress: 12.5
2021/12/04 00:30:42 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:30:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:30:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 00:30:42 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/04 00:30:47 Rebalance progress: 12.5
2021/12/04 00:30:52 Rebalance progress: 12.5
2021/12/04 00:30:57 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:30:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:30:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 00:30:57 clusterSetup.go:253: Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/04 00:31:02 Rebalance progress: 12.5
2021/12/04 00:31:07 Rebalance progress: 12.5
2021/12/04 00:31:12 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:31:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:31:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 00:31:12 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 00:31:17 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get un-deployed
2021/12/04 00:31:22 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get un-deployed
2021/12/04 00:31:27 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get un-deployed
2021/12/04 00:31:32 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get un-deployed
2021/12/04 00:31:32 clusterSetup.go:405: App: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op got un-deployed
2021/12/04 00:31:32 eventingSetup.go:46: No eventing-con process running
2021/12/04 00:31:32 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op deleting in the background"
}
2021/12/04 00:31:36 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:31:41 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 00:31:46 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12154
2021/12/04 00:31:46 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 00:31:48 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:31:53 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 00:31:58 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12154
2021/12/04 00:31:58 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 00:31:58 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 00:32:03 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get un-deployed
2021/12/04 00:32:08 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get un-deployed
2021/12/04 00:32:13 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get un-deployed
2021/12/04 00:32:18 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get un-deployed
2021/12/04 00:32:18 clusterSetup.go:405: App: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update got un-deployed
2021/12/04 00:32:18 eventingSetup.go:46: No eventing-con process running
2021/12/04 00:32:18 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update deleting in the background"
}
2021/12/04 00:32:20 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:32:25 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 00:32:25 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 00:32:27 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:32:32 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 00:32:32 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebMultipleHandlersOneByOne (237.10s)
=== RUN   TestEventingRebMultipleHandlersAllAtOnce
2021/12/04 00:32:42 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get un-deployed
2021/12/04 00:32:42 clusterSetup.go:405: App: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op got un-deployed
2021/12/04 00:32:42 eventingSetup.go:46: No eventing-con process running
2021/12/04 00:32:42 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op not found"
 }
}
2021/12/04 00:32:44 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:32:49 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 00:32:49 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 00:32:50 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:32:55 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 00:32:55 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:33:00 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 00:33:00 eventingSetup.go:125: Deploying app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op. Item count src bucket: 0 dst bucket: 0
2021/12/04 00:33:01 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op' in metakv",
  "warnings": null
 }
}
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:33:01 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 00:33:01 eventingSetup.go:125: Deploying app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/04 00:33:03 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/04 00:33:08 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/04 00:33:13 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get deployed
2021/12/04 00:33:18 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get deployed
2021/12/04 00:33:23 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get deployed
2021/12/04 00:33:23 clusterSetup.go:354: App: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op got deployed
2021/12/04 00:33:28 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get deployed
2021/12/04 00:33:33 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get deployed
2021/12/04 00:33:38 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get deployed
2021/12/04 00:33:38 clusterSetup.go:354: App: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:33:38 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:33:38 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 00:33:38 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/04 00:33:38 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/04 00:33:38 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/04 00:33:39 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/04 00:33:44 Rebalance progress: 12.5
2021/12/04 00:33:49 Rebalance progress: 12.5
2021/12/04 00:33:54 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:33:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:33:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 00:33:54 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002 127.0.0.1:19003] from the cluster
2021/12/04 00:33:59 Rebalance progress: 12.5
2021/12/04 00:34:04 Rebalance progress: 12.5
2021/12/04 00:34:09 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:34:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:34:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 00:34:09 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 00:34:14 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get un-deployed
2021/12/04 00:34:19 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get un-deployed
2021/12/04 00:34:24 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get un-deployed
2021/12/04 00:34:29 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get un-deployed
2021/12/04 00:34:29 clusterSetup.go:405: App: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op got un-deployed
2021/12/04 00:34:29 eventingSetup.go:46: No eventing-con process running
2021/12/04 00:34:29 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op deleting in the background"
}
2021/12/04 00:34:33 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:34:38 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 00:34:43 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6104
2021/12/04 00:34:48 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6104
2021/12/04 00:34:48 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 00:34:50 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:34:55 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 00:35:00 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 118
2021/12/04 00:35:05 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 118
2021/12/04 00:35:05 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 00:35:05 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 00:35:10 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get un-deployed
2021/12/04 00:35:15 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get un-deployed
2021/12/04 00:35:20 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get un-deployed
2021/12/04 00:35:25 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get un-deployed
2021/12/04 00:35:25 clusterSetup.go:405: App: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update got un-deployed
2021/12/04 00:35:25 eventingSetup.go:46: No eventing-con process running
2021/12/04 00:35:25 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update deleting in the background"
}
2021/12/04 00:35:27 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:35:32 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 00:35:32 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 00:35:34 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:35:39 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 00:35:39 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebMultipleHandlersAllAtOnce (187.06s)
=== RUN   TestEventingSwapRebMultipleHandlers
2021/12/04 00:35:49 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get un-deployed
2021/12/04 00:35:49 clusterSetup.go:405: App: TestEventingSwapRebMultipleHandlersbucket_op_on_update got un-deployed
2021/12/04 00:35:49 eventingSetup.go:46: No eventing-con process running
2021/12/04 00:35:49 eventingSetup.go:418: Function: TestEventingSwapRebMultipleHandlersbucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingSwapRebMultipleHandlersbucket_op_on_update not found"
 }
}
2021/12/04 00:35:51 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:35:56 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 00:35:56 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 00:35:57 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:36:02 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 00:36:02 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 00:36:07 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get un-deployed
2021/12/04 00:36:07 clusterSetup.go:405: App: TestEventingSwapRebMultipleHandlerssys_test_bucket_op got un-deployed
2021/12/04 00:36:07 eventingSetup.go:46: No eventing-con process running
2021/12/04 00:36:07 eventingSetup.go:418: Function: TestEventingSwapRebMultipleHandlerssys_test_bucket_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingSwapRebMultipleHandlerssys_test_bucket_op not found"
 }
}
2021/12/04 00:36:09 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:36:14 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 00:36:14 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 00:36:16 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:36:21 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 00:36:21 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:36:21 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 00:36:21 eventingSetup.go:125: Deploying app: TestEventingSwapRebMultipleHandlersbucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/04 00:36:22 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingSwapRebMultipleHandlersbucket_op_on_update' in metakv",
  "warnings": null
 }
}
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:36:22 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 00:36:22 eventingSetup.go:125: Deploying app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op. Item count src bucket: 0 dst bucket: 0
2021/12/04 00:36:23 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingSwapRebMultipleHandlerssys_test_bucket_op' in metakv",
  "warnings": null
 }
}
2021/12/04 00:36:28 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/04 00:36:33 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get deployed
2021/12/04 00:36:38 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get deployed
2021/12/04 00:36:43 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get deployed
2021/12/04 00:36:43 clusterSetup.go:354: App: TestEventingSwapRebMultipleHandlersbucket_op_on_update got deployed
2021/12/04 00:36:48 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get deployed
2021/12/04 00:36:53 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get deployed
2021/12/04 00:36:58 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get deployed
2021/12/04 00:36:58 clusterSetup.go:354: App: TestEventingSwapRebMultipleHandlerssys_test_bucket_op got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:36:58 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:36:58 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 00:36:58 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/04 00:36:59 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/04 00:37:04 Rebalance progress: 12.5
2021/12/04 00:37:09 Rebalance progress: 12.5
2021/12/04 00:37:14 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:37:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:37:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 00:37:14 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/04 00:37:14 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/04 00:37:19 Rebalance progress: 12.5
2021/12/04 00:37:24 Rebalance progress: 12.5
2021/12/04 00:37:29 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:37:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:37:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 00:37:29 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/04 00:37:35 Rebalance progress: 12.5
2021/12/04 00:37:40 Rebalance progress: 12.5
2021/12/04 00:37:45 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:37:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:37:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 00:37:45 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 00:37:50 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get un-deployed
2021/12/04 00:37:55 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get un-deployed
2021/12/04 00:38:00 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get un-deployed
2021/12/04 00:38:05 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get un-deployed
2021/12/04 00:38:05 clusterSetup.go:405: App: TestEventingSwapRebMultipleHandlersbucket_op_on_update got un-deployed
2021/12/04 00:38:05 eventingSetup.go:46: No eventing-con process running
2021/12/04 00:38:05 eventingSetup.go:418: Function: TestEventingSwapRebMultipleHandlersbucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingSwapRebMultipleHandlersbucket_op_on_update deleting in the background"
}
2021/12/04 00:38:08 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:38:13 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 00:38:18 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7624
2021/12/04 00:38:18 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 00:38:20 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:38:25 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 00:38:30 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1975
2021/12/04 00:38:35 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1975
2021/12/04 00:38:35 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 00:38:35 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 00:38:40 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get un-deployed
2021/12/04 00:38:45 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get un-deployed
2021/12/04 00:38:50 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get un-deployed
2021/12/04 00:38:55 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get un-deployed
2021/12/04 00:38:55 clusterSetup.go:405: App: TestEventingSwapRebMultipleHandlerssys_test_bucket_op got un-deployed
2021/12/04 00:38:55 eventingSetup.go:46: No eventing-con process running
2021/12/04 00:38:55 eventingSetup.go:418: Function: TestEventingSwapRebMultipleHandlerssys_test_bucket_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingSwapRebMultipleHandlerssys_test_bucket_op deleting in the background"
}
2021/12/04 00:38:57 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:39:02 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 00:39:02 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 00:39:04 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:39:09 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 00:39:09 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingSwapRebMultipleHandlers (210.33s)
=== RUN   TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne
2021/12/04 00:39:19 clusterSetup.go:397: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/04 00:39:19 clusterSetup.go:405: App: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/04 00:39:19 eventingSetup.go:46: No eventing-con process running
2021/12/04 00:39:19 eventingSetup.go:418: Function: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne not found"
 }
}
2021/12/04 00:39:21 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:39:26 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 00:39:26 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 00:39:28 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:39:33 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 00:39:33 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:39:38 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 00:39:38 eventingSetup.go:125: Deploying app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne. Item count src bucket: 0 dst bucket: 0
2021/12/04 00:39:39 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne' in metakv",
  "warnings": null
 }
}
2021/12/04 00:39:44 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/04 00:39:49 clusterSetup.go:346: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/04 00:39:54 clusterSetup.go:346: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/04 00:39:59 clusterSetup.go:346: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/04 00:39:59 clusterSetup.go:354: App: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:39:59 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:39:59 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 00:39:59 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/04 00:39:59 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/04 00:40:19 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:40:20 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:40:20 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:40:50 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:40:50 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 00:40:50 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/04 00:40:55 Rebalance progress: 12.5
2021/12/04 00:41:00 Rebalance progress: 12.5
2021/12/04 00:41:05 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:41:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:41:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 00:41:05 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/04 00:41:05 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/04 00:41:10 Rebalance progress: 12.5
2021/12/04 00:41:15 Rebalance progress: 12.5
2021/12/04 00:41:20 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:41:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:41:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 00:41:20 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/04 00:41:25 Rebalance progress: 12.5
2021/12/04 00:41:30 Rebalance progress: 12.5
2021/12/04 00:41:35 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:41:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:41:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 00:41:36 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 00:41:41 clusterSetup.go:397: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/04 00:41:46 clusterSetup.go:397: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/04 00:41:51 clusterSetup.go:397: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/04 00:41:56 clusterSetup.go:397: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/04 00:41:56 clusterSetup.go:405: App: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/04 00:41:56 eventingSetup.go:46: No eventing-con process running
2021/12/04 00:41:56 eventingSetup.go:418: Function: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne deleting in the background"
}
2021/12/04 00:41:57 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:42:02 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 00:42:07 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 11164
2021/12/04 00:42:08 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 00:42:09 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:42:14 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 00:42:19 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 11164
2021/12/04 00:42:19 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne (190.27s)
=== RUN   TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne
2021/12/04 00:42:29 clusterSetup.go:397: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/04 00:42:29 clusterSetup.go:405: App: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/04 00:42:29 eventingSetup.go:46: No eventing-con process running
2021/12/04 00:42:29 eventingSetup.go:418: Function: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne not found"
 }
}
2021/12/04 00:42:31 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:42:36 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 00:42:36 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 00:42:38 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:42:43 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 00:42:43 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:42:48 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 00:42:48 eventingSetup.go:125: Deploying app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne. Item count src bucket: 0 dst bucket: 0
2021/12/04 00:42:49 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne' in metakv",
  "warnings": null
 }
}
2021/12/04 00:42:54 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/04 00:42:59 clusterSetup.go:346: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/04 00:43:04 clusterSetup.go:346: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/04 00:43:09 clusterSetup.go:346: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/04 00:43:09 clusterSetup.go:354: App: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:43:09 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:43:09 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 00:43:09 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/04 00:43:09 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/04 00:43:29 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:43:30 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:43:30 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:43:40 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:43:40 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 00:43:40 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/04 00:43:40 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/04 00:44:00 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:44:01 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:44:01 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:44:11 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:44:11 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 00:44:11 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/04 00:44:11 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/04 00:44:31 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:44:32 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:44:32 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:44:42 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:44:42 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 00:44:42 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/04 00:44:42 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/04 00:45:02 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:45:02 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:45:02 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:45:13 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:45:13 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 00:45:13 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/04 00:45:13 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/04 00:45:33 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:45:33 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:45:33 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:45:43 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:45:43 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 00:45:43 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/04 00:45:49 Rebalance progress: 12.5
2021/12/04 00:45:54 Rebalance progress: 12.5
2021/12/04 00:45:59 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:45:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:45:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 00:45:59 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/04 00:45:59 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/04 00:46:04 Rebalance progress: 12.5
2021/12/04 00:46:09 Rebalance progress: 12.5
2021/12/04 00:46:14 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:46:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:46:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 00:46:14 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/04 00:46:19 Rebalance progress: 12.5
2021/12/04 00:46:24 Rebalance progress: 12.5
2021/12/04 00:46:29 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:46:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:46:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 00:46:29 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 00:46:34 clusterSetup.go:397: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/04 00:46:39 clusterSetup.go:397: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/04 00:46:44 clusterSetup.go:397: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/04 00:46:49 clusterSetup.go:397: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/04 00:46:49 clusterSetup.go:405: App: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/04 00:46:49 eventingSetup.go:46: No eventing-con process running
2021/12/04 00:46:49 eventingSetup.go:418: Function: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne deleting in the background"
}
2021/12/04 00:46:51 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:46:56 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 00:47:01 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 20742
2021/12/04 00:47:06 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 20742
2021/12/04 00:47:06 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 00:47:08 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:47:13 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 00:47:18 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 21487
2021/12/04 00:47:18 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne (298.39s)
=== RUN   TestEventingFailoverOnUpdateBucketOp
2021/12/04 00:47:28 clusterSetup.go:397: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get un-deployed
2021/12/04 00:47:28 clusterSetup.go:405: App: TestEventingFailoverOnUpdateBucketOp got un-deployed
2021/12/04 00:47:28 eventingSetup.go:46: No eventing-con process running
2021/12/04 00:47:28 eventingSetup.go:418: Function: TestEventingFailoverOnUpdateBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingFailoverOnUpdateBucketOp not found"
 }
}
2021/12/04 00:47:30 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:47:35 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 00:47:35 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 00:47:36 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:47:41 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 00:47:41 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:47:46 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 00:47:46 eventingSetup.go:125: Deploying app: TestEventingFailoverOnUpdateBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/04 00:47:47 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingFailoverOnUpdateBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/04 00:47:52 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/04 00:47:57 clusterSetup.go:346: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get deployed
2021/12/04 00:48:02 clusterSetup.go:346: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get deployed
2021/12/04 00:48:07 clusterSetup.go:346: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get deployed
2021/12/04 00:48:07 clusterSetup.go:354: App: TestEventingFailoverOnUpdateBucketOp got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:48:07 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:48:07 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 00:48:07 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/04 00:48:08 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/04 00:48:13 Rebalance progress: 12.5
2021/12/04 00:48:18 Rebalance progress: 12.5
2021/12/04 00:48:23 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:48:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:48:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 00:48:23 clusterSetup.go:229: Failing over: [127.0.0.1:19001]
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:48:33 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:48:33 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 00:48:33 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/04 00:48:38 Rebalance progress: 12.5
2021/12/04 00:48:43 Rebalance progress: 12.5
2021/12/04 00:48:48 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:48:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:48:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 00:48:48 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/04 00:48:48 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/04 00:48:53 Rebalance progress: 12.5
2021/12/04 00:48:58 Rebalance progress: 12.5
2021/12/04 00:49:03 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:49:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:49:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 00:49:03 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/04 00:49:08 Rebalance progress: 12.5
2021/12/04 00:49:13 Rebalance progress: 12.5
2021/12/04 00:49:18 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:49:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:49:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 00:49:18 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 00:49:23 clusterSetup.go:397: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get un-deployed
2021/12/04 00:49:28 clusterSetup.go:397: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get un-deployed
2021/12/04 00:49:33 clusterSetup.go:397: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get un-deployed
2021/12/04 00:49:38 clusterSetup.go:397: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get un-deployed
2021/12/04 00:49:38 clusterSetup.go:405: App: TestEventingFailoverOnUpdateBucketOp got un-deployed
2021/12/04 00:49:38 eventingSetup.go:46: No eventing-con process running
2021/12/04 00:49:38 eventingSetup.go:418: Function: TestEventingFailoverOnUpdateBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingFailoverOnUpdateBucketOp deleting in the background"
}
2021/12/04 00:49:40 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:49:45 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 00:49:50 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2356
2021/12/04 00:49:55 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2356
2021/12/04 00:49:55 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 00:49:57 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:50:02 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 00:50:07 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 8561
2021/12/04 00:50:07 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingFailoverOnUpdateBucketOp (169.25s)
=== RUN   TestEventingKVRebalanceOnUpdateBucketOp
2021/12/04 00:50:17 clusterSetup.go:397: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get un-deployed
2021/12/04 00:50:17 clusterSetup.go:405: App: TestEventingKVRebalanceOnUpdateBucketOp got un-deployed
2021/12/04 00:50:17 eventingSetup.go:46: No eventing-con process running
2021/12/04 00:50:17 eventingSetup.go:418: Function: TestEventingKVRebalanceOnUpdateBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingKVRebalanceOnUpdateBucketOp not found"
 }
}
2021/12/04 00:50:19 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:50:24 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 00:50:24 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 00:50:26 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:50:31 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 00:50:31 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:50:36 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 00:50:36 eventingSetup.go:125: Deploying app: TestEventingKVRebalanceOnUpdateBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/04 00:50:37 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingKVRebalanceOnUpdateBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/04 00:50:42 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/04 00:50:47 clusterSetup.go:346: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get deployed
2021/12/04 00:50:52 clusterSetup.go:346: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get deployed
2021/12/04 00:50:57 clusterSetup.go:346: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get deployed
2021/12/04 00:50:57 clusterSetup.go:354: App: TestEventingKVRebalanceOnUpdateBucketOp got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:50:57 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:50:57 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 00:50:57 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing,kv to the cluster
2021/12/04 00:50:57 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/04 00:51:02 Rebalance progress: 12.5
2021/12/04 00:51:07 Rebalance progress: 12.5
2021/12/04 00:51:12 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:51:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:51:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 00:51:12 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing,kv to the cluster
2021/12/04 00:51:12 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/04 00:51:17 Rebalance progress: 12.5
2021/12/04 00:51:22 Rebalance progress: 12.5
2021/12/04 00:51:27 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:51:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:51:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 00:51:27 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/04 00:51:32 Rebalance progress: 12.5
2021/12/04 00:51:37 Rebalance progress: 12.5
2021/12/04 00:51:42 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:51:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:51:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 00:51:43 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 00:51:48 clusterSetup.go:397: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get un-deployed
2021/12/04 00:51:53 clusterSetup.go:397: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get un-deployed
2021/12/04 00:51:58 clusterSetup.go:397: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get un-deployed
2021/12/04 00:52:03 clusterSetup.go:397: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get un-deployed
2021/12/04 00:52:03 clusterSetup.go:405: App: TestEventingKVRebalanceOnUpdateBucketOp got un-deployed
2021/12/04 00:52:03 eventingSetup.go:46: No eventing-con process running
2021/12/04 00:52:03 eventingSetup.go:418: Function: TestEventingKVRebalanceOnUpdateBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingKVRebalanceOnUpdateBucketOp deleting in the background"
}
2021/12/04 00:52:04 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:52:09 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 00:52:14 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6063
2021/12/04 00:52:14 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 00:52:16 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:52:21 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 00:52:26 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6063
2021/12/04 00:52:26 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingKVRebalanceOnUpdateBucketOp (139.27s)
=== RUN   TestEventingRebInWithOps
2021/12/04 00:52:36 clusterSetup.go:397: Waiting for app: TestEventingRebInWithOps to get un-deployed
2021/12/04 00:52:36 clusterSetup.go:405: App: TestEventingRebInWithOps got un-deployed
2021/12/04 00:52:36 eventingSetup.go:46: No eventing-con process running
2021/12/04 00:52:36 eventingSetup.go:418: Function: TestEventingRebInWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebInWithOps not found"
 }
}
2021/12/04 00:52:38 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:52:43 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 00:52:43 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 00:52:45 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:52:50 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 00:52:50 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:52:55 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 00:52:55 eventingSetup.go:125: Deploying app: TestEventingRebInWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/04 00:52:56 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebInWithOps' in metakv",
  "warnings": null
 }
}
2021/12/04 00:53:01 clusterSetup.go:346: Waiting for app: TestEventingRebInWithOps to get deployed
2021/12/04 00:53:06 clusterSetup.go:346: Waiting for app: TestEventingRebInWithOps to get deployed
2021/12/04 00:53:11 clusterSetup.go:346: Waiting for app: TestEventingRebInWithOps to get deployed
2021/12/04 00:53:16 clusterSetup.go:346: Waiting for app: TestEventingRebInWithOps to get deployed
2021/12/04 00:53:16 clusterSetup.go:354: App: TestEventingRebInWithOps got deployed
2021/12/04 00:53:16 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/04 00:53:16 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/04 00:53:16 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/04 00:53:21 Rebalance progress: 12.5
2021/12/04 00:53:26 Rebalance progress: 12.5
2021/12/04 00:53:31 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:53:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:53:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 00:53:31 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/04 00:53:36 Rebalance progress: 12.5
2021/12/04 00:53:41 Rebalance progress: 12.5
2021/12/04 00:53:46 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:53:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 00:53:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 00:53:46 Waiting for src bucket item count to get to: 200000 curr count: 35043
2021/12/04 00:53:52 Waiting for src bucket item count to get to: 200000 curr count: 50671
2021/12/04 00:53:57 Waiting for src bucket item count to get to: 200000 curr count: 50671
2021/12/04 00:54:02 Waiting for src bucket item count to get to: 200000 curr count: 66791
2021/12/04 00:54:07 Waiting for src bucket item count to get to: 200000 curr count: 66791
2021/12/04 00:54:12 Waiting for src bucket item count to get to: 200000 curr count: 81750
2021/12/04 00:54:17 Waiting for src bucket item count to get to: 200000 curr count: 81750
2021/12/04 00:54:22 Waiting for src bucket item count to get to: 200000 curr count: 97322
2021/12/04 00:54:27 Waiting for src bucket item count to get to: 200000 curr count: 97322
2021/12/04 00:54:32 Waiting for src bucket item count to get to: 200000 curr count: 97322
2021/12/04 00:54:37 Waiting for src bucket item count to get to: 200000 curr count: 113303
2021/12/04 00:54:42 Waiting for src bucket item count to get to: 200000 curr count: 113303
2021/12/04 00:54:47 Waiting for src bucket item count to get to: 200000 curr count: 129377
2021/12/04 00:54:52 Waiting for src bucket item count to get to: 200000 curr count: 129377
2021/12/04 00:54:57 Waiting for src bucket item count to get to: 200000 curr count: 145437
2021/12/04 00:55:02 Waiting for src bucket item count to get to: 200000 curr count: 145437
2021/12/04 00:55:07 Waiting for src bucket item count to get to: 200000 curr count: 161238
2021/12/04 00:55:12 Waiting for src bucket item count to get to: 200000 curr count: 161238
2021/12/04 00:55:17 Waiting for src bucket item count to get to: 200000 curr count: 176293
2021/12/04 00:55:22 Waiting for src bucket item count to get to: 200000 curr count: 176293
2021/12/04 00:55:27 Waiting for src bucket item count to get to: 200000 curr count: 192222
2021/12/04 00:55:32 Waiting for src bucket item count to get to: 200000 curr count: 192222
2021/12/04 00:55:37 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/04 00:55:37 Post SET ops. Expected item count: 200000 got 200000
2021/12/04 00:55:37 Starting bucket ops to source bucket
2021/12/04 00:59:28 Waiting for dst bucket item count to get to: 0 curr count: 7047
2021/12/04 00:59:33 Waiting for dst bucket item count to get to: 0 curr count: 7047
2021/12/04 00:59:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 00:59:33 Post DELETE ops. Expected item count: 0 got 0
2021/12/04 00:59:33 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 00:59:38 Waiting for app: TestEventingRebInWithOps to get un-deployed
2021/12/04 00:59:43 Waiting for app: TestEventingRebInWithOps to get un-deployed
2021/12/04 00:59:48 Waiting for app: TestEventingRebInWithOps to get un-deployed
2021/12/04 00:59:53 Waiting for app: TestEventingRebInWithOps to get un-deployed
2021/12/04 00:59:53 App: TestEventingRebInWithOps got un-deployed
2021/12/04 00:59:53 No eventing-con process running
2021/12/04 00:59:53 Function: TestEventingRebInWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebInWithOps deleting in the background"
}
2021/12/04 00:59:55 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 01:00:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 01:00:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 01:00:02 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 01:00:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 01:00:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebInWithOps (460.75s)
=== RUN   TestEventingRebOutWithOps
2021/12/04 01:00:12 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/04 01:00:12 Removing node(s): [] from the cluster
2021/12/04 01:00:17 Rebalance progress: 100
2021/12/04 01:00:22 Waiting for app: TestEventingRebOutWithOps to get un-deployed
2021/12/04 01:00:22 App: TestEventingRebOutWithOps got un-deployed
2021/12/04 01:00:22 No eventing-con process running
2021/12/04 01:00:22 Function: TestEventingRebOutWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebOutWithOps not found"
 }
}
2021/12/04 01:00:24 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 01:00:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 01:00:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 01:00:31 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 01:00:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 01:00:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 01:00:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 01:00:41 Deploying app: TestEventingRebOutWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/04 01:00:42 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebOutWithOps' in metakv",
  "warnings": null
 }
}
2021/12/04 01:00:47 Waiting for app: TestEventingRebOutWithOps to get deployed
2021/12/04 01:00:52 Waiting for app: TestEventingRebOutWithOps to get deployed
2021/12/04 01:00:57 Waiting for app: TestEventingRebOutWithOps to get deployed
2021/12/04 01:01:02 Waiting for app: TestEventingRebOutWithOps to get deployed
2021/12/04 01:01:02 App: TestEventingRebOutWithOps got deployed
2021/12/04 01:01:02 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/04 01:01:02 Starting bucket ops to source bucket
2021/12/04 01:01:07 Rebalance progress: 12.5
2021/12/04 01:01:12 Rebalance progress: 12.5
2021/12/04 01:01:17 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 01:01:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 01:01:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 01:01:17 Waiting for src bucket item count to get to: 200000 curr count: 10630
2021/12/04 01:01:22 Waiting for src bucket item count to get to: 200000 curr count: 26533
2021/12/04 01:01:27 Waiting for src bucket item count to get to: 200000 curr count: 26533
2021/12/04 01:01:32 Waiting for src bucket item count to get to: 200000 curr count: 42793
2021/12/04 01:01:37 Waiting for src bucket item count to get to: 200000 curr count: 42793
2021/12/04 01:01:42 Waiting for src bucket item count to get to: 200000 curr count: 59063
2021/12/04 01:01:47 Waiting for src bucket item count to get to: 200000 curr count: 59063
2021/12/04 01:01:52 Waiting for src bucket item count to get to: 200000 curr count: 59063
2021/12/04 01:01:57 Waiting for src bucket item count to get to: 200000 curr count: 74294
2021/12/04 01:02:02 Waiting for src bucket item count to get to: 200000 curr count: 74294
2021/12/04 01:02:07 Waiting for src bucket item count to get to: 200000 curr count: 90197
2021/12/04 01:02:12 Waiting for src bucket item count to get to: 200000 curr count: 90197
2021/12/04 01:02:17 Waiting for src bucket item count to get to: 200000 curr count: 106423
2021/12/04 01:02:22 Waiting for src bucket item count to get to: 200000 curr count: 106423
2021/12/04 01:02:27 Waiting for src bucket item count to get to: 200000 curr count: 122503
2021/12/04 01:02:32 Waiting for src bucket item count to get to: 200000 curr count: 122503
2021/12/04 01:02:37 Waiting for src bucket item count to get to: 200000 curr count: 138693
2021/12/04 01:02:42 Waiting for src bucket item count to get to: 200000 curr count: 138693
2021/12/04 01:02:47 Waiting for src bucket item count to get to: 200000 curr count: 154831
2021/12/04 01:02:52 Waiting for src bucket item count to get to: 200000 curr count: 154831
2021/12/04 01:02:57 Waiting for src bucket item count to get to: 200000 curr count: 169863
2021/12/04 01:03:02 Waiting for src bucket item count to get to: 200000 curr count: 169863
2021/12/04 01:03:07 Waiting for src bucket item count to get to: 200000 curr count: 185489
2021/12/04 01:03:12 Waiting for src bucket item count to get to: 200000 curr count: 185489
2021/12/04 01:03:17 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/04 01:03:17 Post SET ops. Expected item count: 200000 got 200000
2021/12/04 01:03:17 Starting bucket ops to source bucket
2021/12/04 01:07:09 Waiting for dst bucket item count to get to: 0 curr count: 8334
2021/12/04 01:07:14 Waiting for dst bucket item count to get to: 0 curr count: 8334
2021/12/04 01:07:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 01:07:14 Post DELETE ops. Expected item count: 0 got 0
2021/12/04 01:07:14 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 01:07:19 Waiting for app: TestEventingRebOutWithOps to get un-deployed
2021/12/04 01:07:24 Waiting for app: TestEventingRebOutWithOps to get un-deployed
2021/12/04 01:07:29 Waiting for app: TestEventingRebOutWithOps to get un-deployed
2021/12/04 01:07:34 Waiting for app: TestEventingRebOutWithOps to get un-deployed
2021/12/04 01:07:34 App: TestEventingRebOutWithOps got un-deployed
2021/12/04 01:07:34 No eventing-con process running
2021/12/04 01:07:34 Function: TestEventingRebOutWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebOutWithOps deleting in the background"
}
2021/12/04 01:07:36 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 01:07:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 01:07:41 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 01:07:43 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 01:07:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 01:07:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebOutWithOps (460.65s)
=== RUN   TestEventingSwapRebWithOps
2021/12/04 01:07:58 Waiting for app: TestEventingSwapRebWithOps to get un-deployed
2021/12/04 01:07:58 App: TestEventingSwapRebWithOps got un-deployed
2021/12/04 01:07:58 No eventing-con process running
2021/12/04 01:07:58 Function: TestEventingSwapRebWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingSwapRebWithOps not found"
 }
}
2021/12/04 01:07:59 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 01:08:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 01:08:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 01:08:06 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 01:08:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 01:08:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 01:08:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 01:08:16 Deploying app: TestEventingSwapRebWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/04 01:08:17 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingSwapRebWithOps' in metakv",
  "warnings": null
 }
}
2021/12/04 01:08:22 Waiting for app: TestEventingSwapRebWithOps to get deployed
2021/12/04 01:08:27 Waiting for app: TestEventingSwapRebWithOps to get deployed
2021/12/04 01:08:32 Waiting for app: TestEventingSwapRebWithOps to get deployed
2021/12/04 01:08:37 Waiting for app: TestEventingSwapRebWithOps to get deployed
2021/12/04 01:08:37 App: TestEventingSwapRebWithOps got deployed
2021/12/04 01:08:37 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/04 01:08:37 Starting bucket ops to source bucket
2021/12/04 01:08:38 Removing node(s): [] from the cluster
2021/12/04 01:08:43 Rebalance progress: 12.5
2021/12/04 01:08:48 Rebalance progress: 12.5
2021/12/04 01:08:53 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 01:08:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 01:08:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 01:08:53 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/04 01:08:54 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/04 01:08:59 Rebalance progress: 12.5
2021/12/04 01:09:04 Rebalance progress: 12.5
2021/12/04 01:09:09 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 01:09:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 01:09:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 01:09:09 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/04 01:09:14 Rebalance progress: 12.5
2021/12/04 01:09:19 Rebalance progress: 12.5
2021/12/04 01:09:24 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 01:09:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 01:09:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 01:09:24 Waiting for src bucket item count to get to: 200000 curr count: 63855
2021/12/04 01:09:29 Waiting for src bucket item count to get to: 200000 curr count: 63855
2021/12/04 01:09:34 Waiting for src bucket item count to get to: 200000 curr count: 78500
2021/12/04 01:09:39 Waiting for src bucket item count to get to: 200000 curr count: 78500
2021/12/04 01:09:44 Waiting for src bucket item count to get to: 200000 curr count: 94410
2021/12/04 01:09:49 Waiting for src bucket item count to get to: 200000 curr count: 94410
2021/12/04 01:09:54 Waiting for src bucket item count to get to: 200000 curr count: 110369
2021/12/04 01:09:59 Waiting for src bucket item count to get to: 200000 curr count: 110369
2021/12/04 01:10:04 Waiting for src bucket item count to get to: 200000 curr count: 126604
2021/12/04 01:10:09 Waiting for src bucket item count to get to: 200000 curr count: 126604
2021/12/04 01:10:14 Waiting for src bucket item count to get to: 200000 curr count: 142765
2021/12/04 01:10:19 Waiting for src bucket item count to get to: 200000 curr count: 142765
2021/12/04 01:10:24 Waiting for src bucket item count to get to: 200000 curr count: 158907
2021/12/04 01:10:29 Waiting for src bucket item count to get to: 200000 curr count: 158907
2021/12/04 01:10:34 Waiting for src bucket item count to get to: 200000 curr count: 174034
2021/12/04 01:10:39 Waiting for src bucket item count to get to: 200000 curr count: 174034
2021/12/04 01:10:44 Waiting for src bucket item count to get to: 200000 curr count: 189976
2021/12/04 01:10:49 Waiting for src bucket item count to get to: 200000 curr count: 189976
2021/12/04 01:10:54 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/04 01:10:54 Post SET ops. Expected item count: 200000 got 200000
2021/12/04 01:10:54 Starting bucket ops to source bucket
2021/12/04 01:14:46 Waiting for dst bucket item count to get to: 0 curr count: 20684
2021/12/04 01:14:51 Waiting for dst bucket item count to get to: 0 curr count: 2590
2021/12/04 01:14:56 Waiting for dst bucket item count to get to: 0 curr count: 2590
2021/12/04 01:14:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 01:14:56 Post DELETE ops. Expected item count: 0 got 0
2021/12/04 01:14:56 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 01:15:01 Waiting for app: TestEventingSwapRebWithOps to get un-deployed
2021/12/04 01:15:06 Waiting for app: TestEventingSwapRebWithOps to get un-deployed
2021/12/04 01:15:11 Waiting for app: TestEventingSwapRebWithOps to get un-deployed
2021/12/04 01:15:16 Waiting for app: TestEventingSwapRebWithOps to get un-deployed
2021/12/04 01:15:16 App: TestEventingSwapRebWithOps got un-deployed
2021/12/04 01:15:16 No eventing-con process running
2021/12/04 01:15:16 Function: TestEventingSwapRebWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingSwapRebWithOps deleting in the background"
}
2021/12/04 01:15:18 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 01:15:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 01:15:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 01:15:25 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 01:15:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 01:15:30 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingSwapRebWithOps (461.90s)
=== RUN   TestKVRebInWithOps
2021/12/04 01:15:40 Waiting for app: TestKVRebInWithOps to get un-deployed
2021/12/04 01:15:40 App: TestKVRebInWithOps got un-deployed
2021/12/04 01:15:40 No eventing-con process running
2021/12/04 01:15:40 Function: TestKVRebInWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVRebInWithOps not found"
 }
}
2021/12/04 01:15:41 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 01:15:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 01:15:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 01:15:48 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 01:15:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 01:15:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 01:15:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 01:15:58 Deploying app: TestKVRebInWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/04 01:15:59 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVRebInWithOps' in metakv",
  "warnings": null
 }
}
2021/12/04 01:16:04 Waiting for app: TestKVRebInWithOps to get deployed
2021/12/04 01:16:09 Waiting for app: TestKVRebInWithOps to get deployed
2021/12/04 01:16:14 Waiting for app: TestKVRebInWithOps to get deployed
2021/12/04 01:16:19 Waiting for app: TestKVRebInWithOps to get deployed
2021/12/04 01:16:19 App: TestKVRebInWithOps got deployed
2021/12/04 01:16:19 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/04 01:16:19 Starting bucket ops to source bucket
2021/12/04 01:16:20 Removing node(s): [] from the cluster
2021/12/04 01:16:25 Rebalance progress: 12.5
2021/12/04 01:16:30 Rebalance progress: 12.5
2021/12/04 01:16:35 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 01:16:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 01:16:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 01:16:35 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/04 01:16:40 Rebalance progress: 12.5
2021/12/04 01:16:45 Rebalance progress: 12.5
2021/12/04 01:16:50 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 01:16:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 01:16:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 01:16:50 Waiting for src bucket item count to get to: 200000 curr count: 29736
2021/12/04 01:16:55 Waiting for src bucket item count to get to: 200000 curr count: 45834
2021/12/04 01:17:00 Waiting for src bucket item count to get to: 200000 curr count: 45834
2021/12/04 01:17:05 Waiting for src bucket item count to get to: 200000 curr count: 61466
2021/12/04 01:17:10 Waiting for src bucket item count to get to: 200000 curr count: 61466
2021/12/04 01:17:15 Waiting for src bucket item count to get to: 200000 curr count: 76421
2021/12/04 01:17:20 Waiting for src bucket item count to get to: 200000 curr count: 92374
2021/12/04 01:17:25 Waiting for src bucket item count to get to: 200000 curr count: 92374
2021/12/04 01:17:30 Waiting for src bucket item count to get to: 200000 curr count: 108499
2021/12/04 01:17:35 Waiting for src bucket item count to get to: 200000 curr count: 108499
2021/12/04 01:17:40 Waiting for src bucket item count to get to: 200000 curr count: 124430
2021/12/04 01:17:45 Waiting for src bucket item count to get to: 200000 curr count: 124430
2021/12/04 01:17:50 Waiting for src bucket item count to get to: 200000 curr count: 140727
2021/12/04 01:17:55 Waiting for src bucket item count to get to: 200000 curr count: 140727
2021/12/04 01:18:00 Waiting for src bucket item count to get to: 200000 curr count: 140727
2021/12/04 01:18:05 Waiting for src bucket item count to get to: 200000 curr count: 156802
2021/12/04 01:18:10 Waiting for src bucket item count to get to: 200000 curr count: 156802
2021/12/04 01:18:15 Waiting for src bucket item count to get to: 200000 curr count: 172004
2021/12/04 01:18:20 Waiting for src bucket item count to get to: 200000 curr count: 172004
2021/12/04 01:18:25 Waiting for src bucket item count to get to: 200000 curr count: 187768
2021/12/04 01:18:30 Waiting for src bucket item count to get to: 200000 curr count: 187768
2021/12/04 01:18:35 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/04 01:18:35 Post SET ops. Expected item count: 200000 got 200000
2021/12/04 01:18:35 Starting bucket ops to source bucket
2021/12/04 01:22:27 Waiting for dst bucket item count to get to: 0 curr count: 22890
2021/12/04 01:22:32 Waiting for dst bucket item count to get to: 0 curr count: 4823
2021/12/04 01:22:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 01:22:32 Post DELETE ops. Expected item count: 0 got 0
2021/12/04 01:22:32 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 01:22:37 Waiting for app: TestKVRebInWithOps to get un-deployed
2021/12/04 01:22:42 Waiting for app: TestKVRebInWithOps to get un-deployed
2021/12/04 01:22:47 Waiting for app: TestKVRebInWithOps to get un-deployed
2021/12/04 01:22:52 Waiting for app: TestKVRebInWithOps to get un-deployed
2021/12/04 01:22:52 App: TestKVRebInWithOps got un-deployed
2021/12/04 01:22:52 No eventing-con process running
2021/12/04 01:22:52 Function: TestKVRebInWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVRebInWithOps deleting in the background"
}
2021/12/04 01:22:54 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 01:22:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 01:22:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 01:23:01 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 01:23:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 01:23:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebInWithOps (456.22s)
=== RUN   TestKVRebOutWithOps
2021/12/04 01:23:11 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/04 01:23:11 Removing node(s): [] from the cluster
2021/12/04 01:23:16 Rebalance progress: 100
2021/12/04 01:23:21 Waiting for app: TestKVRebOutWithOps to get un-deployed
2021/12/04 01:23:21 App: TestKVRebOutWithOps got un-deployed
2021/12/04 01:23:21 No eventing-con process running
2021/12/04 01:23:21 Function: TestKVRebOutWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVRebOutWithOps not found"
 }
}
2021/12/04 01:23:23 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 01:23:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 01:23:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 01:23:30 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 01:23:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 01:23:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 01:23:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 01:23:40 Deploying app: TestKVRebOutWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/04 01:23:41 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVRebOutWithOps' in metakv",
  "warnings": null
 }
}
2021/12/04 01:23:46 Waiting for app: TestKVRebOutWithOps to get deployed
2021/12/04 01:23:51 Waiting for app: TestKVRebOutWithOps to get deployed
2021/12/04 01:23:56 Waiting for app: TestKVRebOutWithOps to get deployed
2021/12/04 01:24:01 Waiting for app: TestKVRebOutWithOps to get deployed
2021/12/04 01:24:01 App: TestKVRebOutWithOps got deployed
2021/12/04 01:24:01 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/04 01:24:01 Starting bucket ops to source bucket
2021/12/04 01:24:06 Rebalance progress: 12.5
2021/12/04 01:24:11 Rebalance progress: 12.5
2021/12/04 01:24:16 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 01:24:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 01:24:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 01:24:16 Waiting for src bucket item count to get to: 200000 curr count: 12437
2021/12/04 01:24:21 Waiting for src bucket item count to get to: 200000 curr count: 28354
2021/12/04 01:24:26 Waiting for src bucket item count to get to: 200000 curr count: 28354
2021/12/04 01:24:31 Waiting for src bucket item count to get to: 200000 curr count: 44465
2021/12/04 01:24:36 Waiting for src bucket item count to get to: 200000 curr count: 44465
2021/12/04 01:24:41 Waiting for src bucket item count to get to: 200000 curr count: 60455
2021/12/04 01:24:46 Waiting for src bucket item count to get to: 200000 curr count: 60455
2021/12/04 01:24:51 Waiting for src bucket item count to get to: 200000 curr count: 60455
2021/12/04 01:24:56 Waiting for src bucket item count to get to: 200000 curr count: 75571
2021/12/04 01:25:01 Waiting for src bucket item count to get to: 200000 curr count: 75571
2021/12/04 01:25:06 Waiting for src bucket item count to get to: 200000 curr count: 91269
2021/12/04 01:25:11 Waiting for src bucket item count to get to: 200000 curr count: 91269
2021/12/04 01:25:16 Waiting for src bucket item count to get to: 200000 curr count: 107138
2021/12/04 01:25:21 Waiting for src bucket item count to get to: 200000 curr count: 107138
2021/12/04 01:25:26 Waiting for src bucket item count to get to: 200000 curr count: 122906
2021/12/04 01:25:31 Waiting for src bucket item count to get to: 200000 curr count: 122906
2021/12/04 01:25:36 Waiting for src bucket item count to get to: 200000 curr count: 138993
2021/12/04 01:25:41 Waiting for src bucket item count to get to: 200000 curr count: 138993
2021/12/04 01:25:46 Waiting for src bucket item count to get to: 200000 curr count: 155179
2021/12/04 01:25:51 Waiting for src bucket item count to get to: 200000 curr count: 155179
2021/12/04 01:25:56 Waiting for src bucket item count to get to: 200000 curr count: 170104
2021/12/04 01:26:01 Waiting for src bucket item count to get to: 200000 curr count: 170104
2021/12/04 01:26:06 Waiting for src bucket item count to get to: 200000 curr count: 185783
2021/12/04 01:26:11 Waiting for src bucket item count to get to: 200000 curr count: 185783
2021/12/04 01:26:16 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/04 01:26:16 Post SET ops. Expected item count: 200000 got 200000
2021/12/04 01:26:16 Starting bucket ops to source bucket
2021/12/04 01:30:08 Waiting for dst bucket item count to get to: 0 curr count: 5742
2021/12/04 01:30:13 Waiting for dst bucket item count to get to: 0 curr count: 5742
2021/12/04 01:30:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 01:30:13 Post DELETE ops. Expected item count: 0 got 0
2021/12/04 01:30:13 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 01:30:18 Waiting for app: TestKVRebOutWithOps to get un-deployed
2021/12/04 01:30:23 Waiting for app: TestKVRebOutWithOps to get un-deployed
2021/12/04 01:30:28 Waiting for app: TestKVRebOutWithOps to get un-deployed
2021/12/04 01:30:33 Waiting for app: TestKVRebOutWithOps to get un-deployed
2021/12/04 01:30:33 App: TestKVRebOutWithOps got un-deployed
2021/12/04 01:30:33 No eventing-con process running
2021/12/04 01:30:33 Function: TestKVRebOutWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVRebOutWithOps deleting in the background"
}
2021/12/04 01:30:35 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 01:30:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 01:30:40 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 01:30:41 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 01:30:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 01:30:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebOutWithOps (460.52s)
=== RUN   TestKVSwapRebWithOps
2021/12/04 01:30:56 Waiting for app: TestKVSwapRebWithOps to get un-deployed
2021/12/04 01:30:56 App: TestKVSwapRebWithOps got un-deployed
2021/12/04 01:30:56 No eventing-con process running
2021/12/04 01:30:56 Function: TestKVSwapRebWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVSwapRebWithOps not found"
 }
}
2021/12/04 01:30:58 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 01:31:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 01:31:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 01:31:05 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 01:31:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 01:31:10 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 01:31:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 01:31:15 Deploying app: TestKVSwapRebWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/04 01:31:16 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVSwapRebWithOps' in metakv",
  "warnings": null
 }
}
2021/12/04 01:31:21 Waiting for app: TestKVSwapRebWithOps to get deployed
2021/12/04 01:31:26 Waiting for app: TestKVSwapRebWithOps to get deployed
2021/12/04 01:31:31 Waiting for app: TestKVSwapRebWithOps to get deployed
2021/12/04 01:31:36 Waiting for app: TestKVSwapRebWithOps to get deployed
2021/12/04 01:31:36 App: TestKVSwapRebWithOps got deployed
2021/12/04 01:31:36 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/04 01:31:36 Starting bucket ops to source bucket
2021/12/04 01:31:36 Removing node(s): [] from the cluster
2021/12/04 01:31:41 Rebalance progress: 12.5
2021/12/04 01:31:46 Rebalance progress: 12.5
2021/12/04 01:31:51 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 01:31:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 01:31:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 01:31:51 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/04 01:31:52 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/04 01:31:57 Rebalance progress: 12.5
2021/12/04 01:32:02 Rebalance progress: 12.5
2021/12/04 01:32:07 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 01:32:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 01:32:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 01:32:07 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/04 01:32:12 Rebalance progress: 12.5
2021/12/04 01:32:17 Rebalance progress: 12.5
2021/12/04 01:32:22 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 01:32:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 01:32:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 01:32:22 Waiting for src bucket item count to get to: 200000 curr count: 65804
2021/12/04 01:32:27 Waiting for src bucket item count to get to: 200000 curr count: 65804
2021/12/04 01:32:32 Waiting for src bucket item count to get to: 200000 curr count: 80509
2021/12/04 01:32:37 Waiting for src bucket item count to get to: 200000 curr count: 80509
2021/12/04 01:32:42 Waiting for src bucket item count to get to: 200000 curr count: 96401
2021/12/04 01:32:47 Waiting for src bucket item count to get to: 200000 curr count: 96401
2021/12/04 01:32:52 Waiting for src bucket item count to get to: 200000 curr count: 112360
2021/12/04 01:32:57 Waiting for src bucket item count to get to: 200000 curr count: 112360
2021/12/04 01:33:02 Waiting for src bucket item count to get to: 200000 curr count: 128343
2021/12/04 01:33:07 Waiting for src bucket item count to get to: 200000 curr count: 128343
2021/12/04 01:33:12 Waiting for src bucket item count to get to: 200000 curr count: 144489
2021/12/04 01:33:17 Waiting for src bucket item count to get to: 200000 curr count: 144489
2021/12/04 01:33:22 Waiting for src bucket item count to get to: 200000 curr count: 160487
2021/12/04 01:33:27 Waiting for src bucket item count to get to: 200000 curr count: 160487
2021/12/04 01:33:32 Waiting for src bucket item count to get to: 200000 curr count: 175573
2021/12/04 01:33:37 Waiting for src bucket item count to get to: 200000 curr count: 175573
2021/12/04 01:33:42 Waiting for src bucket item count to get to: 200000 curr count: 175573
2021/12/04 01:33:47 Waiting for src bucket item count to get to: 200000 curr count: 191525
2021/12/04 01:33:52 Waiting for src bucket item count to get to: 200000 curr count: 191525
2021/12/04 01:33:57 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/04 01:33:57 Post SET ops. Expected item count: 200000 got 200000
2021/12/04 01:33:57 Starting bucket ops to source bucket
2021/12/04 01:37:49 Waiting for dst bucket item count to get to: 0 curr count: 8804
2021/12/04 01:37:54 Waiting for dst bucket item count to get to: 0 curr count: 8804
2021/12/04 01:37:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 01:37:54 Post DELETE ops. Expected item count: 0 got 0
2021/12/04 01:37:54 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 01:37:59 Waiting for app: TestKVSwapRebWithOps to get un-deployed
2021/12/04 01:38:04 Waiting for app: TestKVSwapRebWithOps to get un-deployed
2021/12/04 01:38:09 Waiting for app: TestKVSwapRebWithOps to get un-deployed
2021/12/04 01:38:14 Waiting for app: TestKVSwapRebWithOps to get un-deployed
2021/12/04 01:38:14 App: TestKVSwapRebWithOps got un-deployed
2021/12/04 01:38:14 No eventing-con process running
2021/12/04 01:38:14 Function: TestKVSwapRebWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVSwapRebWithOps deleting in the background"
}
2021/12/04 01:38:16 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 01:38:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 01:38:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 01:38:23 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 01:38:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 01:38:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVSwapRebWithOps (461.56s)
=== RUN   TestEventingRebInWithTimers
2021/12/04 01:38:38 Waiting for app: TestEventingRebInWithTimers to get un-deployed
2021/12/04 01:38:38 App: TestEventingRebInWithTimers got un-deployed
2021/12/04 01:38:38 No eventing-con process running
2021/12/04 01:38:38 Function: TestEventingRebInWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebInWithTimers not found"
 }
}
2021/12/04 01:38:40 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 01:38:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 01:38:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 01:38:46 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 01:38:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 01:38:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 01:38:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 01:38:56 Deploying app: TestEventingRebInWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/04 01:38:58 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebInWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/04 01:39:03 Waiting for app: TestEventingRebInWithTimers to get deployed
2021/12/04 01:39:08 Waiting for app: TestEventingRebInWithTimers to get deployed
2021/12/04 01:39:13 Waiting for app: TestEventingRebInWithTimers to get deployed
2021/12/04 01:39:18 Waiting for app: TestEventingRebInWithTimers to get deployed
2021/12/04 01:39:18 App: TestEventingRebInWithTimers got deployed
2021/12/04 01:39:18 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/04 01:39:18 Starting bucket ops to source bucket
2021/12/04 01:39:18 Removing node(s): [] from the cluster
2021/12/04 01:39:23 Rebalance progress: 12.5
2021/12/04 01:39:28 Rebalance progress: 12.5
2021/12/04 01:39:33 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 01:39:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 01:39:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 01:39:33 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/04 01:39:38 Rebalance progress: 12.5
2021/12/04 01:39:43 Rebalance progress: 12.5
2021/12/04 01:39:48 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 01:39:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 01:39:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 01:39:48 Waiting for src bucket item count to get to: 50000 curr count: 25851
2021/12/04 01:39:53 Waiting for src bucket item count to get to: 50000 curr count: 36041
2021/12/04 01:39:58 Waiting for src bucket item count to get to: 50000 curr count: 36041
2021/12/04 01:40:03 Waiting for src bucket item count to get to: 50000 curr count: 48079
2021/12/04 01:40:08 Waiting for src bucket item count to get to: 50000 curr count: 48079
2021/12/04 01:40:18 Waiting for dst bucket item count to get to: 50000 curr count: 47119
2021/12/04 01:40:23 Waiting for dst bucket item count to get to: 50000 curr count: 47119
2021/12/04 01:40:23 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/04 01:40:23 Post SET ops. Expected item count: 50000 got 50000
2021/12/04 01:40:23 Starting bucket ops to source bucket
2021/12/04 01:41:34 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/04 01:41:39 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/04 01:41:44 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/04 01:41:49 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/04 01:41:54 Waiting for dst bucket item count to get to: 0 curr count: 39873
2021/12/04 01:41:59 Waiting for dst bucket item count to get to: 0 curr count: 39873
2021/12/04 01:42:04 Waiting for dst bucket item count to get to: 0 curr count: 25740
2021/12/04 01:42:09 Waiting for dst bucket item count to get to: 0 curr count: 8730
2021/12/04 01:42:14 Waiting for dst bucket item count to get to: 0 curr count: 8730
2021/12/04 01:42:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 01:42:14 Post DELETE ops. Expected item count: 0 got 0
2021/12/04 01:42:14 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 01:42:19 Waiting for app: TestEventingRebInWithTimers to get un-deployed
2021/12/04 01:42:24 Waiting for app: TestEventingRebInWithTimers to get un-deployed
2021/12/04 01:42:29 Waiting for app: TestEventingRebInWithTimers to get un-deployed
2021/12/04 01:42:34 Waiting for app: TestEventingRebInWithTimers to get un-deployed
2021/12/04 01:42:34 App: TestEventingRebInWithTimers got un-deployed
2021/12/04 01:42:34 No eventing-con process running
2021/12/04 01:42:34 Function: TestEventingRebInWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebInWithTimers deleting in the background"
}
2021/12/04 01:42:36 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 01:42:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 01:42:41 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 01:42:43 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 01:42:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 01:42:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebInWithTimers (259.84s)
=== RUN   TestEventingRebOutWithTimers
2021/12/04 01:42:53 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/04 01:42:53 Removing node(s): [] from the cluster
2021/12/04 01:42:58 Rebalance progress: 100
2021/12/04 01:43:03 Waiting for app: TestEventingRebOutWithTimers to get un-deployed
2021/12/04 01:43:03 App: TestEventingRebOutWithTimers got un-deployed
2021/12/04 01:43:03 No eventing-con process running
2021/12/04 01:43:03 Function: TestEventingRebOutWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebOutWithTimers not found"
 }
}
2021/12/04 01:43:05 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 01:43:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 01:43:10 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 01:43:12 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 01:43:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 01:43:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 01:43:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 01:43:22 Deploying app: TestEventingRebOutWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/04 01:43:23 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebOutWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/04 01:43:28 Waiting for app: TestEventingRebOutWithTimers to get deployed
2021/12/04 01:43:33 Waiting for app: TestEventingRebOutWithTimers to get deployed
2021/12/04 01:43:38 Waiting for app: TestEventingRebOutWithTimers to get deployed
2021/12/04 01:43:43 Waiting for app: TestEventingRebOutWithTimers to get deployed
2021/12/04 01:43:43 App: TestEventingRebOutWithTimers got deployed
2021/12/04 01:43:43 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/04 01:43:43 Starting bucket ops to source bucket
2021/12/04 01:43:48 Rebalance progress: 12.5
2021/12/04 01:43:53 Rebalance progress: 12.5
2021/12/04 01:43:58 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 01:43:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 01:43:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 01:43:58 Waiting for src bucket item count to get to: 50000 curr count: 8526
2021/12/04 01:44:03 Waiting for src bucket item count to get to: 50000 curr count: 20528
2021/12/04 01:44:08 Waiting for src bucket item count to get to: 50000 curr count: 20528
2021/12/04 01:44:13 Waiting for src bucket item count to get to: 50000 curr count: 31878
2021/12/04 01:44:18 Waiting for src bucket item count to get to: 50000 curr count: 31878
2021/12/04 01:44:23 Waiting for src bucket item count to get to: 50000 curr count: 31878
2021/12/04 01:44:28 Waiting for src bucket item count to get to: 50000 curr count: 42933
2021/12/04 01:44:33 Waiting for src bucket item count to get to: 50000 curr count: 42933
2021/12/04 01:44:43 Waiting for dst bucket item count to get to: 50000 curr count: 41215
2021/12/04 01:44:48 Waiting for dst bucket item count to get to: 50000 curr count: 41215
2021/12/04 01:44:48 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/04 01:44:48 Post SET ops. Expected item count: 50000 got 50000
2021/12/04 01:44:48 Starting bucket ops to source bucket
2021/12/04 01:45:58 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/04 01:46:03 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/04 01:46:08 Waiting for dst bucket item count to get to: 0 curr count: 44560
2021/12/04 01:46:13 Waiting for dst bucket item count to get to: 0 curr count: 44560
2021/12/04 01:46:18 Waiting for dst bucket item count to get to: 0 curr count: 35457
2021/12/04 01:46:23 Waiting for dst bucket item count to get to: 0 curr count: 35457
2021/12/04 01:46:28 Waiting for dst bucket item count to get to: 0 curr count: 11689
2021/12/04 01:46:33 Waiting for dst bucket item count to get to: 0 curr count: 11689
2021/12/04 01:46:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 01:46:33 Post DELETE ops. Expected item count: 0 got 0
2021/12/04 01:46:34 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 01:46:39 Waiting for app: TestEventingRebOutWithTimers to get un-deployed
2021/12/04 01:46:44 Waiting for app: TestEventingRebOutWithTimers to get un-deployed
2021/12/04 01:46:49 Waiting for app: TestEventingRebOutWithTimers to get un-deployed
2021/12/04 01:46:54 Waiting for app: TestEventingRebOutWithTimers to get un-deployed
2021/12/04 01:46:54 App: TestEventingRebOutWithTimers got un-deployed
2021/12/04 01:46:54 No eventing-con process running
2021/12/04 01:46:54 Function: TestEventingRebOutWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebOutWithTimers deleting in the background"
}
2021/12/04 01:46:55 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 01:47:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 01:47:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 01:47:02 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 01:47:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 01:47:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebOutWithTimers (259.50s)
=== RUN   TestEventingSwapRebWithTimers
2021/12/04 01:47:17 Waiting for app: TestEventingSwapRebWithTimers to get un-deployed
2021/12/04 01:47:17 App: TestEventingSwapRebWithTimers got un-deployed
2021/12/04 01:47:17 No eventing-con process running
2021/12/04 01:47:17 Function: TestEventingSwapRebWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingSwapRebWithTimers not found"
 }
}
2021/12/04 01:47:19 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 01:47:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 01:47:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 01:47:26 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 01:47:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 01:47:31 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 01:47:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 01:47:36 Deploying app: TestEventingSwapRebWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/04 01:47:37 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingSwapRebWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/04 01:47:42 Waiting for app: TestEventingSwapRebWithTimers to get deployed
2021/12/04 01:47:47 Waiting for app: TestEventingSwapRebWithTimers to get deployed
2021/12/04 01:47:52 Waiting for app: TestEventingSwapRebWithTimers to get deployed
2021/12/04 01:47:57 Waiting for app: TestEventingSwapRebWithTimers to get deployed
2021/12/04 01:47:57 App: TestEventingSwapRebWithTimers got deployed
2021/12/04 01:47:57 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/04 01:47:57 Starting bucket ops to source bucket
2021/12/04 01:47:57 Removing node(s): [] from the cluster
2021/12/04 01:48:02 Rebalance progress: 12.5
2021/12/04 01:48:07 Rebalance progress: 12.5
2021/12/04 01:48:12 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 01:48:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 01:48:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 01:48:12 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/04 01:48:13 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/04 01:48:18 Rebalance progress: 12.5
2021/12/04 01:48:23 Rebalance progress: 12.5
2021/12/04 01:48:28 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 01:48:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 01:48:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 01:48:28 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/04 01:48:33 Rebalance progress: 12.5
2021/12/04 01:48:38 Rebalance progress: 12.5
2021/12/04 01:48:43 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 01:48:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 01:48:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 01:48:43 Waiting for src bucket item count to get to: 50000 curr count: 48821
2021/12/04 01:48:48 Waiting for src bucket item count to get to: 50000 curr count: 48821
2021/12/04 01:48:58 Waiting for dst bucket item count to get to: 50000 curr count: 45512
2021/12/04 01:49:03 Waiting for dst bucket item count to get to: 50000 curr count: 45512
2021/12/04 01:49:03 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/04 01:49:03 Post SET ops. Expected item count: 50000 got 50000
2021/12/04 01:49:03 Starting bucket ops to source bucket
2021/12/04 01:50:13 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/04 01:50:18 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/04 01:50:23 Waiting for dst bucket item count to get to: 0 curr count: 49994
2021/12/04 01:50:28 Waiting for dst bucket item count to get to: 0 curr count: 49994
2021/12/04 01:50:33 Waiting for dst bucket item count to get to: 0 curr count: 42135
2021/12/04 01:50:38 Waiting for dst bucket item count to get to: 0 curr count: 42135
2021/12/04 01:50:43 Waiting for dst bucket item count to get to: 0 curr count: 29463
2021/12/04 01:50:48 Waiting for dst bucket item count to get to: 0 curr count: 29463
2021/12/04 01:50:53 Waiting for dst bucket item count to get to: 0 curr count: 8061
2021/12/04 01:50:58 Waiting for dst bucket item count to get to: 0 curr count: 8061
2021/12/04 01:50:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 01:50:58 Post DELETE ops. Expected item count: 0 got 0
2021/12/04 01:50:58 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 01:51:03 Waiting for app: TestEventingSwapRebWithTimers to get un-deployed
2021/12/04 01:51:08 Waiting for app: TestEventingSwapRebWithTimers to get un-deployed
2021/12/04 01:51:13 Waiting for app: TestEventingSwapRebWithTimers to get un-deployed
2021/12/04 01:51:18 Waiting for app: TestEventingSwapRebWithTimers to get un-deployed
2021/12/04 01:51:18 App: TestEventingSwapRebWithTimers got un-deployed
2021/12/04 01:51:18 No eventing-con process running
2021/12/04 01:51:18 Function: TestEventingSwapRebWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingSwapRebWithTimers deleting in the background"
}
2021/12/04 01:51:20 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 01:51:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 01:51:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 01:51:27 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 01:51:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 01:51:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingSwapRebWithTimers (264.88s)
=== RUN   TestKVRebInWithTimers
2021/12/04 01:51:42 Waiting for app: TestKVRebInWithTimers to get un-deployed
2021/12/04 01:51:42 App: TestKVRebInWithTimers got un-deployed
2021/12/04 01:51:42 No eventing-con process running
2021/12/04 01:51:42 Function: TestKVRebInWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVRebInWithTimers not found"
 }
}
2021/12/04 01:51:44 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 01:51:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 01:51:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 01:51:51 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 01:51:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 01:51:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 01:52:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 01:52:01 Deploying app: TestKVRebInWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/04 01:52:02 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVRebInWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/04 01:52:07 Waiting for app: TestKVRebInWithTimers to get deployed
2021/12/04 01:52:12 Waiting for app: TestKVRebInWithTimers to get deployed
2021/12/04 01:52:17 Waiting for app: TestKVRebInWithTimers to get deployed
2021/12/04 01:52:22 Waiting for app: TestKVRebInWithTimers to get deployed
2021/12/04 01:52:22 App: TestKVRebInWithTimers got deployed
2021/12/04 01:52:22 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/04 01:52:22 Starting bucket ops to source bucket
2021/12/04 01:52:22 Removing node(s): [] from the cluster
2021/12/04 01:52:27 Rebalance progress: 12.5
2021/12/04 01:52:32 Rebalance progress: 12.5
2021/12/04 01:52:37 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 01:52:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 01:52:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 01:52:37 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/04 01:52:42 Rebalance progress: 12.5
2021/12/04 01:52:47 Rebalance progress: 12.5
2021/12/04 01:52:52 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 01:52:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 01:52:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 01:52:53 Waiting for src bucket item count to get to: 50000 curr count: 32207
2021/12/04 01:52:58 Waiting for src bucket item count to get to: 50000 curr count: 32207
2021/12/04 01:53:03 Waiting for src bucket item count to get to: 50000 curr count: 42549
2021/12/04 01:53:08 Waiting for src bucket item count to get to: 50000 curr count: 42549
2021/12/04 01:53:18 Waiting for dst bucket item count to get to: 50000 curr count: 39219
2021/12/04 01:53:23 Waiting for dst bucket item count to get to: 50000 curr count: 39219
2021/12/04 01:53:23 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/04 01:53:23 Post SET ops. Expected item count: 50000 got 50000
2021/12/04 01:53:23 Starting bucket ops to source bucket
2021/12/04 01:54:35 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/04 01:54:40 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/04 01:54:45 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/04 01:54:50 Waiting for dst bucket item count to get to: 0 curr count: 43128
2021/12/04 01:54:55 Waiting for dst bucket item count to get to: 0 curr count: 43128
2021/12/04 01:55:00 Waiting for dst bucket item count to get to: 0 curr count: 28862
2021/12/04 01:55:05 Waiting for dst bucket item count to get to: 0 curr count: 28862
2021/12/04 01:55:10 Waiting for dst bucket item count to get to: 0 curr count: 9498
2021/12/04 01:55:15 Waiting for dst bucket item count to get to: 0 curr count: 9498
2021/12/04 01:55:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 01:55:15 Post DELETE ops. Expected item count: 0 got 0
2021/12/04 01:55:15 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 01:55:20 Waiting for app: TestKVRebInWithTimers to get un-deployed
2021/12/04 01:55:25 Waiting for app: TestKVRebInWithTimers to get un-deployed
2021/12/04 01:55:31 Waiting for app: TestKVRebInWithTimers to get un-deployed
2021/12/04 01:55:36 Waiting for app: TestKVRebInWithTimers to get un-deployed
2021/12/04 01:55:36 App: TestKVRebInWithTimers got un-deployed
2021/12/04 01:55:36 No eventing-con process running
2021/12/04 01:55:36 Function: TestKVRebInWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVRebInWithTimers deleting in the background"
}
2021/12/04 01:55:37 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 01:55:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 01:55:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 01:55:44 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 01:55:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 01:55:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebInWithTimers (257.09s)
=== RUN   TestKVRebOutWithTimers
2021/12/04 01:55:54 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/04 01:55:54 Removing node(s): [] from the cluster
2021/12/04 01:55:59 Rebalance progress: 100
2021/12/04 01:56:04 Waiting for app: TestKVRebOutWithTimers to get un-deployed
2021/12/04 01:56:04 App: TestKVRebOutWithTimers got un-deployed
2021/12/04 01:56:04 No eventing-con process running
2021/12/04 01:56:04 Function: TestKVRebOutWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVRebOutWithTimers not found"
 }
}
2021/12/04 01:56:06 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 01:56:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 01:56:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 01:56:13 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 01:56:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 01:56:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 01:56:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 01:56:23 Deploying app: TestKVRebOutWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/04 01:56:24 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVRebOutWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/04 01:56:29 Waiting for app: TestKVRebOutWithTimers to get deployed
2021/12/04 01:56:34 Waiting for app: TestKVRebOutWithTimers to get deployed
2021/12/04 01:56:39 Waiting for app: TestKVRebOutWithTimers to get deployed
2021/12/04 01:56:44 Waiting for app: TestKVRebOutWithTimers to get deployed
2021/12/04 01:56:44 App: TestKVRebOutWithTimers got deployed
2021/12/04 01:56:44 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/04 01:56:44 Starting bucket ops to source bucket
2021/12/04 01:56:49 Rebalance progress: 12.5
2021/12/04 01:56:54 Rebalance progress: 12.5
2021/12/04 01:56:59 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 01:56:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 01:56:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 01:56:59 Waiting for src bucket item count to get to: 50000 curr count: 6425
2021/12/04 01:57:04 Waiting for src bucket item count to get to: 50000 curr count: 20187
2021/12/04 01:57:09 Waiting for src bucket item count to get to: 50000 curr count: 20187
2021/12/04 01:57:14 Waiting for src bucket item count to get to: 50000 curr count: 31639
2021/12/04 01:57:19 Waiting for src bucket item count to get to: 50000 curr count: 31639
2021/12/04 01:57:24 Waiting for src bucket item count to get to: 50000 curr count: 42845
2021/12/04 01:57:29 Waiting for src bucket item count to get to: 50000 curr count: 42845
2021/12/04 01:57:34 Waiting for src bucket item count to get to: 50000 curr count: 42845
2021/12/04 01:57:44 Waiting for dst bucket item count to get to: 50000 curr count: 34940
2021/12/04 01:57:49 Waiting for dst bucket item count to get to: 50000 curr count: 34940
2021/12/04 01:57:54 Waiting for dst bucket item count to get to: 50000 curr count: 44784
2021/12/04 01:57:54 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/04 01:57:54 Post SET ops. Expected item count: 50000 got 50000
2021/12/04 01:57:54 Starting bucket ops to source bucket
2021/12/04 01:59:05 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/04 01:59:10 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/04 01:59:15 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/04 01:59:20 Waiting for dst bucket item count to get to: 0 curr count: 47654
2021/12/04 01:59:25 Waiting for dst bucket item count to get to: 0 curr count: 47654
2021/12/04 01:59:30 Waiting for dst bucket item count to get to: 0 curr count: 30871
2021/12/04 01:59:35 Waiting for dst bucket item count to get to: 0 curr count: 30871
2021/12/04 01:59:40 Waiting for dst bucket item count to get to: 0 curr count: 15684
2021/12/04 01:59:45 Waiting for dst bucket item count to get to: 0 curr count: 15684
2021/12/04 01:59:50 Waiting for dst bucket item count to get to: 0 curr count: 590
2021/12/04 01:59:55 Waiting for dst bucket item count to get to: 0 curr count: 590
2021/12/04 01:59:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 01:59:55 Post DELETE ops. Expected item count: 0 got 0
2021/12/04 01:59:55 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 02:00:00 Waiting for app: TestKVRebOutWithTimers to get un-deployed
2021/12/04 02:00:05 Waiting for app: TestKVRebOutWithTimers to get un-deployed
2021/12/04 02:00:10 Waiting for app: TestKVRebOutWithTimers to get un-deployed
2021/12/04 02:00:15 Waiting for app: TestKVRebOutWithTimers to get un-deployed
2021/12/04 02:00:15 App: TestKVRebOutWithTimers got un-deployed
2021/12/04 02:00:15 No eventing-con process running
2021/12/04 02:00:15 Function: TestKVRebOutWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVRebOutWithTimers deleting in the background"
}
2021/12/04 02:00:17 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:00:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 02:00:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 02:00:23 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:00:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 02:00:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebOutWithTimers (279.35s)
=== RUN   TestKVSwapRebWithTimers
2021/12/04 02:00:39 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/04 02:00:39 App: TestKVSwapRebWithTimers got un-deployed
2021/12/04 02:00:39 No eventing-con process running
2021/12/04 02:00:39 Function: TestKVSwapRebWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVSwapRebWithTimers not found"
 }
}
2021/12/04 02:00:40 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:00:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 02:00:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 02:00:47 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:00:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 02:00:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:00:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 02:00:57 Deploying app: TestKVSwapRebWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/04 02:00:58 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVSwapRebWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/04 02:01:03 Waiting for app: TestKVSwapRebWithTimers to get deployed
2021/12/04 02:01:08 Waiting for app: TestKVSwapRebWithTimers to get deployed
2021/12/04 02:01:13 Waiting for app: TestKVSwapRebWithTimers to get deployed
2021/12/04 02:01:18 Waiting for app: TestKVSwapRebWithTimers to get deployed
2021/12/04 02:01:18 App: TestKVSwapRebWithTimers got deployed
2021/12/04 02:01:18 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/04 02:01:18 Starting bucket ops to source bucket
2021/12/04 02:01:19 Removing node(s): [] from the cluster
2021/12/04 02:01:24 Rebalance progress: 12.5
2021/12/04 02:01:29 Rebalance progress: 12.5
2021/12/04 02:01:34 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:01:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:01:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 02:01:34 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/04 02:01:35 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/04 02:01:40 Rebalance progress: 12.5
2021/12/04 02:01:45 Rebalance progress: 12.5
2021/12/04 02:01:50 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:01:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:01:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 02:01:50 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/04 02:01:55 Rebalance progress: 12.5
2021/12/04 02:02:00 Rebalance progress: 12.5
2021/12/04 02:02:05 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:02:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:02:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 02:02:05 Waiting for src bucket item count to get to: 50000 curr count: 48420
2021/12/04 02:02:10 Waiting for src bucket item count to get to: 50000 curr count: 48420
2021/12/04 02:02:20 Waiting for dst bucket item count to get to: 50000 curr count: 40807
2021/12/04 02:02:25 Waiting for dst bucket item count to get to: 50000 curr count: 40807
2021/12/04 02:02:30 Waiting for dst bucket item count to get to: 50000 curr count: 48944
2021/12/04 02:02:35 Waiting for dst bucket item count to get to: 50000 curr count: 48944
2021/12/04 02:02:35 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/04 02:02:35 Post SET ops. Expected item count: 50000 got 50000
2021/12/04 02:02:35 Starting bucket ops to source bucket
2021/12/04 02:03:46 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/04 02:03:51 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/04 02:03:56 Waiting for dst bucket item count to get to: 0 curr count: 49077
2021/12/04 02:04:01 Waiting for dst bucket item count to get to: 0 curr count: 49077
2021/12/04 02:04:06 Waiting for dst bucket item count to get to: 0 curr count: 36667
2021/12/04 02:04:11 Waiting for dst bucket item count to get to: 0 curr count: 36667
2021/12/04 02:04:16 Waiting for dst bucket item count to get to: 0 curr count: 18417
2021/12/04 02:04:21 Waiting for dst bucket item count to get to: 0 curr count: 18417
2021/12/04 02:04:26 Waiting for dst bucket item count to get to: 0 curr count: 6558
2021/12/04 02:04:31 Waiting for dst bucket item count to get to: 0 curr count: 6558
2021/12/04 02:04:31 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 02:04:31 Post DELETE ops. Expected item count: 0 got 0
2021/12/04 02:04:32 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 02:04:37 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/04 02:04:42 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/04 02:04:47 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/04 02:04:52 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/04 02:04:57 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/04 02:05:02 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/04 02:05:02 App: TestKVSwapRebWithTimers got un-deployed
2021/12/04 02:05:02 No eventing-con process running
2021/12/04 02:05:02 Function: TestKVSwapRebWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVSwapRebWithTimers deleting in the background"
}
2021/12/04 02:05:03 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:05:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 02:05:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 02:05:10 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:05:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 02:05:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVSwapRebWithTimers (286.79s)
=== RUN   TestDuplicateTimerCreationEventingRebIn
2021/12/04 02:05:25 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/04 02:05:25 App: TestDuplicateTimerCreationEventingRebIn got un-deployed
2021/12/04 02:05:25 No eventing-con process running
2021/12/04 02:05:25 Function: TestDuplicateTimerCreationEventingRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationEventingRebIn not found"
 }
}
2021/12/04 02:05:27 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:05:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 02:05:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 02:05:34 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:05:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 02:05:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:05:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 02:05:44 Deploying app: TestDuplicateTimerCreationEventingRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/04 02:05:45 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationEventingRebIn' in metakv",
  "warnings": null
 }
}
2021/12/04 02:05:50 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get deployed
2021/12/04 02:05:55 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get deployed
2021/12/04 02:06:00 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get deployed
2021/12/04 02:06:05 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get deployed
2021/12/04 02:06:05 App: TestDuplicateTimerCreationEventingRebIn got deployed
2021/12/04 02:06:05 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/04 02:06:05 Starting bucket ops to source bucket
2021/12/04 02:06:05 Removing node(s): [] from the cluster
2021/12/04 02:06:10 Rebalance progress: 12.5
2021/12/04 02:06:15 Rebalance progress: 12.5
2021/12/04 02:06:20 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:06:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:06:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 02:06:20 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/04 02:06:26 Rebalance progress: 12.5
2021/12/04 02:06:31 Rebalance progress: 12.5
2021/12/04 02:06:36 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:06:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:06:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 02:06:36 Waiting for src bucket item count to get to: 50000 curr count: 30995
2021/12/04 02:06:41 Waiting for src bucket item count to get to: 50000 curr count: 42148
2021/12/04 02:06:46 Waiting for src bucket item count to get to: 50000 curr count: 42148
2021/12/04 02:06:56 Waiting for dst bucket item count to get to: 50000 curr count: 27124
2021/12/04 02:07:01 Waiting for dst bucket item count to get to: 50000 curr count: 27124
2021/12/04 02:07:06 Waiting for dst bucket item count to get to: 50000 curr count: 35399
2021/12/04 02:07:11 Waiting for dst bucket item count to get to: 50000 curr count: 35399
2021/12/04 02:07:16 Waiting for dst bucket item count to get to: 50000 curr count: 49263
2021/12/04 02:07:21 Waiting for dst bucket item count to get to: 50000 curr count: 49263
2021/12/04 02:07:26 Waiting for dst bucket item count to get to: 50000 curr count: 49263
2021/12/04 02:07:26 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/04 02:07:26 Post SET ops. Expected item count: 50000 got 50000
2021/12/04 02:07:26 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 02:07:31 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/04 02:07:36 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/04 02:07:41 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/04 02:07:46 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/04 02:07:51 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/04 02:07:56 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/04 02:07:56 App: TestDuplicateTimerCreationEventingRebIn got un-deployed
2021/12/04 02:07:56 No eventing-con process running
2021/12/04 02:07:56 Function: TestDuplicateTimerCreationEventingRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationEventingRebIn deleting in the background"
}
2021/12/04 02:07:58 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:08:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 02:08:08 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/04 02:08:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 02:08:09 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:08:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 02:08:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationEventingRebIn (179.20s)
=== RUN   TestDuplicateTimerCreationEventingRebOut
2021/12/04 02:08:19 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/04 02:08:20 Removing node(s): [] from the cluster
2021/12/04 02:08:25 Rebalance progress: 100
2021/12/04 02:08:30 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/04 02:08:30 App: TestDuplicateTimerCreationEventingRebOut got un-deployed
2021/12/04 02:08:30 No eventing-con process running
2021/12/04 02:08:30 Function: TestDuplicateTimerCreationEventingRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationEventingRebOut not found"
 }
}
2021/12/04 02:08:32 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:08:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 02:08:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 02:08:38 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:08:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 02:08:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:08:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 02:08:48 Deploying app: TestDuplicateTimerCreationEventingRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/04 02:08:49 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationEventingRebOut' in metakv",
  "warnings": null
 }
}
2021/12/04 02:08:54 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get deployed
2021/12/04 02:08:59 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get deployed
2021/12/04 02:09:04 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get deployed
2021/12/04 02:09:09 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get deployed
2021/12/04 02:09:09 App: TestDuplicateTimerCreationEventingRebOut got deployed
2021/12/04 02:09:09 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/04 02:09:09 Starting bucket ops to source bucket
2021/12/04 02:09:15 Rebalance progress: 12.5
2021/12/04 02:09:20 Rebalance progress: 12.5
2021/12/04 02:09:25 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:09:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:09:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 02:09:25 Waiting for src bucket item count to get to: 50000 curr count: 12449
2021/12/04 02:09:30 Waiting for src bucket item count to get to: 50000 curr count: 12449
2021/12/04 02:09:35 Waiting for src bucket item count to get to: 50000 curr count: 26150
2021/12/04 02:09:40 Waiting for src bucket item count to get to: 50000 curr count: 26150
2021/12/04 02:09:45 Waiting for src bucket item count to get to: 50000 curr count: 36903
2021/12/04 02:09:50 Waiting for src bucket item count to get to: 50000 curr count: 36903
2021/12/04 02:09:55 Waiting for src bucket item count to get to: 50000 curr count: 47411
2021/12/04 02:10:00 Waiting for src bucket item count to get to: 50000 curr count: 47411
2021/12/04 02:10:10 Waiting for dst bucket item count to get to: 50000 curr count: 38917
2021/12/04 02:10:15 Waiting for dst bucket item count to get to: 50000 curr count: 38917
2021/12/04 02:10:20 Waiting for dst bucket item count to get to: 50000 curr count: 47672
2021/12/04 02:10:25 Waiting for dst bucket item count to get to: 50000 curr count: 47672
2021/12/04 02:10:25 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/04 02:10:25 Post SET ops. Expected item count: 50000 got 50000
2021/12/04 02:10:25 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 02:10:30 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/04 02:10:35 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/04 02:10:40 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/04 02:10:45 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/04 02:10:50 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/04 02:10:55 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/04 02:10:55 App: TestDuplicateTimerCreationEventingRebOut got un-deployed
2021/12/04 02:10:55 No eventing-con process running
2021/12/04 02:10:55 Function: TestDuplicateTimerCreationEventingRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationEventingRebOut deleting in the background"
}
2021/12/04 02:10:57 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:11:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 02:11:07 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/04 02:11:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 02:11:09 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:11:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 02:11:19 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/04 02:11:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationEventingRebOut (184.21s)
=== RUN   TestDuplicateTimerCreationEventingSwapReb
2021/12/04 02:11:29 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/04 02:11:29 App: TestDuplicateTimerCreationEventingSwapReb got un-deployed
2021/12/04 02:11:29 No eventing-con process running
2021/12/04 02:11:29 Function: TestDuplicateTimerCreationEventingSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationEventingSwapReb not found"
 }
}
2021/12/04 02:11:31 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:11:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 02:11:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 02:11:37 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:11:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 02:11:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:11:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 02:11:47 Deploying app: TestDuplicateTimerCreationEventingSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/04 02:11:48 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationEventingSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/04 02:11:53 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get deployed
2021/12/04 02:11:58 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get deployed
2021/12/04 02:12:03 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get deployed
2021/12/04 02:12:08 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get deployed
2021/12/04 02:12:08 App: TestDuplicateTimerCreationEventingSwapReb got deployed
2021/12/04 02:12:08 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/04 02:12:08 Starting bucket ops to source bucket
2021/12/04 02:12:09 Removing node(s): [] from the cluster
2021/12/04 02:12:14 Rebalance progress: 12.5
2021/12/04 02:12:19 Rebalance progress: 12.5
2021/12/04 02:12:24 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:12:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:12:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 02:12:24 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/04 02:12:25 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/04 02:12:30 Rebalance progress: 12.5
2021/12/04 02:12:35 Rebalance progress: 12.5
2021/12/04 02:12:40 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:12:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:12:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 02:12:40 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/04 02:12:45 Rebalance progress: 12.5
2021/12/04 02:12:50 Rebalance progress: 12.5
2021/12/04 02:12:55 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:12:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:12:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 02:12:55 Waiting for src bucket item count to get to: 50000 curr count: 47560
2021/12/04 02:13:00 Waiting for src bucket item count to get to: 50000 curr count: 47560
2021/12/04 02:13:10 Waiting for dst bucket item count to get to: 50000 curr count: 35019
2021/12/04 02:13:15 Waiting for dst bucket item count to get to: 50000 curr count: 35019
2021/12/04 02:13:20 Waiting for dst bucket item count to get to: 50000 curr count: 41178
2021/12/04 02:13:25 Waiting for dst bucket item count to get to: 50000 curr count: 41178
2021/12/04 02:13:25 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/04 02:13:25 Post SET ops. Expected item count: 50000 got 50000
2021/12/04 02:13:25 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 02:13:30 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/04 02:13:35 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/04 02:13:40 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/04 02:13:45 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/04 02:13:50 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/04 02:13:55 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/04 02:13:55 App: TestDuplicateTimerCreationEventingSwapReb got un-deployed
2021/12/04 02:13:55 No eventing-con process running
2021/12/04 02:13:55 Function: TestDuplicateTimerCreationEventingSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationEventingSwapReb deleting in the background"
}
2021/12/04 02:13:57 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:14:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 02:14:07 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/04 02:14:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 02:14:09 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:14:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 02:14:19 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/04 02:14:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationEventingSwapReb (180.20s)
=== RUN   TestDuplicateTimerCreationKVRebIn
2021/12/04 02:14:29 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/04 02:14:29 App: TestDuplicateTimerCreationKVRebIn got un-deployed
2021/12/04 02:14:29 No eventing-con process running
2021/12/04 02:14:29 Function: TestDuplicateTimerCreationKVRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationKVRebIn not found"
 }
}
2021/12/04 02:14:31 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:14:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 02:14:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 02:14:37 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:14:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 02:14:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:14:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 02:14:48 Deploying app: TestDuplicateTimerCreationKVRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/04 02:14:49 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationKVRebIn' in metakv",
  "warnings": null
 }
}
2021/12/04 02:14:54 Waiting for app: TestDuplicateTimerCreationKVRebIn to get deployed
2021/12/04 02:14:59 Waiting for app: TestDuplicateTimerCreationKVRebIn to get deployed
2021/12/04 02:15:04 Waiting for app: TestDuplicateTimerCreationKVRebIn to get deployed
2021/12/04 02:15:09 Waiting for app: TestDuplicateTimerCreationKVRebIn to get deployed
2021/12/04 02:15:09 App: TestDuplicateTimerCreationKVRebIn got deployed
2021/12/04 02:15:09 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/04 02:15:09 Starting bucket ops to source bucket
2021/12/04 02:15:09 Removing node(s): [] from the cluster
2021/12/04 02:15:14 Rebalance progress: 12.5
2021/12/04 02:15:19 Rebalance progress: 12.5
2021/12/04 02:15:24 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:15:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:15:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 02:15:24 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/04 02:15:29 Rebalance progress: 12.5
2021/12/04 02:15:34 Rebalance progress: 12.5
2021/12/04 02:15:39 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:15:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:15:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 02:15:39 Waiting for src bucket item count to get to: 50000 curr count: 26037
2021/12/04 02:15:44 Waiting for src bucket item count to get to: 50000 curr count: 37139
2021/12/04 02:15:49 Waiting for src bucket item count to get to: 50000 curr count: 37139
2021/12/04 02:15:54 Waiting for src bucket item count to get to: 50000 curr count: 47981
2021/12/04 02:15:59 Waiting for src bucket item count to get to: 50000 curr count: 47981
2021/12/04 02:16:09 Waiting for dst bucket item count to get to: 50000 curr count: 35536
2021/12/04 02:16:14 Waiting for dst bucket item count to get to: 50000 curr count: 35536
2021/12/04 02:16:19 Waiting for dst bucket item count to get to: 50000 curr count: 43982
2021/12/04 02:16:24 Waiting for dst bucket item count to get to: 50000 curr count: 43982
2021/12/04 02:16:24 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/04 02:16:24 Post SET ops. Expected item count: 50000 got 50000
2021/12/04 02:16:24 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 02:16:29 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/04 02:16:34 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/04 02:16:39 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/04 02:16:45 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/04 02:16:50 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/04 02:16:55 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/04 02:16:55 App: TestDuplicateTimerCreationKVRebIn got un-deployed
2021/12/04 02:16:55 No eventing-con process running
2021/12/04 02:16:55 Function: TestDuplicateTimerCreationKVRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationKVRebIn deleting in the background"
}
2021/12/04 02:16:56 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:17:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 02:17:06 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/04 02:17:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 02:17:08 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:17:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 02:17:18 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/04 02:17:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationKVRebIn (179.34s)
=== RUN   TestDuplicateTimerCreationKVRebOut
2021/12/04 02:17:23 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/04 02:17:24 Removing node(s): [] from the cluster
2021/12/04 02:17:29 Rebalance progress: 100
2021/12/04 02:17:34 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/04 02:17:34 App: TestDuplicateTimerCreationKVRebOut got un-deployed
2021/12/04 02:17:34 No eventing-con process running
2021/12/04 02:17:34 Function: TestDuplicateTimerCreationKVRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationKVRebOut not found"
 }
}
2021/12/04 02:17:35 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:17:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 02:17:40 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 02:17:42 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:17:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 02:17:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:17:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 02:17:52 Deploying app: TestDuplicateTimerCreationKVRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/04 02:17:53 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationKVRebOut' in metakv",
  "warnings": null
 }
}
2021/12/04 02:17:58 Waiting for app: TestDuplicateTimerCreationKVRebOut to get deployed
2021/12/04 02:18:03 Waiting for app: TestDuplicateTimerCreationKVRebOut to get deployed
2021/12/04 02:18:08 Waiting for app: TestDuplicateTimerCreationKVRebOut to get deployed
2021/12/04 02:18:13 Waiting for app: TestDuplicateTimerCreationKVRebOut to get deployed
2021/12/04 02:18:13 App: TestDuplicateTimerCreationKVRebOut got deployed
2021/12/04 02:18:13 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/04 02:18:13 Starting bucket ops to source bucket
2021/12/04 02:18:18 Rebalance progress: 12.5
2021/12/04 02:18:23 Rebalance progress: 12.5
2021/12/04 02:18:28 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:18:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:18:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 02:18:29 Waiting for src bucket item count to get to: 50000 curr count: 7031
2021/12/04 02:18:34 Waiting for src bucket item count to get to: 50000 curr count: 20629
2021/12/04 02:18:39 Waiting for src bucket item count to get to: 50000 curr count: 20629
2021/12/04 02:18:44 Waiting for src bucket item count to get to: 50000 curr count: 34859
2021/12/04 02:18:49 Waiting for src bucket item count to get to: 50000 curr count: 34859
2021/12/04 02:18:54 Waiting for src bucket item count to get to: 50000 curr count: 45242
2021/12/04 02:18:59 Waiting for src bucket item count to get to: 50000 curr count: 45242
2021/12/04 02:19:04 Waiting for src bucket item count to get to: 50000 curr count: 45242
2021/12/04 02:19:14 Waiting for dst bucket item count to get to: 50000 curr count: 27958
2021/12/04 02:19:19 Waiting for dst bucket item count to get to: 50000 curr count: 27958
2021/12/04 02:19:24 Waiting for dst bucket item count to get to: 50000 curr count: 39145
2021/12/04 02:19:29 Waiting for dst bucket item count to get to: 50000 curr count: 39145
2021/12/04 02:19:29 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/04 02:19:29 Post SET ops. Expected item count: 50000 got 50000
2021/12/04 02:19:29 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 02:19:34 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/04 02:19:39 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/04 02:19:44 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/04 02:19:49 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/04 02:19:54 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/04 02:19:54 App: TestDuplicateTimerCreationKVRebOut got un-deployed
2021/12/04 02:19:54 No eventing-con process running
2021/12/04 02:19:54 Function: TestDuplicateTimerCreationKVRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationKVRebOut deleting in the background"
}
2021/12/04 02:19:56 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:20:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 02:20:06 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/04 02:20:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 02:20:08 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:20:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 02:20:18 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/04 02:20:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationKVRebOut (179.62s)
=== RUN   TestDuplicateTimerCreationKVSwapReb
2021/12/04 02:20:28 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/04 02:20:28 App: TestDuplicateTimerCreationKVSwapReb got un-deployed
2021/12/04 02:20:28 No eventing-con process running
2021/12/04 02:20:28 Function: TestDuplicateTimerCreationKVSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationKVSwapReb not found"
 }
}
2021/12/04 02:20:30 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:20:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 02:20:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 02:20:36 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:20:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 02:20:41 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:20:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 02:20:46 Deploying app: TestDuplicateTimerCreationKVSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/04 02:20:48 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationKVSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/04 02:20:53 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get deployed
2021/12/04 02:20:58 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get deployed
2021/12/04 02:21:03 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get deployed
2021/12/04 02:21:08 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get deployed
2021/12/04 02:21:08 App: TestDuplicateTimerCreationKVSwapReb got deployed
2021/12/04 02:21:08 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/04 02:21:08 Starting bucket ops to source bucket
2021/12/04 02:21:08 Removing node(s): [] from the cluster
2021/12/04 02:21:13 Rebalance progress: 12.5
2021/12/04 02:21:18 Rebalance progress: 12.5
2021/12/04 02:21:23 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:21:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:21:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 02:21:23 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/04 02:21:24 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/04 02:21:29 Rebalance progress: 12.5
2021/12/04 02:21:34 Rebalance progress: 12.5
2021/12/04 02:21:39 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:21:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:21:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 02:21:39 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/04 02:21:44 Rebalance progress: 12.5
2021/12/04 02:21:49 Rebalance progress: 12.5
2021/12/04 02:21:54 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:21:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:21:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 02:21:54 Waiting for src bucket item count to get to: 50000 curr count: 48067
2021/12/04 02:21:59 Waiting for src bucket item count to get to: 50000 curr count: 48067
2021/12/04 02:22:09 Waiting for dst bucket item count to get to: 50000 curr count: 34640
2021/12/04 02:22:14 Waiting for dst bucket item count to get to: 50000 curr count: 34640
2021/12/04 02:22:19 Waiting for dst bucket item count to get to: 50000 curr count: 40596
2021/12/04 02:22:24 Waiting for dst bucket item count to get to: 50000 curr count: 40596
2021/12/04 02:22:24 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/04 02:22:24 Post SET ops. Expected item count: 50000 got 50000
2021/12/04 02:22:24 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 02:22:29 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/04 02:22:34 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/04 02:22:39 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/04 02:22:44 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/04 02:22:49 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/04 02:22:54 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/04 02:22:54 App: TestDuplicateTimerCreationKVSwapReb got un-deployed
2021/12/04 02:22:54 No eventing-con process running
2021/12/04 02:22:55 Function: TestDuplicateTimerCreationKVSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationKVSwapReb deleting in the background"
}
2021/12/04 02:22:56 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:23:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 02:23:06 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/04 02:23:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 02:23:08 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:23:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 02:23:18 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/04 02:23:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationKVSwapReb (180.32s)
=== RUN   TestDuplicateTimerFiringEventingRebIn
2021/12/04 02:23:28 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/04 02:23:28 App: TestDuplicateTimerFiringEventingRebIn got un-deployed
2021/12/04 02:23:28 No eventing-con process running
2021/12/04 02:23:28 Function: TestDuplicateTimerFiringEventingRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringEventingRebIn not found"
 }
}
2021/12/04 02:23:30 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:23:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 02:23:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 02:23:37 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:23:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 02:23:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:23:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 02:23:47 Deploying app: TestDuplicateTimerFiringEventingRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/04 02:23:48 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringEventingRebIn' in metakv",
  "warnings": null
 }
}
2021/12/04 02:23:53 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get deployed
2021/12/04 02:23:58 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get deployed
2021/12/04 02:24:03 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get deployed
2021/12/04 02:24:08 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get deployed
2021/12/04 02:24:08 App: TestDuplicateTimerFiringEventingRebIn got deployed
2021/12/04 02:24:08 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/04 02:24:08 Starting bucket ops to source bucket
2021/12/04 02:24:08 Removing node(s): [] from the cluster
2021/12/04 02:24:13 Rebalance progress: 12.5
2021/12/04 02:24:18 Rebalance progress: 12.5
2021/12/04 02:24:23 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:24:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:24:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 02:24:23 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/04 02:24:29 Rebalance progress: 12.5
2021/12/04 02:24:34 Rebalance progress: 12.5
2021/12/04 02:24:39 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:24:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:24:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 02:24:39 Waiting for src bucket item count to get to: 50000 curr count: 26287
2021/12/04 02:24:44 Waiting for src bucket item count to get to: 50000 curr count: 36025
2021/12/04 02:24:49 Waiting for src bucket item count to get to: 50000 curr count: 36025
2021/12/04 02:24:54 Waiting for src bucket item count to get to: 50000 curr count: 48439
2021/12/04 02:24:59 Waiting for src bucket item count to get to: 50000 curr count: 48439
2021/12/04 02:25:09 Waiting for dst bucket item count to get to: 50000 curr count: 44400
2021/12/04 02:25:14 Waiting for dst bucket item count to get to: 50000 curr count: 44400
2021/12/04 02:25:14 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/04 02:25:14 Post SET ops. Expected item count: 50000 got 50000
2021/12/04 02:25:14 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 02:25:19 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/04 02:25:24 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/04 02:25:29 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/04 02:25:34 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/04 02:25:39 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/04 02:25:44 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/04 02:25:44 App: TestDuplicateTimerFiringEventingRebIn got un-deployed
2021/12/04 02:25:44 No eventing-con process running
2021/12/04 02:25:44 Function: TestDuplicateTimerFiringEventingRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringEventingRebIn deleting in the background"
}
2021/12/04 02:25:46 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:25:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 02:25:56 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/04 02:25:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 02:25:58 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:26:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 02:26:08 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/04 02:26:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringEventingRebIn (169.35s)
=== RUN   TestDuplicateTimerFiringEventingRebOut
2021/12/04 02:26:13 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/04 02:26:13 Removing node(s): [] from the cluster
2021/12/04 02:26:18 Rebalance progress: 100
2021/12/04 02:26:23 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/04 02:26:23 App: TestDuplicateTimerFiringEventingRebOut got un-deployed
2021/12/04 02:26:23 No eventing-con process running
2021/12/04 02:26:23 Function: TestDuplicateTimerFiringEventingRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringEventingRebOut not found"
 }
}
2021/12/04 02:26:25 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:26:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 02:26:30 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 02:26:32 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:26:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 02:26:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:26:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 02:26:42 Deploying app: TestDuplicateTimerFiringEventingRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/04 02:26:43 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringEventingRebOut' in metakv",
  "warnings": null
 }
}
2021/12/04 02:26:48 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get deployed
2021/12/04 02:26:53 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get deployed
2021/12/04 02:26:58 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get deployed
2021/12/04 02:27:03 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get deployed
2021/12/04 02:27:03 App: TestDuplicateTimerFiringEventingRebOut got deployed
2021/12/04 02:27:03 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/04 02:27:03 Starting bucket ops to source bucket
2021/12/04 02:27:08 Rebalance progress: 12.5
2021/12/04 02:27:13 Rebalance progress: 12.5
2021/12/04 02:27:18 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:27:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:27:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 02:27:18 Waiting for src bucket item count to get to: 50000 curr count: 8037
2021/12/04 02:27:23 Waiting for src bucket item count to get to: 50000 curr count: 20618
2021/12/04 02:27:28 Waiting for src bucket item count to get to: 50000 curr count: 20618
2021/12/04 02:27:33 Waiting for src bucket item count to get to: 50000 curr count: 31770
2021/12/04 02:27:38 Waiting for src bucket item count to get to: 50000 curr count: 31770
2021/12/04 02:27:43 Waiting for src bucket item count to get to: 50000 curr count: 42711
2021/12/04 02:27:48 Waiting for src bucket item count to get to: 50000 curr count: 42711
2021/12/04 02:27:53 Waiting for src bucket item count to get to: 50000 curr count: 42711
2021/12/04 02:28:03 Waiting for dst bucket item count to get to: 50000 curr count: 37017
2021/12/04 02:28:08 Waiting for dst bucket item count to get to: 50000 curr count: 37017
2021/12/04 02:28:13 Waiting for dst bucket item count to get to: 50000 curr count: 48911
2021/12/04 02:28:18 Waiting for dst bucket item count to get to: 50000 curr count: 48911
2021/12/04 02:28:18 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/04 02:28:18 Post SET ops. Expected item count: 50000 got 50000
2021/12/04 02:28:18 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 02:28:23 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/04 02:28:28 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/04 02:28:33 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/04 02:28:39 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/04 02:28:44 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/04 02:28:49 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/04 02:28:49 App: TestDuplicateTimerFiringEventingRebOut got un-deployed
2021/12/04 02:28:49 No eventing-con process running
2021/12/04 02:28:49 Function: TestDuplicateTimerFiringEventingRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringEventingRebOut deleting in the background"
}
2021/12/04 02:28:50 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:28:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 02:29:00 Waiting for dst bucket item count to get to: 0 curr count: 24561
2021/12/04 02:29:05 Waiting for dst bucket item count to get to: 0 curr count: 24561
2021/12/04 02:29:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 02:29:07 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:29:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 02:29:17 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/04 02:29:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringEventingRebOut (189.77s)
=== RUN   TestDuplicateTimerFiringEventingSwapReb
2021/12/04 02:29:27 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/04 02:29:27 App: TestDuplicateTimerFiringEventingSwapReb got un-deployed
2021/12/04 02:29:27 No eventing-con process running
2021/12/04 02:29:27 Function: TestDuplicateTimerFiringEventingSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringEventingSwapReb not found"
 }
}
2021/12/04 02:29:29 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:29:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 02:29:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 02:29:36 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:29:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 02:29:41 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:29:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 02:29:46 Deploying app: TestDuplicateTimerFiringEventingSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/04 02:29:47 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringEventingSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/04 02:29:52 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get deployed
2021/12/04 02:29:57 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get deployed
2021/12/04 02:30:02 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get deployed
2021/12/04 02:30:07 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get deployed
2021/12/04 02:30:07 App: TestDuplicateTimerFiringEventingSwapReb got deployed
2021/12/04 02:30:07 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/04 02:30:07 Starting bucket ops to source bucket
2021/12/04 02:30:07 Removing node(s): [] from the cluster
2021/12/04 02:30:12 Rebalance progress: 12.5
2021/12/04 02:30:17 Rebalance progress: 12.5
2021/12/04 02:30:22 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:30:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:30:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 02:30:22 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/04 02:30:23 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/04 02:30:28 Rebalance progress: 12.5
2021/12/04 02:30:33 Rebalance progress: 12.5
2021/12/04 02:30:38 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:30:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:30:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 02:30:38 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/04 02:30:43 Rebalance progress: 12.5
2021/12/04 02:30:48 Rebalance progress: 12.5
2021/12/04 02:30:53 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:30:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:30:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 02:30:53 Waiting for src bucket item count to get to: 50000 curr count: 48983
2021/12/04 02:30:58 Waiting for src bucket item count to get to: 50000 curr count: 48983
2021/12/04 02:31:08 Waiting for dst bucket item count to get to: 50000 curr count: 40248
2021/12/04 02:31:13 Waiting for dst bucket item count to get to: 50000 curr count: 40248
2021/12/04 02:31:18 Waiting for dst bucket item count to get to: 50000 curr count: 48416
2021/12/04 02:31:23 Waiting for dst bucket item count to get to: 50000 curr count: 48416
2021/12/04 02:31:23 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/04 02:31:23 Post SET ops. Expected item count: 50000 got 50000
2021/12/04 02:31:23 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 02:31:28 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/04 02:31:33 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/04 02:31:38 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/04 02:31:44 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/04 02:31:49 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/04 02:31:54 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/04 02:31:54 App: TestDuplicateTimerFiringEventingSwapReb got un-deployed
2021/12/04 02:31:54 No eventing-con process running
2021/12/04 02:31:54 Function: TestDuplicateTimerFiringEventingSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringEventingSwapReb deleting in the background"
}
2021/12/04 02:31:55 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:32:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 02:32:05 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/04 02:32:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 02:32:07 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:32:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 02:32:17 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/04 02:32:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringEventingSwapReb (179.90s)
=== RUN   TestDuplicateTimerFiringKVRebIn
2021/12/04 02:32:27 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/04 02:32:27 App: TestDuplicateTimerFiringKVRebIn got un-deployed
2021/12/04 02:32:27 No eventing-con process running
2021/12/04 02:32:27 Function: TestDuplicateTimerFiringKVRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringKVRebIn not found"
 }
}
2021/12/04 02:32:29 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:32:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 02:32:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 02:32:36 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:32:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 02:32:41 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:32:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 02:32:46 Deploying app: TestDuplicateTimerFiringKVRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/04 02:32:47 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringKVRebIn' in metakv",
  "warnings": null
 }
}
2021/12/04 02:32:52 Waiting for app: TestDuplicateTimerFiringKVRebIn to get deployed
2021/12/04 02:32:57 Waiting for app: TestDuplicateTimerFiringKVRebIn to get deployed
2021/12/04 02:33:02 Waiting for app: TestDuplicateTimerFiringKVRebIn to get deployed
2021/12/04 02:33:07 Waiting for app: TestDuplicateTimerFiringKVRebIn to get deployed
2021/12/04 02:33:07 App: TestDuplicateTimerFiringKVRebIn got deployed
2021/12/04 02:33:07 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/04 02:33:07 Starting bucket ops to source bucket
2021/12/04 02:33:07 Removing node(s): [] from the cluster
2021/12/04 02:33:12 Rebalance progress: 12.5
2021/12/04 02:33:17 Rebalance progress: 12.5
2021/12/04 02:33:22 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:33:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:33:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 02:33:22 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/04 02:33:28 Rebalance progress: 12.5
2021/12/04 02:33:33 Rebalance progress: 12.5
2021/12/04 02:33:38 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:33:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:33:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 02:33:38 Waiting for src bucket item count to get to: 50000 curr count: 26534
2021/12/04 02:33:43 Waiting for src bucket item count to get to: 50000 curr count: 37258
2021/12/04 02:33:48 Waiting for src bucket item count to get to: 50000 curr count: 37258
2021/12/04 02:33:53 Waiting for src bucket item count to get to: 50000 curr count: 48950
2021/12/04 02:33:58 Waiting for src bucket item count to get to: 50000 curr count: 48950
2021/12/04 02:34:08 Waiting for dst bucket item count to get to: 50000 curr count: 44267
2021/12/04 02:34:13 Waiting for dst bucket item count to get to: 50000 curr count: 44267
2021/12/04 02:34:13 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/04 02:34:13 Post SET ops. Expected item count: 50000 got 50000
2021/12/04 02:34:13 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 02:34:18 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/04 02:34:23 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/04 02:34:28 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/04 02:34:33 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/04 02:34:38 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/04 02:34:43 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/04 02:34:43 App: TestDuplicateTimerFiringKVRebIn got un-deployed
2021/12/04 02:34:43 No eventing-con process running
2021/12/04 02:34:43 Function: TestDuplicateTimerFiringKVRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringKVRebIn deleting in the background"
}
2021/12/04 02:34:45 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:34:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 02:34:55 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/04 02:34:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 02:34:57 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:35:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 02:35:07 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/04 02:35:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringKVRebIn (169.38s)
=== RUN   TestDuplicateTimerFiringKVRebOut
2021/12/04 02:35:12 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/04 02:35:12 Removing node(s): [] from the cluster
2021/12/04 02:35:17 Rebalance progress: 100
2021/12/04 02:35:22 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/04 02:35:22 App: TestDuplicateTimerFiringKVRebOut got un-deployed
2021/12/04 02:35:22 No eventing-con process running
2021/12/04 02:35:22 Function: TestDuplicateTimerFiringKVRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringKVRebOut not found"
 }
}
2021/12/04 02:35:24 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:35:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 02:35:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 02:35:31 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:35:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 02:35:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:35:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 02:35:41 Deploying app: TestDuplicateTimerFiringKVRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/04 02:35:42 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringKVRebOut' in metakv",
  "warnings": null
 }
}
2021/12/04 02:35:47 Waiting for app: TestDuplicateTimerFiringKVRebOut to get deployed
2021/12/04 02:35:52 Waiting for app: TestDuplicateTimerFiringKVRebOut to get deployed
2021/12/04 02:35:57 Waiting for app: TestDuplicateTimerFiringKVRebOut to get deployed
2021/12/04 02:36:02 Waiting for app: TestDuplicateTimerFiringKVRebOut to get deployed
2021/12/04 02:36:02 App: TestDuplicateTimerFiringKVRebOut got deployed
2021/12/04 02:36:02 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/04 02:36:02 Starting bucket ops to source bucket
2021/12/04 02:36:07 Rebalance progress: 12.5
2021/12/04 02:36:12 Rebalance progress: 12.5
2021/12/04 02:36:17 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:36:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:36:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 02:36:17 Waiting for src bucket item count to get to: 50000 curr count: 9933
2021/12/04 02:36:22 Waiting for src bucket item count to get to: 50000 curr count: 22605
2021/12/04 02:36:27 Waiting for src bucket item count to get to: 50000 curr count: 22605
2021/12/04 02:36:32 Waiting for src bucket item count to get to: 50000 curr count: 33554
2021/12/04 02:36:37 Waiting for src bucket item count to get to: 50000 curr count: 33554
2021/12/04 02:36:42 Waiting for src bucket item count to get to: 50000 curr count: 33554
2021/12/04 02:36:47 Waiting for src bucket item count to get to: 50000 curr count: 43438
2021/12/04 02:36:52 Waiting for src bucket item count to get to: 50000 curr count: 43438
2021/12/04 02:37:02 Waiting for dst bucket item count to get to: 50000 curr count: 37130
2021/12/04 02:37:07 Waiting for dst bucket item count to get to: 50000 curr count: 37130
2021/12/04 02:37:07 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/04 02:37:07 Post SET ops. Expected item count: 50000 got 50000
2021/12/04 02:37:07 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 02:37:12 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/04 02:37:17 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/04 02:37:22 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/04 02:37:27 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/04 02:37:32 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/04 02:37:37 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/04 02:37:37 App: TestDuplicateTimerFiringKVRebOut got un-deployed
2021/12/04 02:37:37 No eventing-con process running
2021/12/04 02:37:37 Function: TestDuplicateTimerFiringKVRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringKVRebOut deleting in the background"
}
2021/12/04 02:37:39 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:37:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 02:37:49 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/04 02:37:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 02:37:51 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:37:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 02:38:01 Waiting for dst bucket item count to get to: 0 curr count: 46592
2021/12/04 02:38:06 Waiting for dst bucket item count to get to: 0 curr count: 46592
2021/12/04 02:38:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringKVRebOut (179.35s)
=== RUN   TestDuplicateTimerFiringKVSwapReb
2021/12/04 02:38:16 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/04 02:38:16 App: TestDuplicateTimerFiringKVSwapReb got un-deployed
2021/12/04 02:38:16 No eventing-con process running
2021/12/04 02:38:16 Function: TestDuplicateTimerFiringKVSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringKVSwapReb not found"
 }
}
2021/12/04 02:38:18 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:38:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 02:38:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 02:38:25 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:38:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 02:38:30 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:38:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 02:38:35 Deploying app: TestDuplicateTimerFiringKVSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/04 02:38:36 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringKVSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/04 02:38:41 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get deployed
2021/12/04 02:38:46 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get deployed
2021/12/04 02:38:51 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get deployed
2021/12/04 02:38:56 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get deployed
2021/12/04 02:38:56 App: TestDuplicateTimerFiringKVSwapReb got deployed
2021/12/04 02:38:56 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/04 02:38:56 Starting bucket ops to source bucket
2021/12/04 02:38:56 Removing node(s): [] from the cluster
2021/12/04 02:39:01 Rebalance progress: 12.5
2021/12/04 02:39:06 Rebalance progress: 12.5
2021/12/04 02:39:11 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:39:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:39:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 02:39:11 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/04 02:39:12 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/04 02:39:17 Rebalance progress: 12.5
2021/12/04 02:39:22 Rebalance progress: 12.5
2021/12/04 02:39:27 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:39:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:39:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 02:39:27 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/04 02:39:32 Rebalance progress: 12.5
2021/12/04 02:39:37 Rebalance progress: 12.5
2021/12/04 02:39:42 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:39:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:39:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 02:39:42 Waiting for src bucket item count to get to: 50000 curr count: 49384
2021/12/04 02:39:47 Waiting for src bucket item count to get to: 50000 curr count: 49384
2021/12/04 02:39:57 Waiting for dst bucket item count to get to: 50000 curr count: 45422
2021/12/04 02:40:02 Waiting for dst bucket item count to get to: 50000 curr count: 45422
2021/12/04 02:40:02 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/04 02:40:02 Post SET ops. Expected item count: 50000 got 50000
2021/12/04 02:40:02 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 02:40:07 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/04 02:40:12 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/04 02:40:17 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/04 02:40:22 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/04 02:40:27 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/04 02:40:32 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/04 02:40:32 App: TestDuplicateTimerFiringKVSwapReb got un-deployed
2021/12/04 02:40:32 No eventing-con process running
2021/12/04 02:40:32 Function: TestDuplicateTimerFiringKVSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringKVSwapReb deleting in the background"
}
2021/12/04 02:40:34 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:40:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 02:40:44 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/04 02:40:49 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/04 02:40:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 02:40:51 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:40:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 02:41:01 Waiting for dst bucket item count to get to: 0 curr count: 45641
2021/12/04 02:41:06 Waiting for dst bucket item count to get to: 0 curr count: 45641
2021/12/04 02:41:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringKVSwapReb (180.02s)
=== RUN   TestKVRebOnUpdateBucketOpOneByOne
2021/12/04 02:41:16 Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 02:41:16 App: bucket_op_on_update got un-deployed
2021/12/04 02:41:16 No eventing-con process running
2021/12/04 02:41:16 Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/04 02:41:18 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:41:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 02:41:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 02:41:25 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:41:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 02:41:30 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:41:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 02:41:35 Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/04 02:41:36 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/04 02:41:41 Starting bucket ops to source bucket
2021/12/04 02:41:46 Waiting for app: bucket_op_on_update to get deployed
2021/12/04 02:41:51 Waiting for app: bucket_op_on_update to get deployed
2021/12/04 02:41:56 Waiting for app: bucket_op_on_update to get deployed
2021/12/04 02:41:56 App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:41:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:41:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 02:41:56 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/04 02:41:56 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/04 02:42:01 Rebalance progress: 12.5
2021/12/04 02:42:06 Rebalance progress: 12.5
2021/12/04 02:42:11 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:42:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:42:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 02:42:11 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/04 02:42:12 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/04 02:42:17 Rebalance progress: 12.5
2021/12/04 02:42:22 Rebalance progress: 12.5
2021/12/04 02:42:27 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:42:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:42:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 02:42:27 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: kv to the cluster
2021/12/04 02:42:27 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/04 02:42:32 Rebalance progress: 12.5
2021/12/04 02:42:37 Rebalance progress: 12.5
2021/12/04 02:42:42 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:42:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:42:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 02:42:42 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/04 02:42:47 Rebalance progress: 12.5
2021/12/04 02:42:52 Rebalance progress: 12.5
2021/12/04 02:42:57 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:42:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:42:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 02:42:57 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/04 02:43:03 Rebalance progress: 12.5
2021/12/04 02:43:08 Rebalance progress: 12.5
2021/12/04 02:43:13 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:43:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:43:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 02:43:13 clusterSetup.go:253: Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/04 02:43:18 Rebalance progress: 12.5
2021/12/04 02:43:23 Rebalance progress: 12.5
2021/12/04 02:43:28 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:43:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:43:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 02:43:28 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 02:43:33 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 02:43:38 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 02:43:43 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 02:43:48 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 02:43:53 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 02:43:58 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 02:43:58 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/04 02:43:58 eventingSetup.go:46: No eventing-con process running
2021/12/04 02:43:58 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/04 02:44:00 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:44:05 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 02:44:05 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 02:44:07 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:44:12 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 02:44:17 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 10678
2021/12/04 02:44:17 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebOnUpdateBucketOpOneByOne (190.70s)
=== RUN   TestKVForceFailover
2021/12/04 02:44:32 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 02:44:32 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/04 02:44:32 eventingSetup.go:46: No eventing-con process running
2021/12/04 02:44:32 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/04 02:44:33 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:44:38 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 02:44:38 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 02:44:40 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:44:45 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 02:44:45 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 02:44:45 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/04 02:44:45 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/04 02:44:51 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:44:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 02:44:51 Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/04 02:44:52 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/04 02:44:52 Starting bucket ops to source bucket
2021/12/04 02:44:57 Waiting for app: bucket_op_on_update to get deployed
2021/12/04 02:45:02 Waiting for app: bucket_op_on_update to get deployed
2021/12/04 02:45:07 Waiting for app: bucket_op_on_update to get deployed
2021/12/04 02:45:12 Waiting for app: bucket_op_on_update to get deployed
2021/12/04 02:45:12 App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:45:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:45:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 02:45:12 clusterSetup.go:229: Failing over: [127.0.0.1:19001]
2021/12/04 02:45:32 clusterSetup.go:237: Kicking off failover recovery, type: full
2021/12/04 02:45:32 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/04 02:45:37 Rebalance progress: 12.5
2021/12/04 02:45:42 Rebalance progress: 12.5
2021/12/04 02:45:47 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:45:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:45:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 02:45:47 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002] from the cluster
2021/12/04 02:45:52 Rebalance progress: 12.5
2021/12/04 02:45:57 Rebalance progress: 12.5
2021/12/04 02:46:02 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:46:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:46:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 02:46:03 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 02:46:08 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 02:46:13 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 02:46:18 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 02:46:23 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 02:46:28 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 02:46:28 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/04 02:46:28 eventingSetup.go:46: No eventing-con process running
2021/12/04 02:46:28 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/04 02:46:30 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:46:35 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 02:46:35 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 02:46:36 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:46:41 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 02:46:46 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7064
2021/12/04 02:46:46 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVForceFailover (149.81s)
=== RUN   TestKVHardFailoverRecovery
2021/12/04 02:46:56 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 02:46:56 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/04 02:46:56 eventingSetup.go:46: No eventing-con process running
2021/12/04 02:46:56 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/04 02:46:58 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:47:03 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 02:47:03 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 02:47:05 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:47:10 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 02:47:10 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:47:15 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 02:47:15 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/04 02:47:16 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/04 02:47:21 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/04 02:47:26 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/04 02:47:31 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/04 02:47:36 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/04 02:47:36 clusterSetup.go:354: App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:47:36 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:47:36 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 02:47:36 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/04 02:47:36 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/04 02:47:41 Rebalance progress: 12.5
2021/12/04 02:47:46 Rebalance progress: 12.5
2021/12/04 02:47:51 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:47:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:47:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 02:47:51 clusterSetup.go:229: Failing over: [127.0.0.1:19001]
2021/12/04 02:48:51 clusterSetup.go:237: Kicking off failover recovery, type: full
2021/12/04 02:48:57 Rebalance progress: 12.5
2021/12/04 02:49:02 Rebalance progress: 12.5
2021/12/04 02:49:07 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:49:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:49:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 02:49:07 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/04 02:49:12 Rebalance progress: 12.5
2021/12/04 02:49:17 Rebalance progress: 12.5
2021/12/04 02:49:22 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:49:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:49:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 02:49:22 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 02:49:27 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 02:49:32 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 02:49:37 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 02:49:42 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 02:49:47 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 02:49:52 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 02:49:52 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/04 02:49:52 eventingSetup.go:46: No eventing-con process running
2021/12/04 02:49:52 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/04 02:49:54 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:49:59 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 02:50:04 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12039
2021/12/04 02:50:09 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12039
2021/12/04 02:50:09 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 02:50:11 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:50:16 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 02:50:21 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 9417
2021/12/04 02:50:26 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 9417
2021/12/04 02:50:26 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVHardFailoverRecovery (219.35s)
=== RUN   TestKVRebOnUpdateBucketOpAllAtOnce
2021/12/04 02:50:36 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 02:50:36 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/04 02:50:36 eventingSetup.go:46: No eventing-con process running
2021/12/04 02:50:36 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/04 02:50:38 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:50:43 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 02:50:43 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 02:50:44 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:50:49 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 02:50:49 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:50:54 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 02:50:54 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/04 02:50:56 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/04 02:51:01 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/04 02:51:06 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/04 02:51:11 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/04 02:51:16 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/04 02:51:16 clusterSetup.go:354: App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:51:16 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:51:16 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 02:51:16 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/04 02:51:16 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/04 02:51:16 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: kv to the cluster
2021/12/04 02:51:17 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/04 02:51:22 Rebalance progress: 12.5
2021/12/04 02:51:27 Rebalance progress: 12.5
2021/12/04 02:51:32 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:51:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:51:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 02:51:32 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002 127.0.0.1:19003] from the cluster
2021/12/04 02:51:37 Rebalance progress: 12.5
2021/12/04 02:51:42 Rebalance progress: 12.5
2021/12/04 02:51:47 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:51:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:51:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 02:51:47 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 02:51:52 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 02:51:57 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 02:52:02 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 02:52:07 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 02:52:12 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 02:52:17 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 02:52:17 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/04 02:52:17 eventingSetup.go:46: No eventing-con process running
2021/12/04 02:52:17 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/04 02:52:19 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:52:24 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 02:52:29 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4644
2021/12/04 02:52:29 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 02:52:31 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:52:36 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 02:52:41 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4644
2021/12/04 02:52:46 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4644
2021/12/04 02:52:46 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebOnUpdateBucketOpAllAtOnce (140.23s)
=== RUN   TestKVSwapRebOnUpdateBucketOp
2021/12/04 02:52:56 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 02:52:56 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/04 02:52:56 eventingSetup.go:46: No eventing-con process running
2021/12/04 02:52:56 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/04 02:52:58 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:53:03 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 02:53:03 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 02:53:05 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:53:10 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 02:53:10 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:53:10 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 02:53:10 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/04 02:53:11 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/04 02:53:16 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/04 02:53:21 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/04 02:53:26 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/04 02:53:31 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/04 02:53:31 clusterSetup.go:354: App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:53:31 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:53:31 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 02:53:31 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/04 02:53:31 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/04 02:53:36 Rebalance progress: 12.5
2021/12/04 02:53:41 Rebalance progress: 12.5
2021/12/04 02:53:46 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:53:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:53:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 02:53:46 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/04 02:53:46 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/04 02:53:51 Rebalance progress: 12.5
2021/12/04 02:53:56 Rebalance progress: 12.5
2021/12/04 02:54:01 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:54:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:54:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 02:54:01 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/04 02:54:07 Rebalance progress: 12.5
2021/12/04 02:54:12 Rebalance progress: 12.5
2021/12/04 02:54:17 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:54:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:54:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 02:54:17 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 02:54:22 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 02:54:27 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 02:54:32 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 02:54:37 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 02:54:42 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 02:54:47 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 02:54:47 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/04 02:54:47 eventingSetup.go:46: No eventing-con process running
2021/12/04 02:54:47 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/04 02:54:49 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:54:54 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 02:54:59 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6073
2021/12/04 02:54:59 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 02:55:00 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:55:05 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 02:55:11 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3665
2021/12/04 02:55:16 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3665
2021/12/04 02:55:16 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 02:55:21 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 02:55:21 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/04 02:55:21 eventingSetup.go:46: No eventing-con process running
2021/12/04 02:55:21 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/04 02:55:22 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:55:27 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 02:55:27 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 02:55:29 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:55:34 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 02:55:34 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVSwapRebOnUpdateBucketOp (168.08s)
=== RUN   TestKVRebalanceWithMultipleHandlers
2021/12/04 02:55:44 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 02:55:44 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/04 02:55:44 eventingSetup.go:46: No eventing-con process running
2021/12/04 02:55:44 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/04 02:55:46 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:55:51 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 02:55:51 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 02:55:53 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:55:58 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 02:55:58 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 02:56:03 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/04 02:56:03 clusterSetup.go:405: App: sys_test_bucket_op got un-deployed
2021/12/04 02:56:03 eventingSetup.go:46: No eventing-con process running
2021/12/04 02:56:03 eventingSetup.go:418: Function: sys_test_bucket_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: sys_test_bucket_op not found"
 }
}
2021/12/04 02:56:04 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:56:09 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 02:56:10 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 02:56:11 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:56:16 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 02:56:16 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:56:16 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 02:56:16 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/04 02:56:17 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:56:17 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 02:56:17 eventingSetup.go:125: Deploying app: sys_test_bucket_op. Item count src bucket: 0 dst bucket: 0
2021/12/04 02:56:19 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'sys_test_bucket_op' in metakv",
  "warnings": null
 }
}
2021/12/04 02:56:24 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/04 02:56:29 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/04 02:56:34 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/04 02:56:39 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/04 02:56:39 clusterSetup.go:354: App: bucket_op_on_update got deployed
2021/12/04 02:56:44 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/04 02:56:49 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/04 02:56:54 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/04 02:56:54 clusterSetup.go:354: App: sys_test_bucket_op got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:56:54 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:56:54 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 02:56:54 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/04 02:56:54 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/04 02:56:59 Rebalance progress: 12.5
2021/12/04 02:57:04 Rebalance progress: 12.5
2021/12/04 02:57:09 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:57:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:57:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 02:57:09 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/04 02:57:10 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/04 02:57:15 Rebalance progress: 12.5
2021/12/04 02:57:20 Rebalance progress: 12.5
2021/12/04 02:57:25 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:57:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:57:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 02:57:25 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: kv to the cluster
2021/12/04 02:57:25 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/04 02:57:30 Rebalance progress: 12.5
2021/12/04 02:57:35 Rebalance progress: 12.5
2021/12/04 02:57:40 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:57:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:57:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 02:57:40 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/04 02:57:45 Rebalance progress: 12.5
2021/12/04 02:57:50 Rebalance progress: 12.5
2021/12/04 02:57:55 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:57:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:57:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 02:57:55 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/04 02:58:00 Rebalance progress: 12.5
2021/12/04 02:58:05 Rebalance progress: 12.5
2021/12/04 02:58:10 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:58:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:58:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 02:58:10 clusterSetup.go:253: Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/04 02:58:16 Rebalance progress: 12.5
2021/12/04 02:58:21 Rebalance progress: 12.5
2021/12/04 02:58:26 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:58:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:58:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 02:58:26 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 02:58:31 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 02:58:36 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 02:58:41 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 02:58:46 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 02:58:51 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 02:58:56 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 02:58:56 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/04 02:58:56 eventingSetup.go:46: No eventing-con process running
2021/12/04 02:58:56 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/04 02:58:59 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:59:04 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 02:59:09 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12184
2021/12/04 02:59:09 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 02:59:11 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 02:59:16 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 02:59:21 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 10021
2021/12/04 02:59:26 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 10021
2021/12/04 02:59:26 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 02:59:26 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 02:59:31 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/04 02:59:36 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/04 02:59:41 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/04 02:59:46 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/04 02:59:51 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/04 02:59:56 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/04 02:59:56 clusterSetup.go:405: App: sys_test_bucket_op got un-deployed
2021/12/04 02:59:56 eventingSetup.go:46: No eventing-con process running
2021/12/04 02:59:56 eventingSetup.go:418: Function: sys_test_bucket_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: sys_test_bucket_op deleting in the background"
}
2021/12/04 02:59:58 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 03:00:03 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 03:00:03 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 03:00:05 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 03:00:10 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 03:00:10 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebalanceWithMultipleHandlers (275.64s)
=== RUN   TestKVSwapRebalanceWithMultipleHandlers
2021/12/04 03:00:20 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 03:00:20 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/04 03:00:20 eventingSetup.go:46: No eventing-con process running
2021/12/04 03:00:20 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/04 03:00:22 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 03:00:27 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 03:00:27 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 03:00:28 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 03:00:33 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 03:00:33 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 03:00:38 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/04 03:00:38 clusterSetup.go:405: App: sys_test_bucket_op got un-deployed
2021/12/04 03:00:38 eventingSetup.go:46: No eventing-con process running
2021/12/04 03:00:38 eventingSetup.go:418: Function: sys_test_bucket_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: sys_test_bucket_op not found"
 }
}
2021/12/04 03:00:40 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 03:00:45 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 03:00:45 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 03:00:47 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 03:00:52 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 03:00:52 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 03:00:52 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 03:00:52 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/04 03:00:53 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 03:00:53 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 03:00:53 eventingSetup.go:125: Deploying app: sys_test_bucket_op. Item count src bucket: 0 dst bucket: 0
2021/12/04 03:00:54 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'sys_test_bucket_op' in metakv",
  "warnings": null
 }
}
2021/12/04 03:00:59 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/04 03:01:04 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/04 03:01:09 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/04 03:01:14 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/04 03:01:14 clusterSetup.go:354: App: bucket_op_on_update got deployed
2021/12/04 03:01:19 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/04 03:01:24 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/04 03:01:29 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/04 03:01:29 clusterSetup.go:354: App: sys_test_bucket_op got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 03:01:29 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 03:01:29 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 03:01:29 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/04 03:01:30 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/04 03:01:35 Rebalance progress: 12.5
2021/12/04 03:01:40 Rebalance progress: 12.5
2021/12/04 03:01:45 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 03:01:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 03:01:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 03:01:45 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/04 03:01:45 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/04 03:01:50 Rebalance progress: 12.5
2021/12/04 03:01:55 Rebalance progress: 12.5
2021/12/04 03:02:00 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 03:02:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 03:02:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 03:02:00 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/04 03:02:05 Rebalance progress: 12.5
2021/12/04 03:02:10 Rebalance progress: 12.5
2021/12/04 03:02:15 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 03:02:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 03:02:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 03:02:16 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 03:02:21 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 03:02:26 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 03:02:31 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 03:02:36 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 03:02:41 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 03:02:46 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 03:02:46 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/04 03:02:46 eventingSetup.go:46: No eventing-con process running
2021/12/04 03:02:46 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/04 03:02:49 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 03:02:54 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 03:02:54 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 03:02:56 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 03:03:01 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 03:03:06 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7612
2021/12/04 03:03:06 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 03:03:06 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 03:03:11 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/04 03:03:16 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/04 03:03:21 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/04 03:03:26 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/04 03:03:32 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/04 03:03:37 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/04 03:03:37 clusterSetup.go:405: App: sys_test_bucket_op got un-deployed
2021/12/04 03:03:37 eventingSetup.go:46: No eventing-con process running
2021/12/04 03:03:37 eventingSetup.go:418: Function: sys_test_bucket_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: sys_test_bucket_op deleting in the background"
}
2021/12/04 03:03:38 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 03:03:43 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 03:03:43 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 03:03:45 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 03:03:50 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 03:03:50 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVSwapRebalanceWithMultipleHandlers (220.44s)
=== RUN   TestKVRebStopStartKVOpsOnUpdateBucketOpOneByOne
2021/12/04 03:04:00 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 03:04:00 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/04 03:04:00 eventingSetup.go:46: No eventing-con process running
2021/12/04 03:04:00 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/04 03:04:02 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 03:04:07 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 03:04:07 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 03:04:09 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 03:04:14 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 03:04:14 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 03:04:19 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 03:04:19 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/04 03:04:20 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/04 03:04:25 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/04 03:04:30 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/04 03:04:35 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/04 03:04:40 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/04 03:04:40 clusterSetup.go:354: App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 03:04:40 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 03:04:40 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 03:04:40 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/04 03:04:40 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/04 03:05:00 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 03:05:01 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 03:05:01 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 03:05:11 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 03:05:11 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 03:05:11 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/04 03:05:16 Rebalance progress: 12.5
2021/12/04 03:05:21 Rebalance progress: 12.5
2021/12/04 03:05:26 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 03:05:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 03:05:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 03:05:26 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/04 03:05:26 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/04 03:05:31 Rebalance progress: 12.5
2021/12/04 03:05:36 Rebalance progress: 12.5
2021/12/04 03:05:41 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 03:05:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 03:05:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 03:05:41 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/04 03:05:46 Rebalance progress: 12.5
2021/12/04 03:05:51 Rebalance progress: 12.5
2021/12/04 03:05:56 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 03:05:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 03:05:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 03:05:57 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 03:06:02 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 03:06:07 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 03:06:12 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 03:06:17 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 03:06:22 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 03:06:27 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 03:06:27 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/04 03:06:27 eventingSetup.go:46: No eventing-con process running
2021/12/04 03:06:27 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/04 03:06:29 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 03:06:34 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 03:06:39 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 9143
2021/12/04 03:06:39 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 03:06:40 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 03:06:45 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 03:06:50 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3897
2021/12/04 03:06:55 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3897
2021/12/04 03:06:55 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebStopStartKVOpsOnUpdateBucketOpOneByOne (185.17s)
=== RUN   TestKVFailoverOnUpdateBucketOp
2021/12/04 03:07:05 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 03:07:05 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/04 03:07:05 eventingSetup.go:46: No eventing-con process running
2021/12/04 03:07:05 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/04 03:07:07 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 03:07:12 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 03:07:12 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 03:07:14 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 03:07:19 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 03:07:19 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 03:07:24 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 03:07:24 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/04 03:07:25 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/04 03:07:30 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/04 03:07:35 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/04 03:07:40 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/04 03:07:45 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/04 03:07:45 clusterSetup.go:354: App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 03:07:45 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 03:07:45 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 03:07:45 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/04 03:07:45 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/04 03:07:50 Rebalance progress: 12.5
2021/12/04 03:07:55 Rebalance progress: 12.5
2021/12/04 03:08:00 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 03:08:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 03:08:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 03:08:00 clusterSetup.go:229: Failing over: [127.0.0.1:19001]
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 03:08:11 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 03:08:11 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 03:08:11 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/04 03:08:16 Rebalance progress: 12.5
2021/12/04 03:08:21 Rebalance progress: 12.5
2021/12/04 03:08:26 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 03:08:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 03:08:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 03:08:26 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/04 03:08:26 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/04 03:08:31 Rebalance progress: 12.5
2021/12/04 03:08:36 Rebalance progress: 12.5
2021/12/04 03:08:41 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 03:08:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 03:08:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 03:08:41 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/04 03:08:46 Rebalance progress: 12.5
2021/12/04 03:08:51 Rebalance progress: 12.5
2021/12/04 03:08:56 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 03:08:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 03:08:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 03:08:56 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 03:09:01 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 03:09:06 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 03:09:11 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 03:09:17 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 03:09:22 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 03:09:22 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/04 03:09:22 eventingSetup.go:46: No eventing-con process running
2021/12/04 03:09:22 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/04 03:09:23 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 03:09:28 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 03:09:33 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 8608
2021/12/04 03:09:38 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 8608
2021/12/04 03:09:38 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 03:09:40 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 03:09:45 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 03:09:50 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2685
2021/12/04 03:09:55 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2685
2021/12/04 03:09:55 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVFailoverOnUpdateBucketOp (179.86s)
=== RUN   TestBootstrapAfterKVHardFailover
2021/12/04 03:10:00 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/04 03:10:01 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/04 03:10:06 Rebalance progress: 100
2021/12/04 03:10:06 Failing over: [127.0.0.1:19001]
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 03:10:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 03:10:36 Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/04 03:10:37 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/04 03:10:42 Removing node(s): [] from the cluster
2021/12/04 03:10:42 Starting bucket ops to source bucket
2021/12/04 03:10:47 Rebalance failed. See logs for detailed reason. You can try again.
2021/12/04 03:10:52 Waiting for app: bucket_op_on_update to get deployed
2021/12/04 03:10:57 Waiting for app: bucket_op_on_update to get deployed
2021/12/04 03:10:57 App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 03:10:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 03:10:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 03:10:57 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 03:11:02 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 03:11:07 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 03:11:12 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 03:11:17 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 03:11:22 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 03:11:27 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 03:11:27 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/04 03:11:27 eventingSetup.go:46: No eventing-con process running
2021/12/04 03:11:28 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/04 03:11:29 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 03:11:34 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 03:11:39 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1506
2021/12/04 03:11:39 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 03:11:41 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 03:11:46 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 03:11:51 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1506
2021/12/04 03:11:56 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1506
2021/12/04 03:11:56 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestBootstrapAfterKVHardFailover (120.89s)
PASS
ok  	github.com/couchbase/eventing/tests/functional_tests	12184.022s

Rebalance tests took 3 hours, 23 minutes and 6 seconds.

Testing Succeeded


Version: versions-03.12.2021-19.33.cfg
Build Log: make-03.12.2021-19.33.log
Server Log: logs-03.12.2021-19.33.tar.gz

Finished