Building

Started building at 2021/12/01 19:27:10
Calculating base
Updating mirror
Basing run on 7.1.0-1817 50858fc74
Updating tree for run 01.12.2021-19.27
gocb is at 635fa6c, changes since last good build: none
libcouchbase is at f6822fc, changes since last good build: none
indexing is at a00b35f, changes since last good build: none
kv_engine is at d83d53e, changes since last good build: 
 d83d53e65 MB-47318: Address double acking when closing a stream
 d9406ce11 MB-49782: [2/2] Replace ScopeTimer{1,2} usage with ScopeTimer
 7f9e8ff01 ActiveStream: Simplify backfill completion functions
 98c166018 [Nexus] Flush the logger before aborting the process
 74e96cd66 MB-49782: [1/2] Avoid unnecessary move construction of ScopeTimer1
ns_server is at 9756174, changes since last good build: none
couchstore is at 05c4320, changes since last good build: none
couchdb is at fca2e6b, changes since last good build: none
testrunner is at 6061aba, changes since last good build: none
Switching eventing to unstable
eventing is at e4e62e2, changes since last good build: none
Switching eventing-ee to unstable
eventing-ee is at 7060ef8, changes since last good build: none
Pulling in uncommitted change 158037 at refs/changes/37/158037/4
Total 7 (delta 5), reused 7 (delta 5)
[unstable a832e9f2] MB-47385: Move eventing-producer and cbevent go version to 1.16.5
 Author: AnkitPrabhu 
 Date: Thu Jul 22 11:08:10 2021 +0530
 3 files changed, 23 insertions(+), 4 deletions(-)
Pulling in uncommitted change 160397 at refs/changes/97/160397/3
Total 23 (delta 15), reused 17 (delta 15)
[unstable 0be59c81] MB-100 : Do not fail n1ql timeout test if we are catching timeout exceptions successfully
 Author: abhijpes 
 Date: Mon Aug 30 08:24:38 2021 +0530
 1 file changed, 2 insertions(+), 2 deletions(-)
Pulling in uncommitted change 163449 at refs/changes/49/163449/1
Total 48 (delta 27), reused 39 (delta 27)
[unstable b418129d] MB-100: collect data when test fails
 Author: AnkitPrabhu 
 Date: Tue Oct 12 14:02:31 2021 +0530
 1 file changed, 2 insertions(+)
Pulling in uncommitted change 165976 at refs/changes/76/165976/16
Total 5 (delta 3), reused 5 (delta 3)
[unstable 00a61841] MB-48538: Add new audit ids for different events
 Author: AnkitPrabhu 
 Date: Tue Nov 23 14:29:10 2021 +0530
 1 file changed, 337 insertions(+), 61 deletions(-)
Pulling in uncommitted change 166486 at refs/changes/86/166486/3
Total 7 (delta 2), reused 2 (delta 2)
[unstable 09612721] MB-49148: Grant SystemXattrWrite privilege to on_behalf_of_ user
 Author: AnkitPrabhu 
 Date: Sat Nov 27 15:22:26 2021 +0530
 2 files changed, 31 insertions(+), 13 deletions(-)
Pulling in uncommitted change 166698 at refs/changes/98/166698/3
Total 4 (delta 3), reused 4 (delta 3)
[unstable 617964b9] MB-48334 : Fail rebalance unless all functions deployed on current encryption level
 Author: abhijpes 
 Date: Wed Dec 1 14:51:29 2021 +0530
 1 file changed, 9 insertions(+)
Building cmakefiles and deps
Building main product

Testing

Started testing at 2021/12/01 19:42:35
Testing mode: rebalance,n1ql,functional
Starting server
Starting server: attempt 1

Functional tests

2021/12/01 19:44:20 Started functional
Post "http://127.0.0.1:9000/pools/default": EOF
Quota setup Post "http://127.0.0.1:9000/pools/default": EOF
2021/12/01 19:44:33 Adding node: https://127.0.0.1:19001 with role: kv,index,n1ql to the cluster
2021/12/01 19:44:46 Error in adding nodes : , response: {"otpNode":"n_1@127.0.0.1"}
2021/12/01 19:44:46 Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/01 19:44:51 Removing node(s): [] from the cluster
2021/12/01 19:44:56 Rebalance progress: 2.640335648148148
2021/12/01 19:45:01 Rebalance progress: 7.396556712962963
2021/12/01 19:45:06 Rebalance progress: 10.6517650462963
2021/12/01 19:45:11 Rebalance progress: 15.29947916666667
2021/12/01 19:45:17 Rebalance progress: 18.69936342592592
2021/12/01 19:45:21 Rebalance progress: 23.14814814814815
2021/12/01 19:45:26 Rebalance progress: 27.77777777777778
2021/12/01 19:45:31 Rebalance progress: 100
=== RUN   TestAdvancedGetOps
2021/12/01 19:45:37 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/01 19:45:52 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/01 19:46:07 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/01 19:46:22 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/01 19:46:37 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/01 19:46:52 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/01 19:47:07 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/01 19:47:22 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/01 19:47:37 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/01 19:47:52 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/01 19:48:07 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/01 19:48:22 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/01 19:48:37 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/01 19:48:52 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/01 19:49:07 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/01 19:49:22 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/01 19:49:37 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/01 19:49:52 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/01 19:50:07 Deploying app: TestAdvancedGetOps. Item count src bucket: 0 dst bucket: 0
2021/12/01 19:50:08 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedGetOps' in metakv",
  "warnings": null
 }
}
2021/12/01 19:50:13 Waiting for app: TestAdvancedGetOps to get deployed
2021/12/01 19:50:18 Waiting for app: TestAdvancedGetOps to get deployed
2021/12/01 19:50:23 Waiting for app: TestAdvancedGetOps to get deployed
2021/12/01 19:50:28 Waiting for app: TestAdvancedGetOps to get deployed
2021/12/01 19:50:28 App: TestAdvancedGetOps got deployed
2021/12/01 19:50:28 Starting bucket ops to source bucket
2021/12/01 19:50:34 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/01 19:50:39 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/01 19:50:44 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/01 19:50:44 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/01 19:50:44 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 19:50:44 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/01 19:50:44 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 19:50:49 Waiting for app: TestAdvancedGetOps to get un-deployed
2021/12/01 19:50:54 Waiting for app: TestAdvancedGetOps to get un-deployed
2021/12/01 19:50:59 Waiting for app: TestAdvancedGetOps to get un-deployed
2021/12/01 19:51:04 Waiting for app: TestAdvancedGetOps to get un-deployed
2021/12/01 19:51:04 App: TestAdvancedGetOps got un-deployed
2021/12/01 19:51:04 No eventing-con process running
2021/12/01 19:51:04 Function: TestAdvancedGetOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedGetOps deleting in the background"
}
2021/12/01 19:51:06 Context: Bucket flush, response code: 200 dump: 
2021/12/01 19:51:16 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/01 19:51:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 19:51:17 Context: Bucket flush, response code: 200 dump: 
2021/12/01 19:51:27 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/01 19:51:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedGetOps (350.28s)
=== RUN   TestAdvancedInsertOps
2021/12/01 19:51:27 Deploying app: TestAdvancedInsertOps. Item count src bucket: 0 dst bucket: 0
2021/12/01 19:51:28 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedInsertOps' in metakv",
  "warnings": null
 }
}
2021/12/01 19:51:33 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/01 19:51:39 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/01 19:51:44 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/01 19:51:49 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/01 19:51:49 App: TestAdvancedInsertOps got deployed
2021/12/01 19:51:49 Starting bucket ops to source bucket
2021/12/01 19:51:54 Waiting for dst bucket item count to get to: 1024 curr count: 0
2021/12/01 19:51:59 Waiting for dst bucket item count to get to: 1024 curr count: 417
2021/12/01 19:52:04 Waiting for dst bucket item count to get to: 1024 curr count: 929
2021/12/01 19:52:04 src & dst bucket item count matched up. src bucket count: 1024 dst bucket count: 1024
2021/12/01 19:52:05 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 19:52:05 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/01 19:52:05 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 19:52:10 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/01 19:52:15 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/01 19:52:20 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/01 19:52:25 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/01 19:52:25 App: TestAdvancedInsertOps got un-deployed
2021/12/01 19:52:25 No eventing-con process running
2021/12/01 19:52:25 Function: TestAdvancedInsertOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedInsertOps deleting in the background"
}
2021/12/01 19:52:26 Context: Bucket flush, response code: 200 dump: 
2021/12/01 19:52:36 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/01 19:52:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 19:52:38 Context: Bucket flush, response code: 200 dump: 
2021/12/01 19:52:48 Waiting for dst bucket item count to get to: 0 curr count: 789
2021/12/01 19:52:53 Waiting for dst bucket item count to get to: 0 curr count: 277
2021/12/01 19:52:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 19:52:53 Testing insert operation on source bucket
2021/12/01 19:52:53 Deploying app: TestAdvancedInsertOps. Item count src bucket: 0 dst bucket: 0
2021/12/01 19:52:54 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedInsertOps' in metakv",
  "warnings": null
 }
}
2021/12/01 19:52:59 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/01 19:53:04 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/01 19:53:09 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/01 19:53:14 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/01 19:53:14 App: TestAdvancedInsertOps got deployed
2021/12/01 19:53:14 Starting bucket ops to source bucket
2021/12/01 19:53:20 Waiting for dst bucket item count to get to: 2048 curr count: 0
2021/12/01 19:53:25 Waiting for dst bucket item count to get to: 2048 curr count: 1024
2021/12/01 19:53:25 src & dst bucket item count matched up. src bucket count: 2048 dst bucket count: 2048
2021/12/01 19:53:25 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 19:53:25 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/01 19:53:25 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 19:53:30 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/01 19:53:35 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/01 19:53:40 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/01 19:53:46 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/01 19:53:46 App: TestAdvancedInsertOps got un-deployed
2021/12/01 19:53:46 No eventing-con process running
2021/12/01 19:53:46 Function: TestAdvancedInsertOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedInsertOps deleting in the background"
}
2021/12/01 19:53:47 Context: Bucket flush, response code: 200 dump: 
2021/12/01 19:53:57 Waiting for dst bucket item count to get to: 0 curr count: 2048
2021/12/01 19:53:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 19:53:59 Context: Bucket flush, response code: 200 dump: 
2021/12/01 19:54:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedInsertOps (156.66s)
=== RUN   TestAdvancedUpsertOps
2021/12/01 19:54:04 Starting bucket ops to source bucket
2021/12/01 19:54:05 Deploying app: TestAdvancedUpsertOps. Item count src bucket: 0 dst bucket: 0
2021/12/01 19:54:06 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedUpsertOps' in metakv",
  "warnings": null
 }
}
2021/12/01 19:54:11 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/01 19:54:16 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/01 19:54:21 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/01 19:54:26 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/01 19:54:26 App: TestAdvancedUpsertOps got deployed
2021/12/01 19:54:26 Starting bucket ops to source bucket
2021/12/01 19:54:32 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/01 19:54:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 19:54:32 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 19:54:32 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/01 19:54:32 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 19:54:37 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/01 19:54:42 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/01 19:54:47 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/01 19:54:52 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/01 19:54:52 App: TestAdvancedUpsertOps got un-deployed
2021/12/01 19:54:52 No eventing-con process running
2021/12/01 19:54:52 Function: TestAdvancedUpsertOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedUpsertOps deleting in the background"
}
2021/12/01 19:54:54 Context: Bucket flush, response code: 200 dump: 
2021/12/01 19:55:04 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/01 19:55:09 Waiting for dst bucket item count to get to: 0 curr count: 512
2021/12/01 19:55:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 19:55:11 Context: Bucket flush, response code: 200 dump: 
2021/12/01 19:55:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 19:55:16 Testing upsert operation on source bucket
2021/12/01 19:55:16 Deploying app: TestAdvancedUpsertOps. Item count src bucket: 0 dst bucket: 0
2021/12/01 19:55:17 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedUpsertOps' in metakv",
  "warnings": null
 }
}
2021/12/01 19:55:22 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/01 19:55:27 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/01 19:55:32 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/01 19:55:37 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/01 19:55:37 App: TestAdvancedUpsertOps got deployed
2021/12/01 19:55:37 Starting bucket ops to source bucket
2021/12/01 19:55:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 19:55:38 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 19:55:38 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/01 19:55:38 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 19:55:43 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/01 19:55:48 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/01 19:55:53 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/01 19:55:58 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/01 19:55:58 App: TestAdvancedUpsertOps got un-deployed
2021/12/01 19:55:58 No eventing-con process running
2021/12/01 19:55:58 Function: TestAdvancedUpsertOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedUpsertOps deleting in the background"
}
2021/12/01 19:56:00 Context: Bucket flush, response code: 200 dump: 
2021/12/01 19:56:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 19:56:07 Context: Bucket flush, response code: 200 dump: 
2021/12/01 19:56:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedUpsertOps (128.03s)
=== RUN   TestAdvancedReplaceOps
2021/12/01 19:56:12 Starting bucket ops to source bucket
2021/12/01 19:56:13 Deploying app: TestAdvancedReplaceOps. Item count src bucket: 0 dst bucket: 0
2021/12/01 19:56:14 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedReplaceOps' in metakv",
  "warnings": null
 }
}
2021/12/01 19:56:19 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/01 19:56:24 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/01 19:56:29 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/01 19:56:34 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/01 19:56:34 App: TestAdvancedReplaceOps got deployed
2021/12/01 19:56:34 Starting bucket ops to source bucket
2021/12/01 19:56:40 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/01 19:56:45 Waiting for dst bucket item count to get to: 0 curr count: 512
2021/12/01 19:56:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 19:56:45 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 19:56:45 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/01 19:56:45 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 19:56:50 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/01 19:56:55 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/01 19:57:00 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/01 19:57:05 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/01 19:57:05 App: TestAdvancedReplaceOps got un-deployed
2021/12/01 19:57:05 No eventing-con process running
2021/12/01 19:57:05 Function: TestAdvancedReplaceOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedReplaceOps deleting in the background"
}
2021/12/01 19:57:07 Context: Bucket flush, response code: 200 dump: 
2021/12/01 19:57:17 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/01 19:57:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 19:57:19 Context: Bucket flush, response code: 200 dump: 
2021/12/01 19:57:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 19:57:24 Testing upsert operation on source bucket
2021/12/01 19:57:24 Deploying app: TestAdvancedReplaceOps. Item count src bucket: 0 dst bucket: 0
2021/12/01 19:57:25 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedReplaceOps' in metakv",
  "warnings": null
 }
}
2021/12/01 19:57:30 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/01 19:57:35 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/01 19:57:40 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/01 19:57:45 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/01 19:57:45 App: TestAdvancedReplaceOps got deployed
2021/12/01 19:57:45 Starting bucket ops to source bucket
2021/12/01 19:57:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 19:57:46 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 19:57:46 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/01 19:57:46 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 19:57:51 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/01 19:57:56 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/01 19:58:01 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/01 19:58:06 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/01 19:58:06 App: TestAdvancedReplaceOps got un-deployed
2021/12/01 19:58:06 No eventing-con process running
2021/12/01 19:58:07 Function: TestAdvancedReplaceOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedReplaceOps deleting in the background"
}
2021/12/01 19:58:08 Context: Bucket flush, response code: 200 dump: 
2021/12/01 19:58:18 Waiting for dst bucket item count to get to: 0 curr count: 92
2021/12/01 19:58:23 Waiting for dst bucket item count to get to: 0 curr count: 49
2021/12/01 19:58:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 19:58:25 Context: Bucket flush, response code: 200 dump: 
2021/12/01 19:58:30 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedReplaceOps (137.98s)
=== RUN   TestAdvancedDeleteOps
2021/12/01 19:58:30 Starting bucket ops to source bucket
2021/12/01 19:58:30 Deploying app: TestAdvancedDeleteOps. Item count src bucket: 0 dst bucket: 0
2021/12/01 19:58:31 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedDeleteOps' in metakv",
  "warnings": null
 }
}
2021/12/01 19:58:36 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/01 19:58:41 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/01 19:58:46 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/01 19:58:51 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/01 19:58:51 App: TestAdvancedDeleteOps got deployed
2021/12/01 19:58:51 Starting bucket ops to source bucket
2021/12/01 19:58:56 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/01 19:59:01 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/01 19:59:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 19:59:01 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 19:59:01 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/01 19:59:02 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 19:59:07 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/01 19:59:12 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/01 19:59:17 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/01 19:59:22 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/01 19:59:22 App: TestAdvancedDeleteOps got un-deployed
2021/12/01 19:59:22 No eventing-con process running
2021/12/01 19:59:22 Function: TestAdvancedDeleteOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedDeleteOps deleting in the background"
}
2021/12/01 19:59:23 Context: Bucket flush, response code: 200 dump: 
2021/12/01 19:59:33 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/01 19:59:38 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/01 19:59:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 19:59:40 Context: Bucket flush, response code: 200 dump: 
2021/12/01 19:59:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 19:59:45 Testing delete operation on source bucket
2021/12/01 19:59:45 Deploying app: TestAdvancedDeleteOps. Item count src bucket: 0 dst bucket: 0
2021/12/01 19:59:46 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedDeleteOps' in metakv",
  "warnings": null
 }
}
2021/12/01 19:59:51 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/01 19:59:56 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/01 20:00:01 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/01 20:00:06 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/01 20:00:06 App: TestAdvancedDeleteOps got deployed
2021/12/01 20:00:06 Starting bucket ops to source bucket
2021/12/01 20:00:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 20:00:07 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 20:00:07 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/01 20:00:07 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 20:00:12 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/01 20:00:17 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/01 20:00:22 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/01 20:00:27 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/01 20:00:27 App: TestAdvancedDeleteOps got un-deployed
2021/12/01 20:00:27 No eventing-con process running
2021/12/01 20:00:27 Function: TestAdvancedDeleteOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedDeleteOps deleting in the background"
}
2021/12/01 20:00:28 Context: Bucket flush, response code: 200 dump: 
2021/12/01 20:00:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 20:00:36 Context: Bucket flush, response code: 200 dump: 
2021/12/01 20:00:41 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedDeleteOps (130.72s)
=== RUN   TestEnoentAdvancedGet
2021/12/01 20:00:41 Deploying app: TestEnoentAdvancedGet. Item count src bucket: 0 dst bucket: 0
2021/12/01 20:00:42 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedGet' in metakv",
  "warnings": null
 }
}
2021/12/01 20:00:47 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/01 20:00:52 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/01 20:00:57 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/01 20:01:02 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/01 20:01:02 App: TestEnoentAdvancedGet got deployed
2021/12/01 20:01:02 Starting bucket ops to source bucket
2021/12/01 20:01:07 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/01 20:01:12 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/01 20:01:12 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/01 20:01:12 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 20:01:12 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/01 20:01:12 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 20:01:17 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/01 20:01:22 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/01 20:01:27 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/01 20:01:32 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/01 20:01:32 App: TestEnoentAdvancedGet got un-deployed
2021/12/01 20:01:32 No eventing-con process running
2021/12/01 20:01:32 Function: TestEnoentAdvancedGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedGet deleting in the background"
}
2021/12/01 20:01:34 Context: Bucket flush, response code: 200 dump: 
2021/12/01 20:01:44 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/01 20:01:49 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/01 20:01:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 20:01:51 Context: Bucket flush, response code: 200 dump: 
2021/12/01 20:02:01 Waiting for dst bucket item count to get to: 0 curr count: 91
2021/12/01 20:02:06 Waiting for dst bucket item count to get to: 0 curr count: 91
2021/12/01 20:02:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 20:02:06 Testing get enoent operation on source bucket
2021/12/01 20:02:06 Deploying app: TestEnoentAdvancedGet. Item count src bucket: 0 dst bucket: 0
2021/12/01 20:02:07 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedGet' in metakv",
  "warnings": null
 }
}
2021/12/01 20:02:12 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/01 20:02:17 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/01 20:02:22 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/01 20:02:27 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/01 20:02:27 App: TestEnoentAdvancedGet got deployed
2021/12/01 20:02:27 Starting bucket ops to source bucket
2021/12/01 20:02:32 Waiting for dst bucket item count to get to: 200 curr count: 0
2021/12/01 20:02:32 src & dst bucket item count matched up. src bucket count: 200 dst bucket count: 200
2021/12/01 20:02:32 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 20:02:32 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/01 20:02:32 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 20:02:37 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/01 20:02:42 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/01 20:02:47 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/01 20:02:52 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/01 20:02:52 App: TestEnoentAdvancedGet got un-deployed
2021/12/01 20:02:52 No eventing-con process running
2021/12/01 20:02:52 Function: TestEnoentAdvancedGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedGet deleting in the background"
}
2021/12/01 20:02:54 Context: Bucket flush, response code: 200 dump: 
2021/12/01 20:03:04 Waiting for dst bucket item count to get to: 0 curr count: 200
2021/12/01 20:03:09 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/01 20:03:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 20:03:11 Context: Bucket flush, response code: 200 dump: 
2021/12/01 20:03:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEnoentAdvancedGet (155.07s)
=== RUN   TestEnoentAdvancedReplace
2021/12/01 20:03:16 Deploying app: TestEnoentAdvancedReplace. Item count src bucket: 0 dst bucket: 0
2021/12/01 20:03:17 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedReplace' in metakv",
  "warnings": null
 }
}
2021/12/01 20:03:22 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/01 20:03:27 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/01 20:03:32 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/01 20:03:37 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/01 20:03:37 App: TestEnoentAdvancedReplace got deployed
2021/12/01 20:03:37 Starting bucket ops to source bucket
2021/12/01 20:03:42 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/01 20:03:42 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/01 20:03:42 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 20:03:42 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/01 20:03:42 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 20:03:47 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/01 20:03:52 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/01 20:03:57 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/01 20:04:02 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/01 20:04:02 App: TestEnoentAdvancedReplace got un-deployed
2021/12/01 20:04:02 No eventing-con process running
2021/12/01 20:04:02 Function: TestEnoentAdvancedReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedReplace deleting in the background"
}
2021/12/01 20:04:04 Context: Bucket flush, response code: 200 dump: 
2021/12/01 20:04:14 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/01 20:04:19 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/01 20:04:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 20:04:21 Context: Bucket flush, response code: 200 dump: 
2021/12/01 20:04:31 Waiting for dst bucket item count to get to: 0 curr count: 95
2021/12/01 20:04:36 Waiting for dst bucket item count to get to: 0 curr count: 95
2021/12/01 20:04:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 20:04:36 Testing get enoent operation on source bucket
2021/12/01 20:04:36 Deploying app: TestEnoentAdvancedReplace. Item count src bucket: 0 dst bucket: 0
2021/12/01 20:04:37 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedReplace' in metakv",
  "warnings": null
 }
}
2021/12/01 20:04:42 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/01 20:04:47 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/01 20:04:52 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/01 20:04:57 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/01 20:04:57 App: TestEnoentAdvancedReplace got deployed
2021/12/01 20:04:57 Starting bucket ops to source bucket
2021/12/01 20:05:02 Waiting for dst bucket item count to get to: 200 curr count: 0
2021/12/01 20:05:07 Waiting for dst bucket item count to get to: 200 curr count: 147
2021/12/01 20:05:12 Waiting for dst bucket item count to get to: 200 curr count: 147
2021/12/01 20:05:12 src & dst bucket item count matched up. src bucket count: 200 dst bucket count: 200
2021/12/01 20:05:12 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 20:05:12 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/01 20:05:12 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 20:05:17 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/01 20:05:22 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/01 20:05:27 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/01 20:05:32 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/01 20:05:32 App: TestEnoentAdvancedReplace got un-deployed
2021/12/01 20:05:32 No eventing-con process running
2021/12/01 20:05:32 Function: TestEnoentAdvancedReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedReplace deleting in the background"
}
2021/12/01 20:05:34 Context: Bucket flush, response code: 200 dump: 
2021/12/01 20:05:44 Waiting for dst bucket item count to get to: 0 curr count: 200
2021/12/01 20:05:49 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/01 20:05:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 20:05:51 Context: Bucket flush, response code: 200 dump: 
2021/12/01 20:05:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEnoentAdvancedReplace (160.12s)
=== RUN   TestEnoentAdvancedDelete
2021/12/01 20:05:56 Deploying app: TestEnoentAdvancedDelete. Item count src bucket: 0 dst bucket: 0
2021/12/01 20:05:57 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedDelete' in metakv",
  "warnings": null
 }
}
2021/12/01 20:06:02 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/01 20:06:07 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/01 20:06:12 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/01 20:06:17 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/01 20:06:17 App: TestEnoentAdvancedDelete got deployed
2021/12/01 20:06:17 Starting bucket ops to source bucket
2021/12/01 20:06:22 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/01 20:06:27 Waiting for dst bucket item count to get to: 100 curr count: 65
2021/12/01 20:06:32 Waiting for dst bucket item count to get to: 100 curr count: 65
2021/12/01 20:06:32 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/01 20:06:32 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 20:06:32 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/01 20:06:32 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 20:06:37 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/01 20:06:42 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/01 20:06:47 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/01 20:06:52 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/01 20:06:52 App: TestEnoentAdvancedDelete got un-deployed
2021/12/01 20:06:52 No eventing-con process running
2021/12/01 20:06:52 Function: TestEnoentAdvancedDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedDelete deleting in the background"
}
2021/12/01 20:06:54 Context: Bucket flush, response code: 200 dump: 
2021/12/01 20:07:04 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/01 20:07:09 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/01 20:07:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 20:07:11 Context: Bucket flush, response code: 200 dump: 
2021/12/01 20:07:21 Waiting for dst bucket item count to get to: 0 curr count: 97
2021/12/01 20:07:26 Waiting for dst bucket item count to get to: 0 curr count: 97
2021/12/01 20:07:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 20:07:26 Testing get enoent operation on source bucket
2021/12/01 20:07:26 Deploying app: TestEnoentAdvancedDelete. Item count src bucket: 0 dst bucket: 0
2021/12/01 20:07:27 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedDelete' in metakv",
  "warnings": null
 }
}
2021/12/01 20:07:32 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/01 20:07:37 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/01 20:07:42 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/01 20:07:47 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/01 20:07:47 App: TestEnoentAdvancedDelete got deployed
2021/12/01 20:07:47 Starting bucket ops to source bucket
2021/12/01 20:07:52 Waiting for dst bucket item count to get to: 200 curr count: 0
2021/12/01 20:07:57 Waiting for dst bucket item count to get to: 200 curr count: 100
2021/12/01 20:08:02 Waiting for dst bucket item count to get to: 200 curr count: 100
2021/12/01 20:08:02 src & dst bucket item count matched up. src bucket count: 200 dst bucket count: 200
2021/12/01 20:08:02 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 20:08:02 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/01 20:08:02 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 20:08:07 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/01 20:08:12 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/01 20:08:18 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/01 20:08:23 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/01 20:08:23 App: TestEnoentAdvancedDelete got un-deployed
2021/12/01 20:08:23 No eventing-con process running
2021/12/01 20:08:23 Function: TestEnoentAdvancedDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedDelete deleting in the background"
}
2021/12/01 20:08:24 Context: Bucket flush, response code: 200 dump: 
2021/12/01 20:08:34 Waiting for dst bucket item count to get to: 0 curr count: 200
2021/12/01 20:08:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 20:08:36 Context: Bucket flush, response code: 200 dump: 
2021/12/01 20:08:41 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEnoentAdvancedDelete (165.18s)
=== RUN   TestAdvancedInsertKeyExist
2021/12/01 20:08:41 Deploying app: TestAdvancedInsertKeyExist. Item count src bucket: 0 dst bucket: 0
2021/12/01 20:08:42 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedInsertKeyExist' in metakv",
  "warnings": null
 }
}
2021/12/01 20:08:47 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/01 20:08:52 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/01 20:08:57 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/01 20:09:02 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/01 20:09:02 App: TestAdvancedInsertKeyExist got deployed
2021/12/01 20:09:02 Starting bucket ops to source bucket
2021/12/01 20:09:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 20:09:02 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 20:09:03 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/01 20:09:03 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 20:09:08 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/01 20:09:13 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/01 20:09:18 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/01 20:09:23 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/01 20:09:23 App: TestAdvancedInsertKeyExist got un-deployed
2021/12/01 20:09:23 No eventing-con process running
2021/12/01 20:09:23 Function: TestAdvancedInsertKeyExist delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedInsertKeyExist deleting in the background"
}
2021/12/01 20:09:24 Context: Bucket flush, response code: 200 dump: 
2021/12/01 20:09:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 20:09:32 Context: Bucket flush, response code: 200 dump: 
2021/12/01 20:09:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 20:09:37 Pumping document in the destination bucket
2021/12/01 20:09:37 Starting bucket ops to source bucket
2021/12/01 20:09:37 Deploying app: TestAdvancedInsertKeyExist. Item count src bucket: 0 dst bucket: 0
2021/12/01 20:09:38 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedInsertKeyExist' in metakv",
  "warnings": null
 }
}
2021/12/01 20:09:43 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/01 20:09:48 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/01 20:09:53 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/01 20:09:58 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/01 20:09:58 App: TestAdvancedInsertKeyExist got deployed
2021/12/01 20:09:58 Starting bucket ops to source bucket
2021/12/01 20:10:03 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/01 20:10:08 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/01 20:10:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 20:10:08 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 20:10:08 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/01 20:10:08 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 20:10:13 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/01 20:10:18 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/01 20:10:23 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/01 20:10:28 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/01 20:10:28 App: TestAdvancedInsertKeyExist got un-deployed
2021/12/01 20:10:29 No eventing-con process running
2021/12/01 20:10:29 Function: TestAdvancedInsertKeyExist delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedInsertKeyExist deleting in the background"
}
2021/12/01 20:10:30 Context: Bucket flush, response code: 200 dump: 
2021/12/01 20:10:40 Waiting for dst bucket item count to get to: 0 curr count: 72
2021/12/01 20:10:45 Waiting for dst bucket item count to get to: 0 curr count: 72
2021/12/01 20:10:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 20:10:47 Context: Bucket flush, response code: 200 dump: 
2021/12/01 20:10:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedInsertKeyExist (130.93s)
=== RUN   TestExipryGet
2021/12/01 20:10:52 Deploying app: TestExipryGet. Item count src bucket: 0 dst bucket: 0
2021/12/01 20:10:53 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryGet' in metakv",
  "warnings": null
 }
}
2021/12/01 20:10:58 Waiting for app: TestExipryGet to get deployed
2021/12/01 20:11:03 Waiting for app: TestExipryGet to get deployed
2021/12/01 20:11:08 Waiting for app: TestExipryGet to get deployed
2021/12/01 20:11:13 Waiting for app: TestExipryGet to get deployed
2021/12/01 20:11:13 App: TestExipryGet got deployed
2021/12/01 20:11:13 Starting bucket ops to source bucket
2021/12/01 20:11:59 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/01 20:12:04 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/01 20:12:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 20:12:04 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 20:12:04 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/01 20:12:04 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 20:12:09 Waiting for app: TestExipryGet to get un-deployed
2021/12/01 20:12:14 Waiting for app: TestExipryGet to get un-deployed
2021/12/01 20:12:19 Waiting for app: TestExipryGet to get un-deployed
2021/12/01 20:12:24 Waiting for app: TestExipryGet to get un-deployed
2021/12/01 20:12:24 App: TestExipryGet got un-deployed
2021/12/01 20:12:24 No eventing-con process running
2021/12/01 20:12:24 Function: TestExipryGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryGet deleting in the background"
}
2021/12/01 20:12:25 Context: Bucket flush, response code: 200 dump: 
2021/12/01 20:12:30 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 20:12:33 Context: Bucket flush, response code: 200 dump: 
2021/12/01 20:12:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 20:12:38 Pumping document in the destination bucket for expiry get
2021/12/01 20:12:38 Starting bucket ops to source bucket
2021/12/01 20:12:38 Deploying app: TestExipryGet. Item count src bucket: 0 dst bucket: 0
2021/12/01 20:12:39 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryGet' in metakv",
  "warnings": null
 }
}
2021/12/01 20:12:44 Waiting for app: TestExipryGet to get deployed
2021/12/01 20:12:49 Waiting for app: TestExipryGet to get deployed
2021/12/01 20:12:54 Waiting for app: TestExipryGet to get deployed
2021/12/01 20:12:59 Waiting for app: TestExipryGet to get deployed
2021/12/01 20:12:59 App: TestExipryGet got deployed
2021/12/01 20:12:59 Starting bucket ops to source bucket
2021/12/01 20:13:44 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/01 20:13:49 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/01 20:13:54 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/01 20:13:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 20:13:54 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 20:13:54 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/01 20:13:54 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 20:13:59 Waiting for app: TestExipryGet to get un-deployed
2021/12/01 20:14:05 Waiting for app: TestExipryGet to get un-deployed
2021/12/01 20:14:10 Waiting for app: TestExipryGet to get un-deployed
2021/12/01 20:14:15 Waiting for app: TestExipryGet to get un-deployed
2021/12/01 20:14:15 App: TestExipryGet got un-deployed
2021/12/01 20:14:15 No eventing-con process running
2021/12/01 20:14:15 Function: TestExipryGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryGet deleting in the background"
}
2021/12/01 20:14:16 Context: Bucket flush, response code: 200 dump: 
2021/12/01 20:14:26 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/01 20:14:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 20:14:28 Context: Bucket flush, response code: 200 dump: 
2021/12/01 20:14:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestExipryGet (221.04s)
=== RUN   TestExipryInsert
2021/12/01 20:14:33 Deploying app: TestExipryInsert. Item count src bucket: 0 dst bucket: 0
2021/12/01 20:14:34 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryInsert' in metakv",
  "warnings": null
 }
}
2021/12/01 20:14:39 Waiting for app: TestExipryInsert to get deployed
2021/12/01 20:14:44 Waiting for app: TestExipryInsert to get deployed
2021/12/01 20:14:49 Waiting for app: TestExipryInsert to get deployed
2021/12/01 20:14:54 Waiting for app: TestExipryInsert to get deployed
2021/12/01 20:14:54 App: TestExipryInsert got deployed
2021/12/01 20:14:54 Starting bucket ops to source bucket
2021/12/01 20:15:40 Waiting for dst bucket item count to get to: 0 curr count: 200
2021/12/01 20:15:45 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/01 20:15:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 20:15:45 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 20:15:45 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/01 20:15:45 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 20:15:50 Waiting for app: TestExipryInsert to get un-deployed
2021/12/01 20:15:55 Waiting for app: TestExipryInsert to get un-deployed
2021/12/01 20:16:00 Waiting for app: TestExipryInsert to get un-deployed
2021/12/01 20:16:05 Waiting for app: TestExipryInsert to get un-deployed
2021/12/01 20:16:05 App: TestExipryInsert got un-deployed
2021/12/01 20:16:05 No eventing-con process running
2021/12/01 20:16:05 Function: TestExipryInsert delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryInsert deleting in the background"
}
2021/12/01 20:16:06 Context: Bucket flush, response code: 200 dump: 
2021/12/01 20:16:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 20:16:14 Context: Bucket flush, response code: 200 dump: 
2021/12/01 20:16:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 20:16:19 Testing on the destination bucket for expiry insert
2021/12/01 20:16:19 Deploying app: TestExipryInsert. Item count src bucket: 0 dst bucket: 0
2021/12/01 20:16:20 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryInsert' in metakv",
  "warnings": null
 }
}
2021/12/01 20:16:25 Waiting for app: TestExipryInsert to get deployed
2021/12/01 20:16:30 Waiting for app: TestExipryInsert to get deployed
2021/12/01 20:16:35 Waiting for app: TestExipryInsert to get deployed
2021/12/01 20:16:40 Waiting for app: TestExipryInsert to get deployed
2021/12/01 20:16:40 App: TestExipryInsert got deployed
2021/12/01 20:16:40 Starting bucket ops to source bucket
2021/12/01 20:17:25 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/01 20:17:30 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/01 20:17:30 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 20:17:30 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 20:17:30 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/01 20:17:30 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 20:17:35 Waiting for app: TestExipryInsert to get un-deployed
2021/12/01 20:17:40 Waiting for app: TestExipryInsert to get un-deployed
2021/12/01 20:17:45 Waiting for app: TestExipryInsert to get un-deployed
2021/12/01 20:17:50 Waiting for app: TestExipryInsert to get un-deployed
2021/12/01 20:17:50 App: TestExipryInsert got un-deployed
2021/12/01 20:17:50 No eventing-con process running
2021/12/01 20:17:50 Function: TestExipryInsert delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryInsert deleting in the background"
}
2021/12/01 20:17:52 Context: Bucket flush, response code: 200 dump: 
2021/12/01 20:18:02 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/01 20:18:07 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/01 20:18:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 20:18:09 Context: Bucket flush, response code: 200 dump: 
2021/12/01 20:18:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestExipryInsert (220.74s)
=== RUN   TestExipryReplace
2021/12/01 20:18:14 Deploying app: TestExipryReplace. Item count src bucket: 0 dst bucket: 0
2021/12/01 20:18:15 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryReplace' in metakv",
  "warnings": null
 }
}
2021/12/01 20:18:20 Waiting for app: TestExipryReplace to get deployed
2021/12/01 20:18:25 Waiting for app: TestExipryReplace to get deployed
2021/12/01 20:18:30 Waiting for app: TestExipryReplace to get deployed
2021/12/01 20:18:35 Waiting for app: TestExipryReplace to get deployed
2021/12/01 20:18:35 App: TestExipryReplace got deployed
2021/12/01 20:18:35 Starting bucket ops to source bucket
2021/12/01 20:19:20 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/01 20:19:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 20:19:20 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 20:19:20 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/01 20:19:20 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 20:19:25 Waiting for app: TestExipryReplace to get un-deployed
2021/12/01 20:19:30 Waiting for app: TestExipryReplace to get un-deployed
2021/12/01 20:19:35 Waiting for app: TestExipryReplace to get un-deployed
2021/12/01 20:19:40 Waiting for app: TestExipryReplace to get un-deployed
2021/12/01 20:19:40 App: TestExipryReplace got un-deployed
2021/12/01 20:19:40 No eventing-con process running
2021/12/01 20:19:41 Function: TestExipryReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryReplace deleting in the background"
}
2021/12/01 20:19:42 Context: Bucket flush, response code: 200 dump: 
2021/12/01 20:19:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 20:19:50 Context: Bucket flush, response code: 200 dump: 
2021/12/01 20:19:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 20:19:55 Testing on the destination bucket for expiry insert
2021/12/01 20:19:55 Deploying app: TestExipryReplace. Item count src bucket: 0 dst bucket: 0
2021/12/01 20:19:56 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryReplace' in metakv",
  "warnings": null
 }
}
2021/12/01 20:20:01 Waiting for app: TestExipryReplace to get deployed
2021/12/01 20:20:06 Waiting for app: TestExipryReplace to get deployed
2021/12/01 20:20:11 Waiting for app: TestExipryReplace to get deployed
2021/12/01 20:20:16 Waiting for app: TestExipryReplace to get deployed
2021/12/01 20:20:16 App: TestExipryReplace got deployed
2021/12/01 20:20:16 Starting bucket ops to source bucket
2021/12/01 20:21:01 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/01 20:21:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 20:21:01 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 20:21:01 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/01 20:21:01 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 20:21:06 Waiting for app: TestExipryReplace to get un-deployed
2021/12/01 20:21:11 Waiting for app: TestExipryReplace to get un-deployed
2021/12/01 20:21:16 Waiting for app: TestExipryReplace to get un-deployed
2021/12/01 20:21:21 Waiting for app: TestExipryReplace to get un-deployed
2021/12/01 20:21:21 App: TestExipryReplace got un-deployed
2021/12/01 20:21:21 No eventing-con process running
2021/12/01 20:21:21 Function: TestExipryReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryReplace deleting in the background"
}
2021/12/01 20:21:23 Context: Bucket flush, response code: 200 dump: 
2021/12/01 20:21:33 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/01 20:21:38 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/01 20:21:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 20:21:40 Context: Bucket flush, response code: 200 dump: 
2021/12/01 20:21:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestExipryReplace (210.76s)
=== RUN   TestExipryUpsert
2021/12/01 20:21:45 Deploying app: TestExipryUpsert. Item count src bucket: 0 dst bucket: 0
2021/12/01 20:21:46 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryUpsert' in metakv",
  "warnings": null
 }
}
2021/12/01 20:21:51 Waiting for app: TestExipryUpsert to get deployed
2021/12/01 20:21:56 Waiting for app: TestExipryUpsert to get deployed
2021/12/01 20:22:01 Waiting for app: TestExipryUpsert to get deployed
2021/12/01 20:22:06 Waiting for app: TestExipryUpsert to get deployed
2021/12/01 20:22:06 App: TestExipryUpsert got deployed
2021/12/01 20:22:06 Starting bucket ops to source bucket
2021/12/01 20:22:51 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/01 20:22:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 20:22:51 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 20:22:51 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/01 20:22:51 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 20:22:56 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/01 20:23:01 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/01 20:23:06 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/01 20:23:11 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/01 20:23:11 App: TestExipryUpsert got un-deployed
2021/12/01 20:23:11 No eventing-con process running
2021/12/01 20:23:11 Function: TestExipryUpsert delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryUpsert deleting in the background"
}
2021/12/01 20:23:13 Context: Bucket flush, response code: 200 dump: 
2021/12/01 20:23:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 20:23:20 Context: Bucket flush, response code: 200 dump: 
2021/12/01 20:23:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 20:23:25 Testing on the destination bucket for expiry insert
2021/12/01 20:23:25 Deploying app: TestExipryUpsert. Item count src bucket: 0 dst bucket: 0
2021/12/01 20:23:27 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryUpsert' in metakv",
  "warnings": null
 }
}
2021/12/01 20:23:32 Waiting for app: TestExipryUpsert to get deployed
2021/12/01 20:23:37 Waiting for app: TestExipryUpsert to get deployed
2021/12/01 20:23:42 Waiting for app: TestExipryUpsert to get deployed
2021/12/01 20:23:47 Waiting for app: TestExipryUpsert to get deployed
2021/12/01 20:23:47 App: TestExipryUpsert got deployed
2021/12/01 20:23:47 Starting bucket ops to source bucket
2021/12/01 20:24:32 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/01 20:24:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 20:24:32 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 20:24:32 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/01 20:24:32 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 20:24:37 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/01 20:24:42 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/01 20:24:47 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/01 20:24:52 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/01 20:24:52 App: TestExipryUpsert got un-deployed
2021/12/01 20:24:52 No eventing-con process running
2021/12/01 20:24:52 Function: TestExipryUpsert delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryUpsert deleting in the background"
}
2021/12/01 20:24:54 Context: Bucket flush, response code: 200 dump: 
2021/12/01 20:25:04 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/01 20:25:09 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/01 20:25:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 20:25:10 Context: Bucket flush, response code: 200 dump: 
2021/12/01 20:25:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestExipryUpsert (210.82s)
=== RUN   TestCasReplace
2021/12/01 20:25:15 Deploying app: TestCasReplace. Item count src bucket: 0 dst bucket: 0
2021/12/01 20:25:17 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCasReplace' in metakv",
  "warnings": null
 }
}
2021/12/01 20:25:22 Waiting for app: TestCasReplace to get deployed
2021/12/01 20:25:27 Waiting for app: TestCasReplace to get deployed
2021/12/01 20:25:32 Waiting for app: TestCasReplace to get deployed
2021/12/01 20:25:37 Waiting for app: TestCasReplace to get deployed
2021/12/01 20:25:37 App: TestCasReplace got deployed
2021/12/01 20:25:37 Starting bucket ops to source bucket
2021/12/01 20:25:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 20:25:37 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 20:25:37 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/01 20:25:37 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 20:25:42 Waiting for app: TestCasReplace to get un-deployed
2021/12/01 20:25:47 Waiting for app: TestCasReplace to get un-deployed
2021/12/01 20:25:52 Waiting for app: TestCasReplace to get un-deployed
2021/12/01 20:25:57 Waiting for app: TestCasReplace to get un-deployed
2021/12/01 20:25:57 App: TestCasReplace got un-deployed
2021/12/01 20:25:57 No eventing-con process running
2021/12/01 20:25:57 Function: TestCasReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCasReplace deleting in the background"
}
2021/12/01 20:25:59 Context: Bucket flush, response code: 200 dump: 
2021/12/01 20:26:09 Waiting for dst bucket item count to get to: 0 curr count: 4
2021/12/01 20:26:14 Waiting for dst bucket item count to get to: 0 curr count: 4
2021/12/01 20:26:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 20:26:16 Context: Bucket flush, response code: 200 dump: 
2021/12/01 20:26:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 20:26:21 Testing on the destination bucket for upsert with cas
2021/12/01 20:26:21 Starting bucket ops to source bucket
2021/12/01 20:26:21 Deploying app: TestCasReplace. Item count src bucket: 0 dst bucket: 0
2021/12/01 20:26:22 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCasReplace' in metakv",
  "warnings": null
 }
}
2021/12/01 20:26:27 Waiting for app: TestCasReplace to get deployed
2021/12/01 20:26:32 Waiting for app: TestCasReplace to get deployed
2021/12/01 20:26:37 Waiting for app: TestCasReplace to get deployed
2021/12/01 20:26:42 Waiting for app: TestCasReplace to get deployed
2021/12/01 20:26:42 App: TestCasReplace got deployed
2021/12/01 20:26:42 Starting bucket ops to source bucket
2021/12/01 20:26:47 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/01 20:26:52 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/01 20:26:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 20:26:52 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 20:26:52 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/01 20:26:52 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 20:26:57 Waiting for app: TestCasReplace to get un-deployed
2021/12/01 20:27:02 Waiting for app: TestCasReplace to get un-deployed
2021/12/01 20:27:07 Waiting for app: TestCasReplace to get un-deployed
2021/12/01 20:27:12 Waiting for app: TestCasReplace to get un-deployed
2021/12/01 20:27:12 App: TestCasReplace got un-deployed
2021/12/01 20:27:12 No eventing-con process running
2021/12/01 20:27:12 Function: TestCasReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCasReplace deleting in the background"
}
2021/12/01 20:27:14 Context: Bucket flush, response code: 200 dump: 
2021/12/01 20:27:24 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/01 20:27:29 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/01 20:27:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 20:27:31 Context: Bucket flush, response code: 200 dump: 
2021/12/01 20:27:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCasReplace (140.26s)
=== RUN   TestCasDelete
2021/12/01 20:27:36 Deploying app: TestCasDelete. Item count src bucket: 0 dst bucket: 0
2021/12/01 20:27:37 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCasDelete' in metakv",
  "warnings": null
 }
}
2021/12/01 20:27:42 Waiting for app: TestCasDelete to get deployed
2021/12/01 20:27:47 Waiting for app: TestCasDelete to get deployed
2021/12/01 20:27:52 Waiting for app: TestCasDelete to get deployed
2021/12/01 20:27:57 Waiting for app: TestCasDelete to get deployed
2021/12/01 20:27:57 App: TestCasDelete got deployed
2021/12/01 20:27:57 Starting bucket ops to source bucket
2021/12/01 20:27:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 20:27:57 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 20:27:57 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/01 20:27:57 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 20:28:02 Waiting for app: TestCasDelete to get un-deployed
2021/12/01 20:28:07 Waiting for app: TestCasDelete to get un-deployed
2021/12/01 20:28:12 Waiting for app: TestCasDelete to get un-deployed
2021/12/01 20:28:17 Waiting for app: TestCasDelete to get un-deployed
2021/12/01 20:28:17 App: TestCasDelete got un-deployed
2021/12/01 20:28:17 No eventing-con process running
2021/12/01 20:28:17 Function: TestCasDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCasDelete deleting in the background"
}
2021/12/01 20:28:19 Context: Bucket flush, response code: 200 dump: 
2021/12/01 20:28:29 Waiting for dst bucket item count to get to: 0 curr count: 4
2021/12/01 20:28:34 Waiting for dst bucket item count to get to: 0 curr count: 4
2021/12/01 20:28:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 20:28:36 Context: Bucket flush, response code: 200 dump: 
2021/12/01 20:28:41 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 20:28:41 Testing on the destination bucket for delete with cas
2021/12/01 20:28:41 Starting bucket ops to source bucket
2021/12/01 20:28:41 Deploying app: TestCasDelete. Item count src bucket: 0 dst bucket: 0
2021/12/01 20:28:42 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCasDelete' in metakv",
  "warnings": null
 }
}
2021/12/01 20:28:47 Waiting for app: TestCasDelete to get deployed
2021/12/01 20:28:52 Waiting for app: TestCasDelete to get deployed
2021/12/01 20:28:57 Waiting for app: TestCasDelete to get deployed
2021/12/01 20:29:02 Waiting for app: TestCasDelete to get deployed
2021/12/01 20:29:02 App: TestCasDelete got deployed
2021/12/01 20:29:02 Starting bucket ops to source bucket
2021/12/01 20:29:07 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/01 20:29:12 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/01 20:29:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 20:29:12 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 20:29:12 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/01 20:29:12 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 20:29:17 Waiting for app: TestCasDelete to get un-deployed
2021/12/01 20:29:22 Waiting for app: TestCasDelete to get un-deployed
2021/12/01 20:29:27 Waiting for app: TestCasDelete to get un-deployed
2021/12/01 20:29:32 Waiting for app: TestCasDelete to get un-deployed
2021/12/01 20:29:32 App: TestCasDelete got un-deployed
2021/12/01 20:29:32 No eventing-con process running
2021/12/01 20:29:32 Function: TestCasDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCasDelete deleting in the background"
}
2021/12/01 20:29:34 Context: Bucket flush, response code: 200 dump: 
2021/12/01 20:29:44 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/01 20:29:49 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/01 20:29:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 20:29:51 Context: Bucket flush, response code: 200 dump: 
2021/12/01 20:29:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCasDelete (140.20s)
=== RUN   TestCountersIncrement
2021/12/01 20:29:56 Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/01 20:30:02 Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/01 20:30:02 Removing node(s): [] from the cluster
2021/12/01 20:30:08 Rebalance progress: 100
2021/12/01 20:30:08 Deploying app: TestCountersIncrement. Item count src bucket: 0 dst bucket: 0
2021/12/01 20:30:09 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCountersIncrement' in metakv",
  "warnings": null
 }
}
2021/12/01 20:30:14 Waiting for app: TestCountersIncrement to get deployed
2021/12/01 20:30:19 Waiting for app: TestCountersIncrement to get deployed
2021/12/01 20:30:24 Waiting for app: TestCountersIncrement to get deployed
2021/12/01 20:30:29 Waiting for app: TestCountersIncrement to get deployed
2021/12/01 20:30:29 App: TestCountersIncrement got deployed
2021/12/01 20:30:29 Starting bucket ops to source bucket
2021/12/01 20:30:35 Waiting for dst bucket item count to get to: 1024 curr count: 0
2021/12/01 20:30:40 Waiting for dst bucket item count to get to: 1024 curr count: 233
2021/12/01 20:30:45 Waiting for dst bucket item count to get to: 1024 curr count: 233
2021/12/01 20:30:45 src & dst bucket item count matched up. src bucket count: 1024 dst bucket count: 1024
2021/12/01 20:30:45 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 20:30:45 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 20:30:50 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/01 20:30:55 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/01 20:31:00 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/01 20:31:05 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/01 20:31:05 App: TestCountersIncrement got un-deployed
2021/12/01 20:31:05 No eventing-con process running
2021/12/01 20:31:05 Function: TestCountersIncrement delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCountersIncrement deleting in the background"
}
2021/12/01 20:31:07 Context: Bucket flush, response code: 200 dump: 
2021/12/01 20:31:17 Waiting for dst bucket item count to get to: 0 curr count: 1025
2021/12/01 20:31:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 20:31:19 Context: Bucket flush, response code: 200 dump: 
2021/12/01 20:31:29 Waiting for dst bucket item count to get to: 0 curr count: 918
2021/12/01 20:31:34 Waiting for dst bucket item count to get to: 0 curr count: 406
2021/12/01 20:31:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 20:31:34 Deploying app: TestCountersIncrement. Item count src bucket: 0 dst bucket: 0
2021/12/01 20:31:35 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCountersIncrement' in metakv",
  "warnings": null
 }
}
2021/12/01 20:31:40 Waiting for app: TestCountersIncrement to get deployed
2021/12/01 20:31:45 Waiting for app: TestCountersIncrement to get deployed
2021/12/01 20:31:50 Waiting for app: TestCountersIncrement to get deployed
2021/12/01 20:31:55 Waiting for app: TestCountersIncrement to get deployed
2021/12/01 20:31:55 App: TestCountersIncrement got deployed
2021/12/01 20:31:55 Starting bucket ops to source bucket
2021/12/01 20:32:01 Waiting for dst bucket item count to get to: 2048 curr count: 0
2021/12/01 20:32:06 Waiting for dst bucket item count to get to: 2048 curr count: 1024
2021/12/01 20:32:06 src & dst bucket item count matched up. src bucket count: 2048 dst bucket count: 2048
2021/12/01 20:32:06 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 20:32:06 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 20:32:11 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/01 20:32:16 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/01 20:32:21 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/01 20:32:26 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/01 20:32:26 App: TestCountersIncrement got un-deployed
2021/12/01 20:32:26 No eventing-con process running
2021/12/01 20:32:26 Function: TestCountersIncrement delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCountersIncrement deleting in the background"
}
2021/12/01 20:32:28 Context: Bucket flush, response code: 200 dump: 
2021/12/01 20:32:38 Waiting for dst bucket item count to get to: 0 curr count: 1460
2021/12/01 20:32:43 Waiting for dst bucket item count to get to: 0 curr count: 436
2021/12/01 20:32:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 20:32:45 Context: Bucket flush, response code: 200 dump: 
2021/12/01 20:32:55 Waiting for dst bucket item count to get to: 0 curr count: 1
2021/12/01 20:32:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 20:32:55 Removing node(s): [http://127.0.0.1:19002 http://127.0.0.1:19003] from the cluster
2021/12/01 20:33:00 Rebalance progress: 100
--- PASS: TestCountersIncrement (184.13s)
=== RUN   TestCountersDecrement
2021/12/01 20:33:00 Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/01 20:33:00 Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/01 20:33:00 Removing node(s): [] from the cluster
2021/12/01 20:33:06 Rebalance progress: 100
2021/12/01 20:33:06 Deploying app: TestCountersDecrement. Item count src bucket: 0 dst bucket: 0
2021/12/01 20:33:07 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCountersDecrement' in metakv",
  "warnings": null
 }
}
2021/12/01 20:33:12 Waiting for app: TestCountersDecrement to get deployed
2021/12/01 20:33:17 Waiting for app: TestCountersDecrement to get deployed
2021/12/01 20:33:22 Waiting for app: TestCountersDecrement to get deployed
2021/12/01 20:33:27 Waiting for app: TestCountersDecrement to get deployed
2021/12/01 20:33:27 App: TestCountersDecrement got deployed
2021/12/01 20:33:27 Starting bucket ops to source bucket
2021/12/01 20:33:33 Waiting for dst bucket item count to get to: 1024 curr count: 0
2021/12/01 20:33:38 Waiting for dst bucket item count to get to: 1024 curr count: 619
2021/12/01 20:33:43 Waiting for dst bucket item count to get to: 1024 curr count: 619
2021/12/01 20:33:43 src & dst bucket item count matched up. src bucket count: 1024 dst bucket count: 1024
2021/12/01 20:33:43 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 20:33:43 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 20:33:48 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/01 20:33:53 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/01 20:33:58 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/01 20:34:03 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/01 20:34:03 App: TestCountersDecrement got un-deployed
2021/12/01 20:34:03 No eventing-con process running
2021/12/01 20:34:03 Function: TestCountersDecrement delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCountersDecrement deleting in the background"
}
2021/12/01 20:34:05 Context: Bucket flush, response code: 200 dump: 
2021/12/01 20:34:15 Waiting for dst bucket item count to get to: 0 curr count: 1025
2021/12/01 20:34:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 20:34:17 Context: Bucket flush, response code: 200 dump: 
2021/12/01 20:34:27 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/01 20:34:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 20:34:27 Deploying app: TestCountersDecrement. Item count src bucket: 0 dst bucket: 0
2021/12/01 20:34:28 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCountersDecrement' in metakv",
  "warnings": null
 }
}
2021/12/01 20:34:33 Waiting for app: TestCountersDecrement to get deployed
2021/12/01 20:34:38 Waiting for app: TestCountersDecrement to get deployed
2021/12/01 20:34:43 Waiting for app: TestCountersDecrement to get deployed
2021/12/01 20:34:48 Waiting for app: TestCountersDecrement to get deployed
2021/12/01 20:34:48 App: TestCountersDecrement got deployed
2021/12/01 20:34:48 Starting bucket ops to source bucket
2021/12/01 20:34:54 Waiting for dst bucket item count to get to: 2048 curr count: 0
2021/12/01 20:34:59 Waiting for dst bucket item count to get to: 2048 curr count: 1024
2021/12/01 20:34:59 src & dst bucket item count matched up. src bucket count: 2048 dst bucket count: 2048
2021/12/01 20:34:59 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 20:34:59 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 20:35:04 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/01 20:35:09 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/01 20:35:14 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/01 20:35:19 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/01 20:35:19 App: TestCountersDecrement got un-deployed
2021/12/01 20:35:19 No eventing-con process running
2021/12/01 20:35:19 Function: TestCountersDecrement delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCountersDecrement deleting in the background"
}
2021/12/01 20:35:21 Context: Bucket flush, response code: 200 dump: 
2021/12/01 20:35:31 Waiting for dst bucket item count to get to: 0 curr count: 2048
2021/12/01 20:35:36 Waiting for dst bucket item count to get to: 0 curr count: 2048
2021/12/01 20:35:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 20:35:38 Context: Bucket flush, response code: 200 dump: 
2021/12/01 20:35:48 Waiting for dst bucket item count to get to: 0 curr count: 1
2021/12/01 20:35:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 20:35:48 Removing node(s): [http://127.0.0.1:19002 http://127.0.0.1:19003] from the cluster
2021/12/01 20:35:53 Rebalance progress: 100
--- PASS: TestCountersDecrement (173.09s)
=== RUN   TestCacheGet
2021/12/01 20:35:53 Deploying app: TestCacheGet. Item count src bucket: 0 dst bucket: 0
2021/12/01 20:35:54 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCacheGet' in metakv",
  "warnings": null
 }
}
2021/12/01 20:35:59 Waiting for app: TestCacheGet to get deployed
2021/12/01 20:36:04 Waiting for app: TestCacheGet to get deployed
2021/12/01 20:36:09 Waiting for app: TestCacheGet to get deployed
2021/12/01 20:36:14 Waiting for app: TestCacheGet to get deployed
2021/12/01 20:36:14 App: TestCacheGet got deployed
2021/12/01 20:36:14 Starting bucket ops to source bucket
2021/12/01 20:36:19 Waiting for dst bucket item count to get to: 20 curr count: 0
2021/12/01 20:36:24 Waiting for dst bucket item count to get to: 20 curr count: 0
2021/12/01 20:36:29 Waiting for dst bucket item count to get to: 20 curr count: 0
2021/12/01 20:36:35 Waiting for dst bucket item count to get to: 20 curr count: 8
2021/12/01 20:36:40 Waiting for dst bucket item count to get to: 20 curr count: 16
2021/12/01 20:36:45 Waiting for dst bucket item count to get to: 20 curr count: 18
2021/12/01 20:36:45 src & dst bucket item count matched up. src bucket count: 20 dst bucket count: 20
2021/12/01 20:36:45 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 20:36:45 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 20:36:50 Waiting for app: TestCacheGet to get un-deployed
2021/12/01 20:36:55 Waiting for app: TestCacheGet to get un-deployed
2021/12/01 20:37:00 Waiting for app: TestCacheGet to get un-deployed
2021/12/01 20:37:05 Waiting for app: TestCacheGet to get un-deployed
2021/12/01 20:37:05 App: TestCacheGet got un-deployed
2021/12/01 20:37:05 No eventing-con process running
2021/12/01 20:37:05 Function: TestCacheGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCacheGet deleting in the background"
}
2021/12/01 20:37:07 Context: Bucket flush, response code: 200 dump: 
2021/12/01 20:37:17 Waiting for dst bucket item count to get to: 0 curr count: 20
2021/12/01 20:37:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 20:37:18 Context: Bucket flush, response code: 200 dump: 
2021/12/01 20:37:28 Waiting for dst bucket item count to get to: 0 curr count: 17
2021/12/01 20:37:33 Waiting for dst bucket item count to get to: 0 curr count: 7
2021/12/01 20:37:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCacheGet (100.31s)
=== RUN   TestCacheRYOW
2021/12/01 20:37:33 Deploying app: TestCacheRYOW. Item count src bucket: 0 dst bucket: 0
2021/12/01 20:37:35 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCacheRYOW' in metakv",
  "warnings": null
 }
}
2021/12/01 20:37:40 Waiting for app: TestCacheRYOW to get deployed
2021/12/01 20:37:45 Waiting for app: TestCacheRYOW to get deployed
2021/12/01 20:37:50 Waiting for app: TestCacheRYOW to get deployed
2021/12/01 20:37:55 Waiting for app: TestCacheRYOW to get deployed
2021/12/01 20:37:55 App: TestCacheRYOW got deployed
2021/12/01 20:37:55 Starting bucket ops to source bucket
2021/12/01 20:38:00 Waiting for dst bucket item count to get to: 50 curr count: 0
2021/12/01 20:38:05 Waiting for dst bucket item count to get to: 50 curr count: 25
2021/12/01 20:38:05 src & dst bucket item count matched up. src bucket count: 50 dst bucket count: 50
2021/12/01 20:38:05 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 20:38:05 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 20:38:10 Waiting for app: TestCacheRYOW to get un-deployed
2021/12/01 20:38:15 Waiting for app: TestCacheRYOW to get un-deployed
2021/12/01 20:38:20 Waiting for app: TestCacheRYOW to get un-deployed
2021/12/01 20:38:25 Waiting for app: TestCacheRYOW to get un-deployed
2021/12/01 20:38:25 App: TestCacheRYOW got un-deployed
2021/12/01 20:38:25 No eventing-con process running
2021/12/01 20:38:25 Function: TestCacheRYOW delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCacheRYOW deleting in the background"
}
2021/12/01 20:38:27 Context: Bucket flush, response code: 200 dump: 
2021/12/01 20:38:37 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/01 20:38:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 20:38:39 Context: Bucket flush, response code: 200 dump: 
2021/12/01 20:38:49 Waiting for dst bucket item count to get to: 0 curr count: 47
2021/12/01 20:38:54 Waiting for dst bucket item count to get to: 0 curr count: 22
2021/12/01 20:38:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCacheRYOW (80.36s)
=== RUN   TestCacheExpiry
2021/12/01 20:38:54 Deploying app: TestCacheExpiry. Item count src bucket: 0 dst bucket: 0
2021/12/01 20:38:55 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCacheExpiry' in metakv",
  "warnings": null
 }
}
2021/12/01 20:39:00 Waiting for app: TestCacheExpiry to get deployed
2021/12/01 20:39:05 Waiting for app: TestCacheExpiry to get deployed
2021/12/01 20:39:10 Waiting for app: TestCacheExpiry to get deployed
2021/12/01 20:39:15 Waiting for app: TestCacheExpiry to get deployed
2021/12/01 20:39:15 App: TestCacheExpiry got deployed
2021/12/01 20:39:15 Starting bucket ops to source bucket
2021/12/01 20:39:20 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/01 20:39:25 Waiting for dst bucket item count to get to: 100 curr count: 14
2021/12/01 20:39:30 Waiting for dst bucket item count to get to: 100 curr count: 37
2021/12/01 20:39:35 Waiting for dst bucket item count to get to: 100 curr count: 99
2021/12/01 20:39:40 Waiting for dst bucket item count to get to: 100 curr count: 99
2021/12/01 20:39:40 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/01 20:39:40 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 20:39:40 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 20:39:45 Waiting for app: TestCacheExpiry to get un-deployed
2021/12/01 20:39:50 Waiting for app: TestCacheExpiry to get un-deployed
2021/12/01 20:39:55 Waiting for app: TestCacheExpiry to get un-deployed
2021/12/01 20:40:00 Waiting for app: TestCacheExpiry to get un-deployed
2021/12/01 20:40:00 App: TestCacheExpiry got un-deployed
2021/12/01 20:40:00 No eventing-con process running
2021/12/01 20:40:00 Function: TestCacheExpiry delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCacheExpiry deleting in the background"
}
2021/12/01 20:40:02 Context: Bucket flush, response code: 200 dump: 
2021/12/01 20:40:12 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/01 20:40:17 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/01 20:40:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 20:40:19 Context: Bucket flush, response code: 200 dump: 
2021/12/01 20:40:29 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/01 20:40:34 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/01 20:40:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCacheExpiry (100.24s)
=== RUN   TestCacheExpiryCustom
2021/12/01 20:40:34 Deploying app: TestCacheExpiryCustom. Item count src bucket: 0 dst bucket: 0
2021/12/01 20:40:35 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCacheExpiryCustom' in metakv",
  "warnings": null
 }
}
2021/12/01 20:40:40 Waiting for app: TestCacheExpiryCustom to get deployed
2021/12/01 20:40:45 Waiting for app: TestCacheExpiryCustom to get deployed
2021/12/01 20:40:50 Waiting for app: TestCacheExpiryCustom to get deployed
2021/12/01 20:40:55 Waiting for app: TestCacheExpiryCustom to get deployed
2021/12/01 20:40:55 App: TestCacheExpiryCustom got deployed
2021/12/01 20:40:55 Starting bucket ops to source bucket
2021/12/01 20:41:00 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/01 20:41:05 Waiting for dst bucket item count to get to: 100 curr count: 19
2021/12/01 20:41:10 Waiting for dst bucket item count to get to: 100 curr count: 19
2021/12/01 20:41:15 Waiting for dst bucket item count to get to: 100 curr count: 94
2021/12/01 20:41:20 Waiting for dst bucket item count to get to: 100 curr count: 94
2021/12/01 20:41:20 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/01 20:41:20 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 20:41:21 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 20:41:26 Waiting for app: TestCacheExpiryCustom to get un-deployed
2021/12/01 20:41:31 Waiting for app: TestCacheExpiryCustom to get un-deployed
2021/12/01 20:41:36 Waiting for app: TestCacheExpiryCustom to get un-deployed
2021/12/01 20:41:41 Waiting for app: TestCacheExpiryCustom to get un-deployed
2021/12/01 20:41:41 App: TestCacheExpiryCustom got un-deployed
2021/12/01 20:41:41 No eventing-con process running
2021/12/01 20:41:41 Function: TestCacheExpiryCustom delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCacheExpiryCustom deleting in the background"
}
2021/12/01 20:41:43 Context: Bucket flush, response code: 200 dump: 
2021/12/01 20:41:53 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/01 20:41:58 Waiting for dst bucket item count to get to: 0 curr count: 25
2021/12/01 20:41:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 20:41:59 Context: Bucket flush, response code: 200 dump: 
2021/12/01 20:42:09 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/01 20:42:14 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/01 20:42:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCacheExpiryCustom (100.31s)
=== RUN   TestCacheOverflowCustom
2021/12/01 20:42:14 Deploying app: TestCacheOverflowCustom. Item count src bucket: 0 dst bucket: 0
2021/12/01 20:42:16 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCacheOverflowCustom' in metakv",
  "warnings": null
 }
}
2021/12/01 20:42:21 Waiting for app: TestCacheOverflowCustom to get deployed
2021/12/01 20:42:26 Waiting for app: TestCacheOverflowCustom to get deployed
2021/12/01 20:42:31 Waiting for app: TestCacheOverflowCustom to get deployed
2021/12/01 20:42:36 Waiting for app: TestCacheOverflowCustom to get deployed
2021/12/01 20:42:36 App: TestCacheOverflowCustom got deployed
2021/12/01 20:42:36 Starting bucket ops to source bucket
2021/12/01 20:42:41 Waiting for dst bucket item count to get to: 35 curr count: 0
2021/12/01 20:42:46 Waiting for dst bucket item count to get to: 35 curr count: 12
2021/12/01 20:42:51 Waiting for dst bucket item count to get to: 35 curr count: 12
2021/12/01 20:42:56 Waiting for dst bucket item count to get to: 35 curr count: 27
2021/12/01 20:43:01 Waiting for dst bucket item count to get to: 35 curr count: 27
2021/12/01 20:43:01 src & dst bucket item count matched up. src bucket count: 35 dst bucket count: 35
2021/12/01 20:43:01 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 20:43:01 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 20:43:06 Waiting for app: TestCacheOverflowCustom to get un-deployed
2021/12/01 20:43:11 Waiting for app: TestCacheOverflowCustom to get un-deployed
2021/12/01 20:43:16 Waiting for app: TestCacheOverflowCustom to get un-deployed
2021/12/01 20:43:21 Waiting for app: TestCacheOverflowCustom to get un-deployed
2021/12/01 20:43:21 App: TestCacheOverflowCustom got un-deployed
2021/12/01 20:43:21 No eventing-con process running
2021/12/01 20:43:21 Function: TestCacheOverflowCustom delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCacheOverflowCustom deleting in the background"
}
2021/12/01 20:43:23 Context: Bucket flush, response code: 200 dump: 
2021/12/01 20:43:33 Waiting for dst bucket item count to get to: 0 curr count: 5
2021/12/01 20:43:38 Waiting for dst bucket item count to get to: 0 curr count: 3
2021/12/01 20:43:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 20:43:40 Context: Bucket flush, response code: 200 dump: 
2021/12/01 20:43:50 Waiting for dst bucket item count to get to: 0 curr count: 17
2021/12/01 20:43:55 Waiting for dst bucket item count to get to: 0 curr count: 17
2021/12/01 20:43:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCacheOverflowCustom (100.34s)
=== RUN   TestStrictMode
2021/12/01 20:43:55 Deploying app: TestStrictMode. Item count src bucket: 0 dst bucket: 0
2021/12/01 20:43:56 Context: Post to main store, response code: 422 dump: {
 "name": "ERR_HANDLER_COMPILATION",
 "code": 27,
 "description": "Function compilation failed",
 "attributes": null,
 "runtime_info": {
  "code": 27,
  "info": {
   "area": "handlerCode",
   "column_number": 63,
   "compile_success": false,
   "description": "        destination[meta.id] = 00000000000000000000000000000101;\n                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nSyntaxError: Octal literals are not allowed in strict mode.\n",
   "index": 132,
   "language": "JavaScript",
   "line_number": 4
  }
 }
}
--- PASS: TestStrictMode (1.06s)
=== RUN   TestDataTypes
2021/12/01 20:43:56 Deploying app: TestDataTypes. Item count src bucket: 0 dst bucket: 0
2021/12/01 20:43:57 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDataTypes' in metakv",
  "warnings": null
 }
}
2021/12/01 20:44:02 Waiting for app: TestDataTypes to get deployed
2021/12/01 20:44:07 Waiting for app: TestDataTypes to get deployed
2021/12/01 20:44:12 Waiting for app: TestDataTypes to get deployed
2021/12/01 20:44:17 Waiting for app: TestDataTypes to get deployed
2021/12/01 20:44:17 App: TestDataTypes got deployed
2021/12/01 20:44:17 Starting bucket ops to source bucket
2021/12/01 20:44:22 Waiting for dst bucket item count to get to: 18 curr count: 0
2021/12/01 20:44:22 src & dst bucket item count matched up. src bucket count: 18 dst bucket count: 18
2021/12/01 20:44:22 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 20:44:22 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 20:44:27 Waiting for app: TestDataTypes to get un-deployed
2021/12/01 20:44:32 Waiting for app: TestDataTypes to get un-deployed
2021/12/01 20:44:37 Waiting for app: TestDataTypes to get un-deployed
2021/12/01 20:44:42 Waiting for app: TestDataTypes to get un-deployed
2021/12/01 20:44:42 App: TestDataTypes got un-deployed
2021/12/01 20:44:42 No eventing-con process running
2021/12/01 20:44:42 Function: TestDataTypes delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDataTypes deleting in the background"
}
2021/12/01 20:44:44 Context: Bucket flush, response code: 200 dump: 
2021/12/01 20:44:54 Waiting for dst bucket item count to get to: 0 curr count: 1
2021/12/01 20:44:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 20:44:56 Context: Bucket flush, response code: 200 dump: 
2021/12/01 20:45:06 Waiting for dst bucket item count to get to: 0 curr count: 18
2021/12/01 20:45:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDataTypes (70.22s)
=== RUN   TestEnoentGet
2021/12/01 20:45:06 Deploying app: TestEnoentGet. Item count src bucket: 0 dst bucket: 0
2021/12/01 20:45:07 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentGet' in metakv",
  "warnings": null
 }
}
2021/12/01 20:45:12 Waiting for app: TestEnoentGet to get deployed
2021/12/01 20:45:17 Waiting for app: TestEnoentGet to get deployed
2021/12/01 20:45:22 Waiting for app: TestEnoentGet to get deployed
2021/12/01 20:45:27 Waiting for app: TestEnoentGet to get deployed
2021/12/01 20:45:27 App: TestEnoentGet got deployed
2021/12/01 20:45:27 Starting bucket ops to source bucket
2021/12/01 20:45:32 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/01 20:45:37 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/01 20:45:42 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/01 20:45:42 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/01 20:45:42 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 20:45:42 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 20:45:47 Waiting for app: TestEnoentGet to get un-deployed
2021/12/01 20:45:52 Waiting for app: TestEnoentGet to get un-deployed
2021/12/01 20:45:57 Waiting for app: TestEnoentGet to get un-deployed
2021/12/01 20:46:02 Waiting for app: TestEnoentGet to get un-deployed
2021/12/01 20:46:02 App: TestEnoentGet got un-deployed
2021/12/01 20:46:02 No eventing-con process running
2021/12/01 20:46:02 Function: TestEnoentGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentGet deleting in the background"
}
2021/12/01 20:46:04 Context: Bucket flush, response code: 200 dump: 
2021/12/01 20:46:14 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/01 20:46:19 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/01 20:46:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 20:46:21 Context: Bucket flush, response code: 200 dump: 
2021/12/01 20:46:31 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/01 20:46:36 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/01 20:46:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 20:46:36 Deploying app: TestEnoentGet. Item count src bucket: 0 dst bucket: 0
2021/12/01 20:46:37 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentGet' in metakv",
  "warnings": null
 }
}
2021/12/01 20:46:42 Waiting for app: TestEnoentGet to get deployed
2021/12/01 20:46:47 Waiting for app: TestEnoentGet to get deployed
2021/12/01 20:46:52 Waiting for app: TestEnoentGet to get deployed
2021/12/01 20:46:57 Waiting for app: TestEnoentGet to get deployed
2021/12/01 20:46:57 App: TestEnoentGet got deployed
2021/12/01 20:46:57 Starting bucket ops to source bucket
2021/12/01 20:47:02 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/01 20:47:07 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/01 20:47:12 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/01 20:47:12 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/01 20:47:12 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 20:47:13 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 20:47:18 Waiting for app: TestEnoentGet to get un-deployed
2021/12/01 20:47:23 Waiting for app: TestEnoentGet to get un-deployed
2021/12/01 20:47:28 Waiting for app: TestEnoentGet to get un-deployed
2021/12/01 20:47:33 Waiting for app: TestEnoentGet to get un-deployed
2021/12/01 20:47:33 App: TestEnoentGet got un-deployed
2021/12/01 20:47:33 No eventing-con process running
2021/12/01 20:47:33 Function: TestEnoentGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentGet deleting in the background"
}
2021/12/01 20:47:34 Context: Bucket flush, response code: 200 dump: 
2021/12/01 20:47:44 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/01 20:47:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 20:47:46 Context: Bucket flush, response code: 200 dump: 
2021/12/01 20:47:56 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/01 20:47:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEnoentGet (170.27s)
=== RUN   TestEnoentDelete
2021/12/01 20:47:56 Deploying app: TestEnoentDelete. Item count src bucket: 0 dst bucket: 0
2021/12/01 20:47:57 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentDelete' in metakv",
  "warnings": null
 }
}
2021/12/01 20:48:02 Waiting for app: TestEnoentDelete to get deployed
2021/12/01 20:48:07 Waiting for app: TestEnoentDelete to get deployed
2021/12/01 20:48:12 Waiting for app: TestEnoentDelete to get deployed
2021/12/01 20:48:17 Waiting for app: TestEnoentDelete to get deployed
2021/12/01 20:48:17 App: TestEnoentDelete got deployed
2021/12/01 20:48:17 Starting bucket ops to source bucket
2021/12/01 20:48:23 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/01 20:48:28 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/01 20:48:33 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/01 20:48:33 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/01 20:48:33 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 20:48:33 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 20:48:38 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/01 20:48:43 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/01 20:48:48 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/01 20:48:53 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/01 20:48:53 App: TestEnoentDelete got un-deployed
2021/12/01 20:48:53 No eventing-con process running
2021/12/01 20:48:53 Function: TestEnoentDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentDelete deleting in the background"
}
2021/12/01 20:48:55 Context: Bucket flush, response code: 200 dump: 
2021/12/01 20:49:05 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/01 20:49:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 20:49:06 Context: Bucket flush, response code: 200 dump: 
2021/12/01 20:49:16 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/01 20:49:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 20:49:17 Deploying app: TestEnoentDelete. Item count src bucket: 0 dst bucket: 0
2021/12/01 20:49:18 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentDelete' in metakv",
  "warnings": null
 }
}
2021/12/01 20:49:23 Waiting for app: TestEnoentDelete to get deployed
2021/12/01 20:49:28 Waiting for app: TestEnoentDelete to get deployed
2021/12/01 20:49:33 Waiting for app: TestEnoentDelete to get deployed
2021/12/01 20:49:38 Waiting for app: TestEnoentDelete to get deployed
2021/12/01 20:49:38 App: TestEnoentDelete got deployed
2021/12/01 20:49:38 Starting bucket ops to source bucket
2021/12/01 20:49:43 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/01 20:49:48 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/01 20:49:53 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/01 20:49:53 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/01 20:49:53 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 20:49:53 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 20:49:58 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/01 20:50:03 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/01 20:50:08 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/01 20:50:13 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/01 20:50:13 App: TestEnoentDelete got un-deployed
2021/12/01 20:50:13 No eventing-con process running
2021/12/01 20:50:13 Function: TestEnoentDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentDelete deleting in the background"
}
2021/12/01 20:50:15 Context: Bucket flush, response code: 200 dump: 
2021/12/01 20:50:25 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/01 20:50:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 20:50:27 Context: Bucket flush, response code: 200 dump: 
2021/12/01 20:50:37 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/01 20:50:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEnoentDelete (160.48s)
=== RUN   TestError
2021/12/01 20:50:47 Waiting for app: TestError to get un-deployed
2021/12/01 20:50:47 App: TestError got un-deployed
2021/12/01 20:50:47 No eventing-con process running
2021/12/01 20:50:47 Function: TestError delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestError not found"
 }
}
2021/12/01 20:50:49 Context: Bucket flush, response code: 200 dump: 
2021/12/01 20:50:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 20:50:55 Context: Bucket flush, response code: 200 dump: 
2021/12/01 20:51:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 20:51:00 Deploying app: TestError. Item count src bucket: 0 dst bucket: 0
2021/12/01 20:51:02 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestError' in metakv",
  "warnings": null
 }
}
2021/12/01 20:51:07 Waiting for app: TestError to get deployed
2021/12/01 20:51:12 Waiting for app: TestError to get deployed
2021/12/01 20:51:17 Waiting for app: TestError to get deployed
2021/12/01 20:51:22 Waiting for app: TestError to get deployed
2021/12/01 20:51:22 App: TestError got deployed
2021/12/01 20:51:22 Starting bucket ops to source bucket
2021/12/01 20:51:27 Waiting for dst bucket item count to get to: 300 curr count: 0
2021/12/01 20:51:32 Waiting for dst bucket item count to get to: 300 curr count: 0
2021/12/01 20:51:32 src & dst bucket item count matched up. src bucket count: 300 dst bucket count: 300
2021/12/01 20:51:32 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 20:51:32 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 20:51:37 Waiting for app: TestError to get un-deployed
2021/12/01 20:51:42 Waiting for app: TestError to get un-deployed
2021/12/01 20:51:47 Waiting for app: TestError to get un-deployed
2021/12/01 20:51:52 Waiting for app: TestError to get un-deployed
2021/12/01 20:51:52 App: TestError got un-deployed
2021/12/01 20:51:52 No eventing-con process running
2021/12/01 20:51:52 Function: TestError delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestError deleting in the background"
}
2021/12/01 20:51:54 Context: Bucket flush, response code: 200 dump: 
2021/12/01 20:52:04 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/01 20:52:09 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/01 20:52:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 20:52:11 Context: Bucket flush, response code: 200 dump: 
2021/12/01 20:52:21 Waiting for dst bucket item count to get to: 0 curr count: 250
2021/12/01 20:52:26 Waiting for dst bucket item count to get to: 0 curr count: 250
2021/12/01 20:52:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestError (108.86s)
=== RUN   TestCRLF
2021/12/01 20:52:36 Waiting for app: TestCRLF to get un-deployed
2021/12/01 20:52:36 App: TestCRLF got un-deployed
2021/12/01 20:52:36 No eventing-con process running
2021/12/01 20:52:36 Function: TestCRLF delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestCRLF not found"
 }
}
2021/12/01 20:52:37 Context: Bucket flush, response code: 200 dump: 
2021/12/01 20:52:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 20:52:44 Context: Bucket flush, response code: 200 dump: 
2021/12/01 20:52:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 20:52:49 Deploying app: TestCRLF. Item count src bucket: 0 dst bucket: 0
2021/12/01 20:52:50 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCRLF' in metakv",
  "warnings": null
 }
}
2021/12/01 20:52:55 Waiting for app: TestCRLF to get deployed
2021/12/01 20:53:00 Waiting for app: TestCRLF to get deployed
2021/12/01 20:53:05 Waiting for app: TestCRLF to get deployed
2021/12/01 20:53:10 Waiting for app: TestCRLF to get deployed
2021/12/01 20:53:11 App: TestCRLF got deployed
2021/12/01 20:53:11 Starting bucket ops to source bucket
2021/12/01 20:53:16 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/01 20:53:21 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/01 20:53:21 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/01 20:53:21 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 20:53:21 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 20:53:26 Waiting for app: TestCRLF to get un-deployed
2021/12/01 20:53:31 Waiting for app: TestCRLF to get un-deployed
2021/12/01 20:53:36 Waiting for app: TestCRLF to get un-deployed
2021/12/01 20:53:41 Waiting for app: TestCRLF to get un-deployed
2021/12/01 20:53:41 App: TestCRLF got un-deployed
2021/12/01 20:53:41 No eventing-con process running
2021/12/01 20:53:41 Function: TestCRLF delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCRLF deleting in the background"
}
2021/12/01 20:53:43 Context: Bucket flush, response code: 200 dump: 
2021/12/01 20:53:53 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/01 20:53:58 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/01 20:53:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 20:54:00 Context: Bucket flush, response code: 200 dump: 
2021/12/01 20:54:10 Waiting for dst bucket item count to get to: 0 curr count: 62
2021/12/01 20:54:15 Waiting for dst bucket item count to get to: 0 curr count: 62
2021/12/01 20:54:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCRLF (109.01s)
=== RUN   TestImportExport
2021/12/01 20:54:25 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/01 20:54:25 App: TestImportExport_n1ql_insert_on_update got un-deployed
2021/12/01 20:54:25 No eventing-con process running
2021/12/01 20:54:25 Function: TestImportExport_n1ql_insert_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestImportExport_n1ql_insert_on_update not found"
 }
}
2021/12/01 20:54:26 Context: Bucket flush, response code: 200 dump: 
2021/12/01 20:54:31 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 20:54:33 Context: Bucket flush, response code: 200 dump: 
2021/12/01 20:54:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 20:54:43 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/01 20:54:43 App: TestImportExport_bucket_op_on_update got un-deployed
2021/12/01 20:54:43 No eventing-con process running
2021/12/01 20:54:43 Function: TestImportExport_bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestImportExport_bucket_op_on_update not found"
 }
}
2021/12/01 20:54:45 Context: Bucket flush, response code: 200 dump: 
2021/12/01 20:54:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 20:54:52 Context: Bucket flush, response code: 200 dump: 
2021/12/01 20:54:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 20:54:57 Deploying app: TestImportExport_n1ql_insert_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/01 20:54:58 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestImportExport_n1ql_insert_on_update' in metakv",
  "warnings": null
 }
}
2021/12/01 20:54:58 Deploying app: TestImportExport_bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/01 20:54:59 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestImportExport_bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/01 20:55:04 Waiting for app: TestImportExport_n1ql_insert_on_update to get deployed
2021/12/01 20:55:09 Waiting for app: TestImportExport_n1ql_insert_on_update to get deployed
2021/12/01 20:55:14 Waiting for app: TestImportExport_n1ql_insert_on_update to get deployed
2021/12/01 20:55:19 Waiting for app: TestImportExport_n1ql_insert_on_update to get deployed
2021/12/01 20:55:19 App: TestImportExport_n1ql_insert_on_update got deployed
2021/12/01 20:55:24 Waiting for app: TestImportExport_bucket_op_on_update to get deployed
2021/12/01 20:55:29 Waiting for app: TestImportExport_bucket_op_on_update to get deployed
2021/12/01 20:55:34 Waiting for app: TestImportExport_bucket_op_on_update to get deployed
2021/12/01 20:55:34 App: TestImportExport_bucket_op_on_update got deployed
2021/12/01 20:55:35 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 20:55:40 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/01 20:55:45 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/01 20:55:50 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/01 20:55:55 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/01 20:55:55 App: TestImportExport_n1ql_insert_on_update got un-deployed
2021/12/01 20:55:55 No eventing-con process running
2021/12/01 20:55:55 Function: TestImportExport_n1ql_insert_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestImportExport_n1ql_insert_on_update deleting in the background"
}
2021/12/01 20:55:55 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 20:56:00 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/01 20:56:05 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/01 20:56:10 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/01 20:56:15 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/01 20:56:15 App: TestImportExport_bucket_op_on_update got un-deployed
2021/12/01 20:56:15 No eventing-con process running
2021/12/01 20:56:15 Function: TestImportExport_bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestImportExport_bucket_op_on_update deleting in the background"
}
2021/12/01 20:56:32 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/01 20:56:32 App: TestImportExport_bucket_op_on_update got un-deployed
2021/12/01 20:56:32 No eventing-con process running
2021/12/01 20:56:32 Function: TestImportExport_bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestImportExport_bucket_op_on_update deleting in the background"
}
2021/12/01 20:56:34 Context: Bucket flush, response code: 200 dump: 
2021/12/01 20:56:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 20:56:41 Context: Bucket flush, response code: 200 dump: 
2021/12/01 20:56:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 20:56:51 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/01 20:56:51 App: TestImportExport_n1ql_insert_on_update got un-deployed
2021/12/01 20:56:51 No eventing-con process running
2021/12/01 20:56:51 Function: TestImportExport_n1ql_insert_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestImportExport_n1ql_insert_on_update deleting in the background"
}
2021/12/01 20:56:53 Context: Bucket flush, response code: 200 dump: 
2021/12/01 20:56:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 20:56:59 Context: Bucket flush, response code: 200 dump: 
2021/12/01 20:57:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestImportExport (169.89s)
=== RUN   TestDeployUndeployLoopNonDefaultSettings
2021/12/01 20:57:14 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/01 20:57:15 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/01 20:57:15 No eventing-con process running
2021/12/01 20:57:15 Function: TestDeployUndeployLoopNonDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDeployUndeployLoopNonDefaultSettings not found"
 }
}
2021/12/01 20:57:16 Context: Bucket flush, response code: 200 dump: 
2021/12/01 20:57:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 20:57:23 Context: Bucket flush, response code: 200 dump: 
2021/12/01 20:57:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 20:57:28 Deploying app: TestDeployUndeployLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/01 20:57:29 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/01 20:57:29 Starting bucket ops to source bucket
2021/12/01 20:57:37 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 20:57:42 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 20:57:47 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 20:57:52 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 20:57:57 Waiting for dst bucket item count to get to: 5000 curr count: 2500
2021/12/01 20:57:57 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/01 20:58:02 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get deployed
2021/12/01 20:58:02 App: TestDeployUndeployLoopNonDefaultSettings got deployed
2021/12/01 20:58:02 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 20:58:03 Undeploying app: bucket_op_on_update
2021/12/01 20:58:03 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 20:58:08 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/01 20:58:13 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/01 20:58:18 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/01 20:58:23 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/01 20:58:23 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/01 20:58:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 20:58:23 No eventing-con process running
2021/12/01 20:58:25 Context: Bucket flush, response code: 200 dump: 
2021/12/01 20:58:32 Context: Bucket flush, response code: 200 dump: 
2021/12/01 20:59:07 Deploying app: TestDeployUndeployLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/01 20:59:08 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/01 20:59:08 Starting bucket ops to source bucket
2021/12/01 20:59:16 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 20:59:21 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 20:59:26 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 20:59:31 Waiting for dst bucket item count to get to: 5000 curr count: 38
2021/12/01 20:59:36 Waiting for dst bucket item count to get to: 5000 curr count: 2538
2021/12/01 20:59:36 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/01 20:59:41 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get deployed
2021/12/01 20:59:41 App: TestDeployUndeployLoopNonDefaultSettings got deployed
2021/12/01 20:59:41 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 20:59:41 Undeploying app: bucket_op_on_update
2021/12/01 20:59:41 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 20:59:46 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/01 20:59:51 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/01 20:59:56 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/01 21:00:01 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/01 21:00:01 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/01 21:00:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 21:00:01 No eventing-con process running
2021/12/01 21:00:03 Context: Bucket flush, response code: 200 dump: 
2021/12/01 21:00:11 Context: Bucket flush, response code: 200 dump: 
2021/12/01 21:00:46 Deploying app: TestDeployUndeployLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/01 21:00:47 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/01 21:00:47 Starting bucket ops to source bucket
2021/12/01 21:00:55 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 21:01:00 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 21:01:05 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 21:01:10 Waiting for dst bucket item count to get to: 5000 curr count: 1002
2021/12/01 21:01:15 Waiting for dst bucket item count to get to: 5000 curr count: 1002
2021/12/01 21:01:15 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/01 21:01:20 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get deployed
2021/12/01 21:01:20 App: TestDeployUndeployLoopNonDefaultSettings got deployed
2021/12/01 21:01:20 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 21:01:20 Undeploying app: bucket_op_on_update
2021/12/01 21:01:20 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 21:01:25 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/01 21:01:30 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/01 21:01:35 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/01 21:01:40 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/01 21:01:40 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/01 21:01:40 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 21:01:40 No eventing-con process running
2021/12/01 21:01:42 Context: Bucket flush, response code: 200 dump: 
2021/12/01 21:01:49 Context: Bucket flush, response code: 200 dump: 
2021/12/01 21:02:24 Deploying app: TestDeployUndeployLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/01 21:02:25 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/01 21:02:25 Starting bucket ops to source bucket
2021/12/01 21:02:33 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 21:02:38 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 21:02:43 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 21:02:48 Waiting for dst bucket item count to get to: 5000 curr count: 2532
2021/12/01 21:02:53 Waiting for dst bucket item count to get to: 5000 curr count: 2532
2021/12/01 21:02:53 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/01 21:02:58 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get deployed
2021/12/01 21:02:58 App: TestDeployUndeployLoopNonDefaultSettings got deployed
2021/12/01 21:02:58 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 21:02:58 Undeploying app: bucket_op_on_update
2021/12/01 21:02:58 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 21:03:03 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/01 21:03:08 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/01 21:03:13 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/01 21:03:18 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/01 21:03:18 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/01 21:03:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 21:03:18 No eventing-con process running
2021/12/01 21:03:20 Context: Bucket flush, response code: 200 dump: 
2021/12/01 21:03:28 Context: Bucket flush, response code: 200 dump: 
2021/12/01 21:04:03 Deploying app: TestDeployUndeployLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/01 21:04:04 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/01 21:04:04 Starting bucket ops to source bucket
2021/12/01 21:04:12 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 21:04:17 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 21:04:22 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 21:04:27 Waiting for dst bucket item count to get to: 5000 curr count: 1944
2021/12/01 21:04:32 Waiting for dst bucket item count to get to: 5000 curr count: 4444
2021/12/01 21:04:32 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/01 21:04:37 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get deployed
2021/12/01 21:04:37 App: TestDeployUndeployLoopNonDefaultSettings got deployed
2021/12/01 21:04:37 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 21:04:37 Undeploying app: bucket_op_on_update
2021/12/01 21:04:37 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 21:04:42 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/01 21:04:47 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/01 21:04:52 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/01 21:04:57 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/01 21:04:57 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/01 21:04:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 21:04:57 No eventing-con process running
2021/12/01 21:04:59 Context: Bucket flush, response code: 200 dump: 
2021/12/01 21:05:06 Context: Bucket flush, response code: 200 dump: 
2021/12/01 21:05:41 Function: TestDeployUndeployLoopNonDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDeployUndeployLoopNonDefaultSettings deleting in the background"
}
--- PASS: TestDeployUndeployLoopNonDefaultSettings (516.62s)
=== RUN   TestOnUpdateN1QLOp
2021/12/01 21:05:51 Waiting for app: TestOnUpdateN1QLOp to get un-deployed
2021/12/01 21:05:51 App: TestOnUpdateN1QLOp got un-deployed
2021/12/01 21:05:51 No eventing-con process running
2021/12/01 21:05:51 Function: TestOnUpdateN1QLOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateN1QLOp not found"
 }
}
2021/12/01 21:05:53 Context: Bucket flush, response code: 200 dump: 
2021/12/01 21:05:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 21:06:00 Context: Bucket flush, response code: 200 dump: 
2021/12/01 21:06:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 21:06:05 Deploying app: TestOnUpdateN1QLOp. Item count src bucket: 0 dst bucket: 0
2021/12/01 21:06:06 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateN1QLOp' in metakv",
  "warnings": null
 }
}
2021/12/01 21:06:06 Starting bucket ops to source bucket
2021/12/01 21:06:14 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 21:06:19 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 21:06:24 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 21:06:29 Waiting for dst bucket item count to get to: 5000 curr count: 673
2021/12/01 21:06:34 Waiting for dst bucket item count to get to: 5000 curr count: 673
2021/12/01 21:06:34 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/01 21:06:34 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 21:06:34 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 21:06:39 Waiting for app: TestOnUpdateN1QLOp to get un-deployed
2021/12/01 21:06:44 Waiting for app: TestOnUpdateN1QLOp to get un-deployed
2021/12/01 21:06:49 Waiting for app: TestOnUpdateN1QLOp to get un-deployed
2021/12/01 21:06:54 Waiting for app: TestOnUpdateN1QLOp to get un-deployed
2021/12/01 21:06:54 App: TestOnUpdateN1QLOp got un-deployed
2021/12/01 21:06:54 No eventing-con process running
2021/12/01 21:06:54 Function: TestOnUpdateN1QLOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateN1QLOp deleting in the background"
}
2021/12/01 21:06:56 Context: Bucket flush, response code: 200 dump: 
2021/12/01 21:07:06 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/01 21:07:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 21:07:08 Context: Bucket flush, response code: 200 dump: 
2021/12/01 21:07:18 Waiting for dst bucket item count to get to: 0 curr count: 3518
2021/12/01 21:07:23 Waiting for dst bucket item count to get to: 0 curr count: 1039
2021/12/01 21:07:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateN1QLOp (101.88s)
=== RUN   TestOnUpdateBucketOpDefaultSettings
2021/12/01 21:07:33 Waiting for app: TestOnUpdateBucketOpDefaultSettings to get un-deployed
2021/12/01 21:07:33 App: TestOnUpdateBucketOpDefaultSettings got un-deployed
2021/12/01 21:07:33 No eventing-con process running
2021/12/01 21:07:33 Function: TestOnUpdateBucketOpDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateBucketOpDefaultSettings not found"
 }
}
2021/12/01 21:07:35 Context: Bucket flush, response code: 200 dump: 
2021/12/01 21:07:40 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 21:07:42 Context: Bucket flush, response code: 200 dump: 
2021/12/01 21:07:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 21:07:47 Deploying app: TestOnUpdateBucketOpDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/01 21:07:48 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateBucketOpDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/01 21:07:48 Starting bucket ops to source bucket
2021/12/01 21:07:56 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 21:08:01 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 21:08:06 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 21:08:11 Waiting for dst bucket item count to get to: 5000 curr count: 265
2021/12/01 21:08:16 Waiting for dst bucket item count to get to: 5000 curr count: 265
2021/12/01 21:08:16 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/01 21:08:16 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 21:08:16 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 21:08:21 Waiting for app: TestOnUpdateBucketOpDefaultSettings to get un-deployed
2021/12/01 21:08:26 Waiting for app: TestOnUpdateBucketOpDefaultSettings to get un-deployed
2021/12/01 21:08:31 Waiting for app: TestOnUpdateBucketOpDefaultSettings to get un-deployed
2021/12/01 21:08:36 Waiting for app: TestOnUpdateBucketOpDefaultSettings to get un-deployed
2021/12/01 21:08:36 App: TestOnUpdateBucketOpDefaultSettings got un-deployed
2021/12/01 21:08:36 No eventing-con process running
2021/12/01 21:08:36 Function: TestOnUpdateBucketOpDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateBucketOpDefaultSettings deleting in the background"
}
2021/12/01 21:08:38 Context: Bucket flush, response code: 200 dump: 
2021/12/01 21:08:48 Waiting for dst bucket item count to get to: 0 curr count: 2909
2021/12/01 21:08:53 Waiting for dst bucket item count to get to: 0 curr count: 409
2021/12/01 21:08:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 21:08:55 Context: Bucket flush, response code: 200 dump: 
2021/12/01 21:09:05 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/01 21:09:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateBucketOpDefaultSettings (101.75s)
=== RUN   TestOnUpdateBucketOpNonDefaultSettings
2021/12/01 21:09:15 Waiting for app: TestOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/01 21:09:15 App: TestOnUpdateBucketOpNonDefaultSettings got un-deployed
2021/12/01 21:09:15 No eventing-con process running
2021/12/01 21:09:15 Function: TestOnUpdateBucketOpNonDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateBucketOpNonDefaultSettings not found"
 }
}
2021/12/01 21:09:17 Context: Bucket flush, response code: 200 dump: 
2021/12/01 21:09:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 21:09:23 Context: Bucket flush, response code: 200 dump: 
2021/12/01 21:09:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 21:09:28 Deploying app: TestOnUpdateBucketOpNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/01 21:09:30 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateBucketOpNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/01 21:09:30 Starting bucket ops to source bucket
2021/12/01 21:09:38 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 21:09:43 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 21:09:48 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 21:09:53 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 21:09:58 Waiting for dst bucket item count to get to: 5000 curr count: 2500
2021/12/01 21:09:58 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/01 21:09:58 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 21:09:58 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 21:10:03 Waiting for app: TestOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/01 21:10:08 Waiting for app: TestOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/01 21:10:13 Waiting for app: TestOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/01 21:10:18 Waiting for app: TestOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/01 21:10:18 App: TestOnUpdateBucketOpNonDefaultSettings got un-deployed
2021/12/01 21:10:18 No eventing-con process running
2021/12/01 21:10:18 Function: TestOnUpdateBucketOpNonDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateBucketOpNonDefaultSettings deleting in the background"
}
2021/12/01 21:10:20 Context: Bucket flush, response code: 200 dump: 
2021/12/01 21:10:30 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/01 21:10:35 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/01 21:10:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 21:10:36 Context: Bucket flush, response code: 200 dump: 
2021/12/01 21:10:46 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/01 21:10:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateBucketOpNonDefaultSettings (101.76s)
=== RUN   TestOnUpdateBucketOpDefaultSettings10K
2021/12/01 21:10:56 Waiting for app: TestOnUpdateBucketOpDefaultSettings10K to get un-deployed
2021/12/01 21:10:57 App: TestOnUpdateBucketOpDefaultSettings10K got un-deployed
2021/12/01 21:10:57 No eventing-con process running
2021/12/01 21:10:57 Function: TestOnUpdateBucketOpDefaultSettings10K delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateBucketOpDefaultSettings10K not found"
 }
}
2021/12/01 21:10:58 Context: Bucket flush, response code: 200 dump: 
2021/12/01 21:11:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 21:11:05 Context: Bucket flush, response code: 200 dump: 
2021/12/01 21:11:10 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 21:11:10 Deploying app: TestOnUpdateBucketOpDefaultSettings10K. Item count src bucket: 0 dst bucket: 0
2021/12/01 21:11:11 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateBucketOpDefaultSettings10K' in metakv",
  "warnings": null
 }
}
2021/12/01 21:11:11 Starting bucket ops to source bucket
2021/12/01 21:11:19 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 21:11:24 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 21:11:29 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 21:11:34 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 21:11:34 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/01 21:11:34 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 21:11:34 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 21:11:39 Waiting for app: TestOnUpdateBucketOpDefaultSettings10K to get un-deployed
2021/12/01 21:11:45 Waiting for app: TestOnUpdateBucketOpDefaultSettings10K to get un-deployed
2021/12/01 21:11:50 Waiting for app: TestOnUpdateBucketOpDefaultSettings10K to get un-deployed
2021/12/01 21:11:55 Waiting for app: TestOnUpdateBucketOpDefaultSettings10K to get un-deployed
2021/12/01 21:11:55 App: TestOnUpdateBucketOpDefaultSettings10K got un-deployed
2021/12/01 21:11:55 No eventing-con process running
2021/12/01 21:11:55 Function: TestOnUpdateBucketOpDefaultSettings10K delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateBucketOpDefaultSettings10K deleting in the background"
}
2021/12/01 21:11:56 Context: Bucket flush, response code: 200 dump: 
2021/12/01 21:12:06 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/01 21:12:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 21:12:08 Context: Bucket flush, response code: 200 dump: 
2021/12/01 21:12:18 Waiting for dst bucket item count to get to: 0 curr count: 4014
2021/12/01 21:12:23 Waiting for dst bucket item count to get to: 0 curr count: 1514
2021/12/01 21:12:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateBucketOpDefaultSettings10K (96.76s)
=== RUN   TestOnUpdateBucketOpDefaultSettings100K
2021/12/01 21:12:33 Waiting for app: TestOnUpdateBucketOpDefaultSettings100K to get un-deployed
2021/12/01 21:12:33 App: TestOnUpdateBucketOpDefaultSettings100K got un-deployed
2021/12/01 21:12:33 No eventing-con process running
2021/12/01 21:12:33 Function: TestOnUpdateBucketOpDefaultSettings100K delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateBucketOpDefaultSettings100K not found"
 }
}
2021/12/01 21:12:35 Context: Bucket flush, response code: 200 dump: 
2021/12/01 21:12:40 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 21:12:42 Context: Bucket flush, response code: 200 dump: 
2021/12/01 21:12:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 21:12:47 Deploying app: TestOnUpdateBucketOpDefaultSettings100K. Item count src bucket: 0 dst bucket: 0
2021/12/01 21:12:49 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateBucketOpDefaultSettings100K' in metakv",
  "warnings": null
 }
}
2021/12/01 21:12:49 Starting bucket ops to source bucket
2021/12/01 21:12:56 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 21:13:01 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 21:13:07 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 21:13:12 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 21:13:17 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 21:13:17 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/01 21:13:17 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 21:13:17 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 21:13:22 Waiting for app: TestOnUpdateBucketOpDefaultSettings100K to get un-deployed
2021/12/01 21:13:27 Waiting for app: TestOnUpdateBucketOpDefaultSettings100K to get un-deployed
2021/12/01 21:13:32 Waiting for app: TestOnUpdateBucketOpDefaultSettings100K to get un-deployed
2021/12/01 21:13:37 Waiting for app: TestOnUpdateBucketOpDefaultSettings100K to get un-deployed
2021/12/01 21:13:37 App: TestOnUpdateBucketOpDefaultSettings100K got un-deployed
2021/12/01 21:13:37 No eventing-con process running
2021/12/01 21:13:37 Function: TestOnUpdateBucketOpDefaultSettings100K delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateBucketOpDefaultSettings100K deleting in the background"
}
2021/12/01 21:13:39 Context: Bucket flush, response code: 200 dump: 
2021/12/01 21:13:49 Waiting for dst bucket item count to get to: 0 curr count: 4496
2021/12/01 21:13:54 Waiting for dst bucket item count to get to: 0 curr count: 1996
2021/12/01 21:13:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 21:13:56 Context: Bucket flush, response code: 200 dump: 
2021/12/01 21:14:06 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/01 21:14:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateBucketOpDefaultSettings100K (102.33s)
=== RUN   TestOnDeleteBucketOp
2021/12/01 21:14:16 Waiting for app: TestOnDeleteBucketOp to get un-deployed
2021/12/01 21:14:16 App: TestOnDeleteBucketOp got un-deployed
2021/12/01 21:14:16 No eventing-con process running
2021/12/01 21:14:16 Function: TestOnDeleteBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnDeleteBucketOp not found"
 }
}
2021/12/01 21:14:17 Context: Bucket flush, response code: 200 dump: 
2021/12/01 21:14:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 21:14:24 Context: Bucket flush, response code: 200 dump: 
2021/12/01 21:14:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 21:14:29 Deploying app: TestOnDeleteBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/01 21:14:30 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnDeleteBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/01 21:14:30 Starting bucket ops to source bucket
2021/12/01 21:14:41 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 21:14:46 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 21:14:51 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 21:14:51 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/01 21:14:51 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 21:14:51 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 21:14:56 Waiting for app: TestOnDeleteBucketOp to get un-deployed
2021/12/01 21:15:02 Waiting for app: TestOnDeleteBucketOp to get un-deployed
2021/12/01 21:15:07 Waiting for app: TestOnDeleteBucketOp to get un-deployed
2021/12/01 21:15:12 Waiting for app: TestOnDeleteBucketOp to get un-deployed
2021/12/01 21:15:12 App: TestOnDeleteBucketOp got un-deployed
2021/12/01 21:15:12 No eventing-con process running
2021/12/01 21:15:12 Function: TestOnDeleteBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnDeleteBucketOp deleting in the background"
}
2021/12/01 21:15:13 Context: Bucket flush, response code: 200 dump: 
2021/12/01 21:15:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 21:15:21 Context: Bucket flush, response code: 200 dump: 
2021/12/01 21:15:31 Waiting for dst bucket item count to get to: 0 curr count: 3849
2021/12/01 21:15:36 Waiting for dst bucket item count to get to: 0 curr count: 3849
2021/12/01 21:15:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnDeleteBucketOp (90.18s)
=== RUN   TestOnDeleteBucketOp5K
2021/12/01 21:15:46 Waiting for app: TestOnDeleteBucketOp5K to get un-deployed
2021/12/01 21:15:46 App: TestOnDeleteBucketOp5K got un-deployed
2021/12/01 21:15:46 No eventing-con process running
2021/12/01 21:15:46 Function: TestOnDeleteBucketOp5K delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnDeleteBucketOp5K not found"
 }
}
2021/12/01 21:15:48 Context: Bucket flush, response code: 200 dump: 
2021/12/01 21:15:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 21:15:54 Context: Bucket flush, response code: 200 dump: 
2021/12/01 21:15:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 21:15:59 Deploying app: TestOnDeleteBucketOp5K. Item count src bucket: 0 dst bucket: 0
2021/12/01 21:16:01 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnDeleteBucketOp5K' in metakv",
  "warnings": null
 }
}
2021/12/01 21:16:01 Starting bucket ops to source bucket
2021/12/01 21:16:11 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 21:16:16 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 21:16:21 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 21:16:21 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/01 21:16:21 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 21:16:22 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 21:16:27 Waiting for app: TestOnDeleteBucketOp5K to get un-deployed
2021/12/01 21:16:32 Waiting for app: TestOnDeleteBucketOp5K to get un-deployed
2021/12/01 21:16:37 Waiting for app: TestOnDeleteBucketOp5K to get un-deployed
2021/12/01 21:16:42 Waiting for app: TestOnDeleteBucketOp5K to get un-deployed
2021/12/01 21:16:42 App: TestOnDeleteBucketOp5K got un-deployed
2021/12/01 21:16:42 No eventing-con process running
2021/12/01 21:16:42 Function: TestOnDeleteBucketOp5K delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnDeleteBucketOp5K deleting in the background"
}
2021/12/01 21:16:43 Context: Bucket flush, response code: 200 dump: 
2021/12/01 21:16:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 21:16:51 Context: Bucket flush, response code: 200 dump: 
2021/12/01 21:17:01 Waiting for dst bucket item count to get to: 0 curr count: 4145
2021/12/01 21:17:06 Waiting for dst bucket item count to get to: 0 curr count: 4145
2021/12/01 21:17:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnDeleteBucketOp5K (90.11s)
=== RUN   TestTimerBucketOp
2021/12/01 21:17:16 Waiting for app: TestTimerBucketOp to get un-deployed
2021/12/01 21:17:16 App: TestTimerBucketOp got un-deployed
2021/12/01 21:17:16 No eventing-con process running
2021/12/01 21:17:16 Function: TestTimerBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestTimerBucketOp not found"
 }
}
2021/12/01 21:17:18 Context: Bucket flush, response code: 200 dump: 
2021/12/01 21:17:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 21:17:24 Context: Bucket flush, response code: 200 dump: 
2021/12/01 21:17:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 21:17:30 Deploying app: TestTimerBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/01 21:17:31 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestTimerBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/01 21:17:31 Starting bucket ops to source bucket
2021/12/01 21:17:39 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 21:17:44 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 21:17:49 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 21:17:54 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 21:17:59 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 21:18:04 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 21:18:09 Waiting for dst bucket item count to get to: 5000 curr count: 2402
2021/12/01 21:18:14 Waiting for dst bucket item count to get to: 5000 curr count: 2402
2021/12/01 21:18:14 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/01 21:18:14 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 21:18:14 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 21:18:19 Waiting for app: TestTimerBucketOp to get un-deployed
2021/12/01 21:18:24 Waiting for app: TestTimerBucketOp to get un-deployed
2021/12/01 21:18:29 Waiting for app: TestTimerBucketOp to get un-deployed
2021/12/01 21:18:34 Waiting for app: TestTimerBucketOp to get un-deployed
2021/12/01 21:18:34 App: TestTimerBucketOp got un-deployed
2021/12/01 21:18:34 No eventing-con process running
2021/12/01 21:18:34 Function: TestTimerBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestTimerBucketOp deleting in the background"
}
2021/12/01 21:18:36 Context: Bucket flush, response code: 200 dump: 
2021/12/01 21:18:46 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/01 21:18:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 21:18:48 Context: Bucket flush, response code: 200 dump: 
2021/12/01 21:18:58 Waiting for dst bucket item count to get to: 0 curr count: 2569
2021/12/01 21:19:03 Waiting for dst bucket item count to get to: 0 curr count: 69
2021/12/01 21:19:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestTimerBucketOp (116.79s)
=== RUN   TestTimerInPastBucketOp
2021/12/01 21:19:13 Waiting for app: TestTimerInPastBucketOp to get un-deployed
2021/12/01 21:19:13 App: TestTimerInPastBucketOp got un-deployed
2021/12/01 21:19:13 No eventing-con process running
2021/12/01 21:19:13 Function: TestTimerInPastBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestTimerInPastBucketOp not found"
 }
}
2021/12/01 21:19:14 Context: Bucket flush, response code: 200 dump: 
2021/12/01 21:19:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 21:19:21 Context: Bucket flush, response code: 200 dump: 
2021/12/01 21:19:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 21:19:26 Deploying app: TestTimerInPastBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/01 21:19:27 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestTimerInPastBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/01 21:19:27 Starting bucket ops to source bucket
2021/12/01 21:19:35 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 21:19:40 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 21:19:45 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 21:19:50 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 21:19:55 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 21:20:00 Waiting for dst bucket item count to get to: 5000 curr count: 329
2021/12/01 21:20:05 Waiting for dst bucket item count to get to: 5000 curr count: 329
2021/12/01 21:20:05 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/01 21:20:05 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 21:20:06 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 21:20:11 Waiting for app: TestTimerInPastBucketOp to get un-deployed
2021/12/01 21:20:16 Waiting for app: TestTimerInPastBucketOp to get un-deployed
2021/12/01 21:20:21 Waiting for app: TestTimerInPastBucketOp to get un-deployed
2021/12/01 21:20:26 Waiting for app: TestTimerInPastBucketOp to get un-deployed
2021/12/01 21:20:26 App: TestTimerInPastBucketOp got un-deployed
2021/12/01 21:20:26 No eventing-con process running
2021/12/01 21:20:26 Function: TestTimerInPastBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestTimerInPastBucketOp deleting in the background"
}
2021/12/01 21:20:27 Context: Bucket flush, response code: 200 dump: 
2021/12/01 21:20:37 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/01 21:20:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 21:20:39 Context: Bucket flush, response code: 200 dump: 
2021/12/01 21:20:49 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/01 21:20:54 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/01 21:20:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestTimerInPastBucketOp (111.56s)
=== RUN   TestDeployUndeployLoopTimer
2021/12/01 21:21:04 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/01 21:21:04 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/01 21:21:04 No eventing-con process running
2021/12/01 21:21:04 Function: TestDeployUndeployLoopTimer delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDeployUndeployLoopTimer not found"
 }
}
2021/12/01 21:21:06 Context: Bucket flush, response code: 200 dump: 
2021/12/01 21:21:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 21:21:13 Context: Bucket flush, response code: 200 dump: 
2021/12/01 21:21:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 21:21:18 Deploying app: TestDeployUndeployLoopTimer. Item count src bucket: 0 dst bucket: 0
2021/12/01 21:21:19 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopTimer' in metakv",
  "warnings": null
 }
}
2021/12/01 21:21:19 Starting bucket ops to source bucket
2021/12/01 21:21:27 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 21:21:32 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 21:21:37 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 21:21:42 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 21:21:47 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 21:21:52 Waiting for dst bucket item count to get to: 5000 curr count: 223
2021/12/01 21:21:57 Waiting for dst bucket item count to get to: 5000 curr count: 2366
2021/12/01 21:22:02 Waiting for dst bucket item count to get to: 5000 curr count: 4643
2021/12/01 21:22:02 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/01 21:22:07 Waiting for app: TestDeployUndeployLoopTimer to get deployed
2021/12/01 21:22:07 App: TestDeployUndeployLoopTimer got deployed
2021/12/01 21:22:07 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 21:22:07 Undeploying app: bucket_op_with_timer
2021/12/01 21:22:07 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 21:22:12 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/01 21:22:17 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/01 21:22:22 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/01 21:22:27 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/01 21:22:27 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/01 21:22:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 21:22:27 No eventing-con process running
2021/12/01 21:22:29 Context: Bucket flush, response code: 200 dump: 
2021/12/01 21:22:37 Context: Bucket flush, response code: 200 dump: 
2021/12/01 21:23:12 Deploying app: TestDeployUndeployLoopTimer. Item count src bucket: 0 dst bucket: 0
2021/12/01 21:23:13 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopTimer' in metakv",
  "warnings": null
 }
}
2021/12/01 21:23:13 Starting bucket ops to source bucket
2021/12/01 21:23:21 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 21:23:26 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 21:23:31 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 21:23:36 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 21:23:41 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 21:23:46 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 21:23:51 Waiting for dst bucket item count to get to: 5000 curr count: 271
2021/12/01 21:23:56 Waiting for dst bucket item count to get to: 5000 curr count: 2771
2021/12/01 21:23:56 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/01 21:24:01 Waiting for app: TestDeployUndeployLoopTimer to get deployed
2021/12/01 21:24:01 App: TestDeployUndeployLoopTimer got deployed
2021/12/01 21:24:01 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 21:24:01 Undeploying app: bucket_op_with_timer
2021/12/01 21:24:01 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 21:24:06 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/01 21:24:11 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/01 21:24:16 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/01 21:24:21 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/01 21:24:21 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/01 21:24:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 21:24:21 No eventing-con process running
2021/12/01 21:24:23 Context: Bucket flush, response code: 200 dump: 
2021/12/01 21:24:30 Context: Bucket flush, response code: 200 dump: 
2021/12/01 21:25:05 Deploying app: TestDeployUndeployLoopTimer. Item count src bucket: 0 dst bucket: 0
2021/12/01 21:25:06 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopTimer' in metakv",
  "warnings": null
 }
}
2021/12/01 21:25:06 Starting bucket ops to source bucket
2021/12/01 21:25:14 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 21:25:19 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 21:25:24 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 21:25:29 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 21:25:34 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 21:25:39 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 21:25:44 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 21:25:44 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/01 21:25:49 Waiting for app: TestDeployUndeployLoopTimer to get deployed
2021/12/01 21:25:49 App: TestDeployUndeployLoopTimer got deployed
2021/12/01 21:25:49 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 21:25:49 Undeploying app: bucket_op_with_timer
2021/12/01 21:25:49 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 21:25:54 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/01 21:25:59 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/01 21:26:04 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/01 21:26:09 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/01 21:26:09 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/01 21:26:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 21:26:09 No eventing-con process running
2021/12/01 21:26:11 Context: Bucket flush, response code: 200 dump: 
2021/12/01 21:26:18 Context: Bucket flush, response code: 200 dump: 
2021/12/01 21:26:53 Deploying app: TestDeployUndeployLoopTimer. Item count src bucket: 0 dst bucket: 0
2021/12/01 21:26:54 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopTimer' in metakv",
  "warnings": null
 }
}
2021/12/01 21:26:54 Starting bucket ops to source bucket
2021/12/01 21:27:02 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 21:27:07 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 21:27:12 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 21:27:17 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 21:27:22 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 21:27:27 Waiting for dst bucket item count to get to: 5000 curr count: 1125
2021/12/01 21:27:33 Waiting for dst bucket item count to get to: 5000 curr count: 2302
2021/12/01 21:27:38 Waiting for dst bucket item count to get to: 5000 curr count: 3677
2021/12/01 21:27:38 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/01 21:27:43 Waiting for app: TestDeployUndeployLoopTimer to get deployed
2021/12/01 21:27:43 App: TestDeployUndeployLoopTimer got deployed
2021/12/01 21:27:43 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 21:27:43 Undeploying app: bucket_op_with_timer
2021/12/01 21:27:43 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 21:27:48 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/01 21:27:53 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/01 21:27:58 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/01 21:28:03 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/01 21:28:03 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/01 21:28:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 21:28:03 No eventing-con process running
2021/12/01 21:28:04 Context: Bucket flush, response code: 200 dump: 
2021/12/01 21:28:12 Context: Bucket flush, response code: 200 dump: 
2021/12/01 21:28:47 Deploying app: TestDeployUndeployLoopTimer. Item count src bucket: 0 dst bucket: 0
2021/12/01 21:28:48 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopTimer' in metakv",
  "warnings": null
 }
}
2021/12/01 21:28:48 Starting bucket ops to source bucket
2021/12/01 21:28:56 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 21:29:01 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 21:29:06 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 21:29:11 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 21:29:16 Waiting for dst bucket item count to get to: 5000 curr count: 2
2021/12/01 21:29:21 Waiting for dst bucket item count to get to: 5000 curr count: 113
2021/12/01 21:29:26 Waiting for dst bucket item count to get to: 5000 curr count: 2610
2021/12/01 21:29:31 Waiting for dst bucket item count to get to: 5000 curr count: 4999
2021/12/01 21:29:31 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/01 21:29:36 Waiting for app: TestDeployUndeployLoopTimer to get deployed
2021/12/01 21:29:36 App: TestDeployUndeployLoopTimer got deployed
2021/12/01 21:29:36 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 21:29:36 Undeploying app: bucket_op_with_timer
2021/12/01 21:29:36 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 21:29:41 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/01 21:29:46 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/01 21:29:51 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/01 21:29:56 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/01 21:29:56 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/01 21:29:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 21:29:56 No eventing-con process running
2021/12/01 21:29:58 Context: Bucket flush, response code: 200 dump: 
2021/12/01 21:30:05 Context: Bucket flush, response code: 200 dump: 
2021/12/01 21:30:40 Function: TestDeployUndeployLoopTimer delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDeployUndeployLoopTimer deleting in the background"
}
--- PASS: TestDeployUndeployLoopTimer (586.10s)
=== RUN   TestMultipleHandlers
2021/12/01 21:30:50 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get un-deployed
2021/12/01 21:30:50 App: TestMultipleHandlers_bucket_op_on_update got un-deployed
2021/12/01 21:30:50 No eventing-con process running
2021/12/01 21:30:50 Function: TestMultipleHandlers_bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMultipleHandlers_bucket_op_on_update not found"
 }
}
2021/12/01 21:30:52 Context: Bucket flush, response code: 200 dump: 
2021/12/01 21:30:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 21:30:59 Context: Bucket flush, response code: 200 dump: 
2021/12/01 21:31:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 21:31:09 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get un-deployed
2021/12/01 21:31:09 App: TestMultipleHandlers_bucket_op_on_delete got un-deployed
2021/12/01 21:31:09 No eventing-con process running
2021/12/01 21:31:09 Function: TestMultipleHandlers_bucket_op_on_delete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMultipleHandlers_bucket_op_on_delete not found"
 }
}
2021/12/01 21:31:11 Context: Bucket flush, response code: 200 dump: 
2021/12/01 21:31:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 21:31:18 Context: Bucket flush, response code: 200 dump: 
2021/12/01 21:31:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 21:31:23 Deploying app: TestMultipleHandlers_bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/01 21:31:24 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMultipleHandlers_bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/01 21:31:24 Deploying app: TestMultipleHandlers_bucket_op_on_delete. Item count src bucket: 0 dst bucket: 0
2021/12/01 21:31:25 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMultipleHandlers_bucket_op_on_delete' in metakv",
  "warnings": null
 }
}
2021/12/01 21:31:30 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get deployed
2021/12/01 21:31:35 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get deployed
2021/12/01 21:31:40 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get deployed
2021/12/01 21:31:45 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get deployed
2021/12/01 21:31:45 App: TestMultipleHandlers_bucket_op_on_update got deployed
2021/12/01 21:31:50 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get deployed
2021/12/01 21:31:55 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get deployed
2021/12/01 21:32:00 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get deployed
2021/12/01 21:32:00 App: TestMultipleHandlers_bucket_op_on_delete got deployed
2021/12/01 21:32:00 Starting bucket ops to source bucket
2021/12/01 21:32:10 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 21:32:15 Waiting for dst bucket item count to get to: 5000 curr count: 2500
2021/12/01 21:32:15 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/01 21:32:15 Starting bucket ops to source bucket
2021/12/01 21:32:30 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/01 21:32:35 Waiting for dst bucket item count to get to: 10000 curr count: 7504
2021/12/01 21:32:35 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/01 21:32:35 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 21:32:35 Function: TestMultipleHandlers_bucket_op_on_update update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/01 21:32:35 Function: TestMultipleHandlers_bucket_op_on_delete update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/01 21:32:35 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 21:32:40 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get un-deployed
2021/12/01 21:32:45 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get un-deployed
2021/12/01 21:32:50 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get un-deployed
2021/12/01 21:32:55 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get un-deployed
2021/12/01 21:32:55 App: TestMultipleHandlers_bucket_op_on_update got un-deployed
2021/12/01 21:32:55 No eventing-con process running
2021/12/01 21:32:56 Function: TestMultipleHandlers_bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMultipleHandlers_bucket_op_on_update deleting in the background"
}
2021/12/01 21:32:57 Context: Bucket flush, response code: 200 dump: 
2021/12/01 21:33:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 21:33:05 Context: Bucket flush, response code: 200 dump: 
2021/12/01 21:33:15 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/01 21:33:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 21:33:15 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 21:33:20 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get un-deployed
2021/12/01 21:33:25 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get un-deployed
2021/12/01 21:33:30 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get un-deployed
2021/12/01 21:33:35 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get un-deployed
2021/12/01 21:33:35 App: TestMultipleHandlers_bucket_op_on_delete got un-deployed
2021/12/01 21:33:35 No eventing-con process running
2021/12/01 21:33:35 Function: TestMultipleHandlers_bucket_op_on_delete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMultipleHandlers_bucket_op_on_delete deleting in the background"
}
2021/12/01 21:33:37 Context: Bucket flush, response code: 200 dump: 
2021/12/01 21:33:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 21:33:43 Context: Bucket flush, response code: 200 dump: 
2021/12/01 21:33:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestMultipleHandlers (188.15s)
=== RUN   TestPauseResumeLoopDefaultSettings
2021/12/01 21:33:58 Waiting for app: TestPauseResumeLoopDefaultSettings to get un-deployed
2021/12/01 21:33:58 App: TestPauseResumeLoopDefaultSettings got un-deployed
2021/12/01 21:33:58 No eventing-con process running
2021/12/01 21:33:58 Function: TestPauseResumeLoopDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestPauseResumeLoopDefaultSettings not found"
 }
}
2021/12/01 21:34:00 Context: Bucket flush, response code: 200 dump: 
2021/12/01 21:34:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 21:34:07 Context: Bucket flush, response code: 200 dump: 
2021/12/01 21:34:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 21:34:12 Deploying app: TestPauseResumeLoopDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/01 21:34:13 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestPauseResumeLoopDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/01 21:34:13 Starting bucket ops to source bucket
2021/12/01 21:34:21 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 21:34:26 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 21:34:31 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 21:34:36 Waiting for dst bucket item count to get to: 5000 curr count: 2500
2021/12/01 21:34:36 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/01 21:34:36 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 21:34:41 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to deployed
2021/12/01 21:34:41 Function: TestPauseResumeLoopDefaultSettings status changed to deployed
2021/12/01 21:34:41 Pausing app: bucket_op_on_update
2021/12/01 21:34:41 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/01 21:34:46 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to paused
2021/12/01 21:34:46 Function: TestPauseResumeLoopDefaultSettings status changed to paused
2021/12/01 21:34:46 Resuming app: bucket_op_on_update
2021/12/01 21:34:48 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/01 21:34:48 Starting bucket ops to source bucket
2021/12/01 21:34:56 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/01 21:35:01 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/01 21:35:06 Waiting for dst bucket item count to get to: 10000 curr count: 7500
2021/12/01 21:35:06 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/01 21:35:06 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 21:35:11 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to deployed
2021/12/01 21:35:11 Function: TestPauseResumeLoopDefaultSettings status changed to deployed
2021/12/01 21:35:11 Pausing app: bucket_op_on_update
2021/12/01 21:35:11 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/01 21:35:16 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to paused
2021/12/01 21:35:16 Function: TestPauseResumeLoopDefaultSettings status changed to paused
2021/12/01 21:35:16 Resuming app: bucket_op_on_update
2021/12/01 21:35:17 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/01 21:35:17 Starting bucket ops to source bucket
2021/12/01 21:35:25 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/01 21:35:30 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/01 21:35:35 Waiting for dst bucket item count to get to: 15000 curr count: 12500
2021/12/01 21:35:35 src & dst bucket item count matched up. src bucket count: 15000 dst bucket count: 15000
2021/12/01 21:35:35 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 21:35:40 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to deployed
2021/12/01 21:35:40 Function: TestPauseResumeLoopDefaultSettings status changed to deployed
2021/12/01 21:35:40 Pausing app: bucket_op_on_update
2021/12/01 21:35:40 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/01 21:35:45 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to paused
2021/12/01 21:35:45 Function: TestPauseResumeLoopDefaultSettings status changed to paused
2021/12/01 21:35:45 Resuming app: bucket_op_on_update
2021/12/01 21:35:46 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/01 21:35:46 Starting bucket ops to source bucket
2021/12/01 21:35:54 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/01 21:35:59 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/01 21:36:04 Waiting for dst bucket item count to get to: 20000 curr count: 17500
2021/12/01 21:36:04 src & dst bucket item count matched up. src bucket count: 20000 dst bucket count: 20000
2021/12/01 21:36:04 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 21:36:09 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to deployed
2021/12/01 21:36:09 Function: TestPauseResumeLoopDefaultSettings status changed to deployed
2021/12/01 21:36:09 Pausing app: bucket_op_on_update
2021/12/01 21:36:10 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/01 21:36:15 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to paused
2021/12/01 21:36:15 Function: TestPauseResumeLoopDefaultSettings status changed to paused
2021/12/01 21:36:15 Resuming app: bucket_op_on_update
2021/12/01 21:36:16 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/01 21:36:16 Starting bucket ops to source bucket
2021/12/01 21:36:24 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/01 21:36:29 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/01 21:36:34 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/01 21:36:39 Waiting for dst bucket item count to get to: 25000 curr count: 22500
2021/12/01 21:36:39 src & dst bucket item count matched up. src bucket count: 25000 dst bucket count: 25000
2021/12/01 21:36:39 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 21:36:44 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to deployed
2021/12/01 21:36:44 Function: TestPauseResumeLoopDefaultSettings status changed to deployed
2021/12/01 21:36:44 Pausing app: bucket_op_on_update
2021/12/01 21:36:44 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/01 21:36:49 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to paused
2021/12/01 21:36:49 Function: TestPauseResumeLoopDefaultSettings status changed to paused
2021/12/01 21:36:49 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 21:36:54 Waiting for app: TestPauseResumeLoopDefaultSettings to get un-deployed
2021/12/01 21:36:59 Waiting for app: TestPauseResumeLoopDefaultSettings to get un-deployed
2021/12/01 21:37:04 Waiting for app: TestPauseResumeLoopDefaultSettings to get un-deployed
2021/12/01 21:37:09 Waiting for app: TestPauseResumeLoopDefaultSettings to get un-deployed
2021/12/01 21:37:09 App: TestPauseResumeLoopDefaultSettings got un-deployed
2021/12/01 21:37:09 No eventing-con process running
2021/12/01 21:37:09 Function: TestPauseResumeLoopDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestPauseResumeLoopDefaultSettings deleting in the background"
}
2021/12/01 21:37:11 Context: Bucket flush, response code: 200 dump: 
2021/12/01 21:37:21 Waiting for dst bucket item count to get to: 0 curr count: 23055
2021/12/01 21:37:26 Waiting for dst bucket item count to get to: 0 curr count: 23055
2021/12/01 21:37:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 21:37:28 Context: Bucket flush, response code: 200 dump: 
2021/12/01 21:37:38 Waiting for dst bucket item count to get to: 0 curr count: 15506
2021/12/01 21:37:43 Waiting for dst bucket item count to get to: 0 curr count: 3006
2021/12/01 21:37:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestPauseResumeLoopDefaultSettings (234.36s)
=== RUN   TestPauseResumeLoopNonDefaultSettings
2021/12/01 21:37:53 Waiting for app: TestPauseResumeLoopNonDefaultSettings to get un-deployed
2021/12/01 21:37:53 App: TestPauseResumeLoopNonDefaultSettings got un-deployed
2021/12/01 21:37:53 No eventing-con process running
2021/12/01 21:37:53 Function: TestPauseResumeLoopNonDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestPauseResumeLoopNonDefaultSettings not found"
 }
}
2021/12/01 21:37:55 Context: Bucket flush, response code: 200 dump: 
2021/12/01 21:38:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 21:38:02 Context: Bucket flush, response code: 200 dump: 
2021/12/01 21:38:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 21:38:07 Deploying app: TestPauseResumeLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/01 21:38:08 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestPauseResumeLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/01 21:38:08 Starting bucket ops to source bucket
2021/12/01 21:38:16 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 21:38:21 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 21:38:26 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 21:38:31 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 21:38:36 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 21:38:41 Waiting for dst bucket item count to get to: 5000 curr count: 56
2021/12/01 21:38:46 Waiting for dst bucket item count to get to: 5000 curr count: 56
2021/12/01 21:38:46 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/01 21:38:46 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 21:38:51 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to deployed
2021/12/01 21:38:51 Function: TestPauseResumeLoopNonDefaultSettings status changed to deployed
2021/12/01 21:38:51 Pausing app: bucket_op_with_timer
2021/12/01 21:38:51 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/01 21:38:56 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to paused
2021/12/01 21:38:56 Function: TestPauseResumeLoopNonDefaultSettings status changed to paused
2021/12/01 21:38:56 Resuming app: bucket_op_with_timer
2021/12/01 21:38:57 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/01 21:38:57 Starting bucket ops to source bucket
2021/12/01 21:39:05 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/01 21:39:10 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/01 21:39:15 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/01 21:39:20 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/01 21:39:25 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/01 21:39:30 Waiting for dst bucket item count to get to: 10000 curr count: 6515
2021/12/01 21:39:35 Waiting for dst bucket item count to get to: 10000 curr count: 9015
2021/12/01 21:39:35 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/01 21:39:35 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 21:39:40 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to deployed
2021/12/01 21:39:40 Function: TestPauseResumeLoopNonDefaultSettings status changed to deployed
2021/12/01 21:39:40 Pausing app: bucket_op_with_timer
2021/12/01 21:39:40 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/01 21:39:45 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to paused
2021/12/01 21:39:45 Function: TestPauseResumeLoopNonDefaultSettings status changed to paused
2021/12/01 21:39:45 Resuming app: bucket_op_with_timer
2021/12/01 21:39:46 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/01 21:39:46 Starting bucket ops to source bucket
2021/12/01 21:39:54 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/01 21:39:59 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/01 21:40:04 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/01 21:40:09 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/01 21:40:14 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/01 21:40:20 Waiting for dst bucket item count to get to: 15000 curr count: 12322
2021/12/01 21:40:25 Waiting for dst bucket item count to get to: 15000 curr count: 14822
2021/12/01 21:40:25 src & dst bucket item count matched up. src bucket count: 15000 dst bucket count: 15000
2021/12/01 21:40:25 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 21:40:30 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to deployed
2021/12/01 21:40:30 Function: TestPauseResumeLoopNonDefaultSettings status changed to deployed
2021/12/01 21:40:30 Pausing app: bucket_op_with_timer
2021/12/01 21:40:30 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/01 21:40:35 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to paused
2021/12/01 21:40:35 Function: TestPauseResumeLoopNonDefaultSettings status changed to paused
2021/12/01 21:40:35 Resuming app: bucket_op_with_timer
2021/12/01 21:40:36 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/01 21:40:36 Starting bucket ops to source bucket
2021/12/01 21:40:44 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/01 21:40:49 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/01 21:40:54 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/01 21:40:59 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/01 21:41:04 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/01 21:41:09 Waiting for dst bucket item count to get to: 20000 curr count: 15153
2021/12/01 21:41:14 Waiting for dst bucket item count to get to: 20000 curr count: 17653
2021/12/01 21:41:14 src & dst bucket item count matched up. src bucket count: 20000 dst bucket count: 20000
2021/12/01 21:41:14 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 21:41:19 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to deployed
2021/12/01 21:41:19 Function: TestPauseResumeLoopNonDefaultSettings status changed to deployed
2021/12/01 21:41:19 Pausing app: bucket_op_with_timer
2021/12/01 21:41:19 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/01 21:41:24 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to paused
2021/12/01 21:41:24 Function: TestPauseResumeLoopNonDefaultSettings status changed to paused
2021/12/01 21:41:24 Resuming app: bucket_op_with_timer
2021/12/01 21:41:25 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/01 21:41:25 Starting bucket ops to source bucket
2021/12/01 21:41:33 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/01 21:41:38 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/01 21:41:43 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/01 21:41:48 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/01 21:41:53 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/01 21:41:58 Waiting for dst bucket item count to get to: 25000 curr count: 23572
2021/12/01 21:42:03 Waiting for dst bucket item count to get to: 25000 curr count: 23572
2021/12/01 21:42:03 src & dst bucket item count matched up. src bucket count: 25000 dst bucket count: 25000
2021/12/01 21:42:03 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 21:42:08 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to deployed
2021/12/01 21:42:08 Function: TestPauseResumeLoopNonDefaultSettings status changed to deployed
2021/12/01 21:42:08 Pausing app: bucket_op_with_timer
2021/12/01 21:42:08 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/01 21:42:13 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to paused
2021/12/01 21:42:13 Function: TestPauseResumeLoopNonDefaultSettings status changed to paused
2021/12/01 21:42:13 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 21:42:18 Waiting for app: TestPauseResumeLoopNonDefaultSettings to get un-deployed
2021/12/01 21:42:23 Waiting for app: TestPauseResumeLoopNonDefaultSettings to get un-deployed
2021/12/01 21:42:28 Waiting for app: TestPauseResumeLoopNonDefaultSettings to get un-deployed
2021/12/01 21:42:33 Waiting for app: TestPauseResumeLoopNonDefaultSettings to get un-deployed
2021/12/01 21:42:33 App: TestPauseResumeLoopNonDefaultSettings got un-deployed
2021/12/01 21:42:33 No eventing-con process running
2021/12/01 21:42:33 Function: TestPauseResumeLoopNonDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestPauseResumeLoopNonDefaultSettings deleting in the background"
}
2021/12/01 21:42:35 Context: Bucket flush, response code: 200 dump: 
2021/12/01 21:42:45 Waiting for dst bucket item count to get to: 0 curr count: 25000
2021/12/01 21:42:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 21:42:47 Context: Bucket flush, response code: 200 dump: 
2021/12/01 21:42:57 Waiting for dst bucket item count to get to: 0 curr count: 25000
2021/12/01 21:42:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestPauseResumeLoopNonDefaultSettings (314.25s)
=== RUN   TestPauseAndResumeWithWorkerCountChange
2021/12/01 21:43:07 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get un-deployed
2021/12/01 21:43:07 App: TestPauseAndResumeWithWorkerCountChange got un-deployed
2021/12/01 21:43:07 No eventing-con process running
2021/12/01 21:43:07 Function: TestPauseAndResumeWithWorkerCountChange delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestPauseAndResumeWithWorkerCountChange not found"
 }
}
2021/12/01 21:43:09 Context: Bucket flush, response code: 200 dump: 
2021/12/01 21:43:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 21:43:16 Context: Bucket flush, response code: 200 dump: 
2021/12/01 21:43:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 21:43:21 Deploying app: TestPauseAndResumeWithWorkerCountChange. Item count src bucket: 0 dst bucket: 0
2021/12/01 21:43:22 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestPauseAndResumeWithWorkerCountChange' in metakv",
  "warnings": null
 }
}
2021/12/01 21:43:27 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get deployed
2021/12/01 21:43:32 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get deployed
2021/12/01 21:43:37 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get deployed
2021/12/01 21:43:42 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get deployed
2021/12/01 21:43:42 App: TestPauseAndResumeWithWorkerCountChange got deployed
2021/12/01 21:43:42 Starting bucket ops to source bucket
2021/12/01 21:43:51 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 21:43:56 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 21:44:01 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 21:44:06 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 21:44:11 Waiting for dst bucket item count to get to: 5000 curr count: 4450
2021/12/01 21:44:16 Waiting for dst bucket item count to get to: 5000 curr count: 4450
2021/12/01 21:44:16 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/01 21:44:16 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 21:44:16 Pausing app: bucket_op_with_timer
2021/12/01 21:44:16 Function: TestPauseAndResumeWithWorkerCountChange update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/01 21:44:21 Waiting for function: TestPauseAndResumeWithWorkerCountChange status to change to paused
2021/12/01 21:44:21 Function: TestPauseAndResumeWithWorkerCountChange status changed to paused
2021/12/01 21:44:21 Resuming app: bucket_op_with_timer
2021/12/01 21:44:22 Function: TestPauseAndResumeWithWorkerCountChange update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:6] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/01 21:44:22 Starting bucket ops to source bucket
2021/12/01 21:44:35 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/01 21:44:40 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/01 21:44:45 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/01 21:44:50 Waiting for dst bucket item count to get to: 10000 curr count: 5287
2021/12/01 21:44:55 Waiting for dst bucket item count to get to: 10000 curr count: 5287
2021/12/01 21:45:00 Waiting for dst bucket item count to get to: 10000 curr count: 9728
2021/12/01 21:45:05 Waiting for dst bucket item count to get to: 10000 curr count: 9728
2021/12/01 21:45:05 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/01 21:45:05 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 21:45:10 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get un-deployed
2021/12/01 21:45:15 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get un-deployed
2021/12/01 21:45:20 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get un-deployed
2021/12/01 21:45:25 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get un-deployed
2021/12/01 21:45:25 App: TestPauseAndResumeWithWorkerCountChange got un-deployed
2021/12/01 21:45:25 No eventing-con process running
2021/12/01 21:45:25 Function: TestPauseAndResumeWithWorkerCountChange delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestPauseAndResumeWithWorkerCountChange deleting in the background"
}
2021/12/01 21:45:27 Context: Bucket flush, response code: 200 dump: 
2021/12/01 21:45:37 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/01 21:45:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 21:45:39 Context: Bucket flush, response code: 200 dump: 
2021/12/01 21:45:49 Waiting for dst bucket item count to get to: 0 curr count: 9714
2021/12/01 21:45:54 Waiting for dst bucket item count to get to: 0 curr count: 4714
2021/12/01 21:45:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestPauseAndResumeWithWorkerCountChange (176.79s)
=== RUN   TestPauseResumeWithEventingReb
2021/12/01 21:46:04 Waiting for app: TestPauseResumeWithEventingReb to get un-deployed
2021/12/01 21:46:04 App: TestPauseResumeWithEventingReb got un-deployed
2021/12/01 21:46:04 No eventing-con process running
2021/12/01 21:46:04 Function: TestPauseResumeWithEventingReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestPauseResumeWithEventingReb not found"
 }
}
2021/12/01 21:46:06 Context: Bucket flush, response code: 200 dump: 
2021/12/01 21:46:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 21:46:13 Context: Bucket flush, response code: 200 dump: 
2021/12/01 21:46:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 21:46:18 Deploying app: TestPauseResumeWithEventingReb. Item count src bucket: 0 dst bucket: 0
2021/12/01 21:46:19 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestPauseResumeWithEventingReb' in metakv",
  "warnings": null
 }
}
2021/12/01 21:46:24 Waiting for app: TestPauseResumeWithEventingReb to get deployed
2021/12/01 21:46:29 Waiting for app: TestPauseResumeWithEventingReb to get deployed
2021/12/01 21:46:34 Waiting for app: TestPauseResumeWithEventingReb to get deployed
2021/12/01 21:46:39 Waiting for app: TestPauseResumeWithEventingReb to get deployed
2021/12/01 21:46:39 App: TestPauseResumeWithEventingReb got deployed
2021/12/01 21:46:39 Starting bucket ops to source bucket
2021/12/01 21:46:48 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 21:46:53 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 21:46:58 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 21:47:03 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 21:47:08 Waiting for dst bucket item count to get to: 5000 curr count: 2500
2021/12/01 21:47:08 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/01 21:47:08 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 21:47:08 Pausing app: bucket_op_with_timer
2021/12/01 21:47:08 Function: TestPauseResumeWithEventingReb update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/01 21:47:13 Waiting for function: TestPauseResumeWithEventingReb status to change to paused
2021/12/01 21:47:13 Function: TestPauseResumeWithEventingReb status changed to paused
2021/12/01 21:47:13 Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/01 21:47:13 Removing node(s): [] from the cluster
2021/12/01 21:47:18 Rebalance progress: 29.16666666666666
2021/12/01 21:47:23 Rebalance progress: 29.16666666666666
2021/12/01 21:47:28 Rebalance progress: 100
2021/12/01 21:47:28 Resuming app: bucket_op_with_timer
2021/12/01 21:47:30 Function: TestPauseResumeWithEventingReb update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:6] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/01 21:47:30 Starting bucket ops to source bucket
2021/12/01 21:47:42 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/01 21:47:47 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/01 21:47:52 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/01 21:47:57 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/01 21:48:02 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/01 21:48:07 Waiting for dst bucket item count to get to: 10000 curr count: 6678
2021/12/01 21:48:12 Waiting for dst bucket item count to get to: 10000 curr count: 8984
2021/12/01 21:48:17 Waiting for dst bucket item count to get to: 10000 curr count: 9806
2021/12/01 21:48:17 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/01 21:48:17 Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/01 21:48:22 Rebalance progress: 29.16666666666666
2021/12/01 21:48:27 Rebalance progress: 29.16666666666666
2021/12/01 21:48:32 Rebalance progress: 100

VB distribution across eventing nodes and their workers::

2021/12/01 21:48:38 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:57} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/01 21:48:53 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:57} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/01 21:49:08 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:57} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/01 21:49:23 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:57} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/01 21:49:38 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:57} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/01 21:49:53 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:57} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/01 21:50:08 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:57} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/01 21:50:23 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:57} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/01 21:50:38 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:57} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/01 21:50:53 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:57} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/01 21:51:08 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:57} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/01 21:51:23 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:57} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/01 21:51:38 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:57} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/01 21:51:53 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:57} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/01 21:52:08 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:57} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/01 21:52:23 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:57} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/01 21:52:38 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:57} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/01 21:52:53 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:57} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/01 21:53:08 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:57} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/01 21:53:23 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:57} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/01 21:53:38 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:57} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/01 21:53:53 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:57} {Name:#primary Bucket:hello-world Completion:100}]

DCP Stream statuses:
	stream status: running
	len: 1024
	vb list dump: [0-1023]

vbucket curr owner:
Producer node: 127.0.0.1:9302	Node UUID: 8fba8ff49cea46a70914fc3ca45a5ed0
	workerID: worker_TestPauseResumeWithEventingReb_0
	len: 57
	v: [0-56]
	workerID: worker_TestPauseResumeWithEventingReb_1
	len: 57
	v: [57-113]
	workerID: worker_TestPauseResumeWithEventingReb_2
	len: 57
	v: [114-170]
	workerID: worker_TestPauseResumeWithEventingReb_3
	len: 57
	v: [171-227]
	workerID: worker_TestPauseResumeWithEventingReb_4
	len: 57
	v: [228-284]
	workerID: worker_TestPauseResumeWithEventingReb_5
	len: 57
	v: [285-341]
Producer node: 172.31.2.134:9300	Node UUID: 21889364c47a615566e201ad2e8a9316
	workerID: worker_TestPauseResumeWithEventingReb_1
	len: 57
	v: [740-796]
	workerID: worker_TestPauseResumeWithEventingReb_2
	len: 57
	v: [797-853]
	workerID: worker_TestPauseResumeWithEventingReb_3
	len: 57
	v: [854-910]
	workerID: worker_TestPauseResumeWithEventingReb_4
	len: 57
	v: [911-967]
	workerID: worker_TestPauseResumeWithEventingReb_5
	len: 56
	v: [968-1023]
	workerID: worker_TestPauseResumeWithEventingReb_0
	len: 57
	v: [683-739]
Producer node: 127.0.0.1:9303	Node UUID: 60cefba8cf95278cc87f2b36a423c038
	workerID: worker_TestPauseResumeWithEventingReb_3
	len: 57
	v: [513-569]
	workerID: worker_TestPauseResumeWithEventingReb_4
	len: 57
	v: [570-626]
	workerID: worker_TestPauseResumeWithEventingReb_5
	len: 56
	v: [627-682]
	workerID: worker_TestPauseResumeWithEventingReb_0
	len: 57
	v: [342-398]
	workerID: worker_TestPauseResumeWithEventingReb_1
	len: 57
	v: [399-455]
	workerID: worker_TestPauseResumeWithEventingReb_2
	len: 57
	v: [456-512]


2021/12/01 21:54:08 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 21:54:08 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 21:54:13 clusterSetup.go:397: Waiting for app: TestPauseResumeWithEventingReb to get un-deployed
2021/12/01 21:54:18 clusterSetup.go:397: Waiting for app: TestPauseResumeWithEventingReb to get un-deployed
2021/12/01 21:54:23 clusterSetup.go:397: Waiting for app: TestPauseResumeWithEventingReb to get un-deployed
2021/12/01 21:54:28 clusterSetup.go:397: Waiting for app: TestPauseResumeWithEventingReb to get un-deployed
2021/12/01 21:54:28 clusterSetup.go:405: App: TestPauseResumeWithEventingReb got un-deployed
2021/12/01 21:54:28 eventingSetup.go:46: No eventing-con process running
2021/12/01 21:54:28 eventingSetup.go:418: Function: TestPauseResumeWithEventingReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestPauseResumeWithEventingReb deleting in the background"
}
2021/12/01 21:54:30 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/01 21:54:40 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6058
2021/12/01 21:54:45 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6058
2021/12/01 21:54:45 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 21:54:47 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/01 21:54:57 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/01 21:54:57 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestPauseResumeWithEventingReb (542.98s)
=== RUN   TestBucketDeleteAfterPause
2021/12/01 21:55:07 clusterSetup.go:397: Waiting for app: TestBucketDeleteAfterPause to get un-deployed
2021/12/01 21:55:07 clusterSetup.go:405: App: TestBucketDeleteAfterPause got un-deployed
2021/12/01 21:55:07 eventingSetup.go:46: No eventing-con process running
2021/12/01 21:55:07 eventingSetup.go:418: Function: TestBucketDeleteAfterPause delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestBucketDeleteAfterPause not found"
 }
}
2021/12/01 21:55:09 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/01 21:55:14 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 21:55:15 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/01 21:55:20 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 21:55:21 eventingSetup.go:125: Deploying app: TestBucketDeleteAfterPause. Item count src bucket: 0 dst bucket: 0
2021/12/01 21:55:22 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestBucketDeleteAfterPause' in metakv",
  "warnings": null
 }
}
2021/12/01 21:55:27 clusterSetup.go:346: Waiting for app: TestBucketDeleteAfterPause to get deployed
2021/12/01 21:55:32 clusterSetup.go:346: Waiting for app: TestBucketDeleteAfterPause to get deployed
2021/12/01 21:55:37 clusterSetup.go:346: Waiting for app: TestBucketDeleteAfterPause to get deployed
2021/12/01 21:55:42 clusterSetup.go:346: Waiting for app: TestBucketDeleteAfterPause to get deployed
2021/12/01 21:55:42 clusterSetup.go:354: App: TestBucketDeleteAfterPause got deployed
2021/12/01 21:55:42 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/01 21:55:51 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 21:55:51 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/01 21:55:51 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 21:55:51 handler_test.go:712: Pausing app: bucket_op_on_update
2021/12/01 21:55:52 eventingSetup.go:270: Function: TestBucketDeleteAfterPause update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/01 21:55:57 eventingSetup.go:719: Waiting for function: TestBucketDeleteAfterPause status to change to paused
2021/12/01 21:55:57 eventingSetup.go:727: Function: TestBucketDeleteAfterPause status changed to paused
2021/12/01 21:55:57 handler_test.go:716: Deleting source bucket: default
2021/12/01 21:56:03 eventingSetup.go:719: Waiting for function: TestBucketDeleteAfterPause status to change to undeployed
2021/12/01 21:56:03 eventingSetup.go:727: Function: TestBucketDeleteAfterPause status changed to undeployed
2021/12/01 21:56:08 clusterSetup.go:397: Waiting for app: TestBucketDeleteAfterPause to get un-deployed
2021/12/01 21:56:08 clusterSetup.go:405: App: TestBucketDeleteAfterPause got un-deployed
2021/12/01 21:56:08 eventingSetup.go:46: No eventing-con process running
2021/12/01 21:56:08 eventingSetup.go:418: Function: TestBucketDeleteAfterPause delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestBucketDeleteAfterPause deleting in the background"
}
2021/12/01 21:56:10 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/01 21:56:15 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 21:56:17 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/01 21:56:27 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/01 21:56:27 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestBucketDeleteAfterPause (89.80s)
=== RUN   TestChangeFnCodeBetweenPauseResume
2021/12/01 21:56:37 clusterSetup.go:397: Waiting for app: TestChangeFnCodeBetweenPauseResume to get un-deployed
2021/12/01 21:56:37 clusterSetup.go:405: App: TestChangeFnCodeBetweenPauseResume got un-deployed
2021/12/01 21:56:37 eventingSetup.go:46: No eventing-con process running
2021/12/01 21:56:37 eventingSetup.go:418: Function: TestChangeFnCodeBetweenPauseResume delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestChangeFnCodeBetweenPauseResume not found"
 }
}
2021/12/01 21:56:38 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/01 21:56:43 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 21:56:45 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/01 21:56:50 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 21:56:50 eventingSetup.go:125: Deploying app: TestChangeFnCodeBetweenPauseResume. Item count src bucket: 0 dst bucket: 0
2021/12/01 21:56:51 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestChangeFnCodeBetweenPauseResume' in metakv",
  "warnings": null
 }
}
2021/12/01 21:56:56 clusterSetup.go:346: Waiting for app: TestChangeFnCodeBetweenPauseResume to get deployed
2021/12/01 21:57:01 clusterSetup.go:346: Waiting for app: TestChangeFnCodeBetweenPauseResume to get deployed
2021/12/01 21:57:06 clusterSetup.go:346: Waiting for app: TestChangeFnCodeBetweenPauseResume to get deployed
2021/12/01 21:57:11 clusterSetup.go:346: Waiting for app: TestChangeFnCodeBetweenPauseResume to get deployed
2021/12/01 21:57:11 clusterSetup.go:354: App: TestChangeFnCodeBetweenPauseResume got deployed
2021/12/01 21:57:11 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/01 21:57:46 handler_test.go:739: Pausing function: TestChangeFnCodeBetweenPauseResume
2021/12/01 21:57:46 eventingSetup.go:270: Function: TestChangeFnCodeBetweenPauseResume update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/01 21:57:51 eventingSetup.go:719: Waiting for function: TestChangeFnCodeBetweenPauseResume status to change to paused
2021/12/01 21:57:51 eventingSetup.go:727: Function: TestChangeFnCodeBetweenPauseResume status changed to paused
2021/12/01 22:00:51 handler_test.go:748: Resuming function: TestChangeFnCodeBetweenPauseResume
2021/12/01 22:00:51 eventingSetup.go:125: Deploying app: TestChangeFnCodeBetweenPauseResume. Item count src bucket: 5000 dst bucket: 0
2021/12/01 22:00:52 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestChangeFnCodeBetweenPauseResume' in metakv",
  "warnings": null
 }
}
2021/12/01 22:00:57 eventingSetup.go:795: Waiting for function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter to get to 5000
2021/12/01 22:00:57 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 22:00:57 eventingSetup.go:799: Function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter got to 0 expected 5000
2021/12/01 22:01:02 eventingSetup.go:795: Waiting for function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter to get to 5000
2021/12/01 22:01:02 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 22:01:02 eventingSetup.go:799: Function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter got to 2764 expected 5000
2021/12/01 22:01:07 eventingSetup.go:795: Waiting for function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter to get to 5000
2021/12/01 22:01:07 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 22:01:07 eventingSetup.go:799: Function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter got to 3353 expected 5000
2021/12/01 22:01:12 eventingSetup.go:795: Waiting for function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter to get to 5000
2021/12/01 22:01:12 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 22:01:12 eventingSetup.go:804: Function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter got to 5000
2021/12/01 22:01:12 handler_test.go:753: Undeploying function: TestChangeFnCodeBetweenPauseResume
2021/12/01 22:01:12 eventingSetup.go:270: Function: TestChangeFnCodeBetweenPauseResume update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/01 22:01:22 clusterSetup.go:397: Waiting for app: TestChangeFnCodeBetweenPauseResume to get un-deployed
2021/12/01 22:01:28 clusterSetup.go:397: Waiting for app: TestChangeFnCodeBetweenPauseResume to get un-deployed
2021/12/01 22:01:33 clusterSetup.go:397: Waiting for app: TestChangeFnCodeBetweenPauseResume to get un-deployed
2021/12/01 22:01:33 clusterSetup.go:405: App: TestChangeFnCodeBetweenPauseResume got un-deployed
2021/12/01 22:01:33 eventingSetup.go:46: No eventing-con process running
2021/12/01 22:01:33 eventingSetup.go:418: Function: TestChangeFnCodeBetweenPauseResume delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestChangeFnCodeBetweenPauseResume deleting in the background"
}
2021/12/01 22:01:34 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/01 22:01:44 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/01 22:01:49 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/01 22:01:49 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 22:01:51 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/01 22:01:56 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestChangeFnCodeBetweenPauseResume (329.04s)
=== RUN   TestCommentUnCommentOnDelete
2021/12/01 22:02:06 clusterSetup.go:397: Waiting for app: TestCommentUnCommentOnDelete to get un-deployed
2021/12/01 22:02:06 clusterSetup.go:405: App: TestCommentUnCommentOnDelete got un-deployed
2021/12/01 22:02:06 eventingSetup.go:46: No eventing-con process running
2021/12/01 22:02:06 eventingSetup.go:418: Function: TestCommentUnCommentOnDelete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestCommentUnCommentOnDelete not found"
 }
}
2021/12/01 22:02:07 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/01 22:02:12 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 22:02:14 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/01 22:02:19 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 22:02:19 eventingSetup.go:125: Deploying app: TestCommentUnCommentOnDelete. Item count src bucket: 0 dst bucket: 0
2021/12/01 22:02:20 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCommentUnCommentOnDelete' in metakv",
  "warnings": null
 }
}
2021/12/01 22:02:25 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/01 22:02:30 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/01 22:02:35 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/01 22:02:40 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/01 22:02:40 clusterSetup.go:354: App: TestCommentUnCommentOnDelete got deployed
2021/12/01 22:02:40 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/01 22:02:49 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 22:02:54 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 22:02:54 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/01 22:02:54 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 22:02:54 handler_test.go:780: Undeploying app: TestCommentUnCommentOnDelete
2021/12/01 22:02:54 eventingSetup.go:270: Function: TestCommentUnCommentOnDelete update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/01 22:03:24 eventingSetup.go:125: Deploying app: TestCommentUnCommentOnDelete. Item count src bucket: 5000 dst bucket: 5000
2021/12/01 22:03:25 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCommentUnCommentOnDelete' in metakv",
  "warnings": null
 }
}
2021/12/01 22:03:30 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/01 22:03:35 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/01 22:03:41 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/01 22:03:46 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/01 22:03:46 clusterSetup.go:354: App: TestCommentUnCommentOnDelete got deployed
2021/12/01 22:03:46 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/01 22:03:58 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/01 22:04:03 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/01 22:04:03 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 22:04:03 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 22:04:03 handler_test.go:799: Undeploying app: TestCommentUnCommentOnDelete
2021/12/01 22:04:04 eventingSetup.go:270: Function: TestCommentUnCommentOnDelete update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/01 22:04:14 clusterSetup.go:397: Waiting for app: TestCommentUnCommentOnDelete to get un-deployed
2021/12/01 22:04:19 clusterSetup.go:397: Waiting for app: TestCommentUnCommentOnDelete to get un-deployed
2021/12/01 22:04:24 clusterSetup.go:397: Waiting for app: TestCommentUnCommentOnDelete to get un-deployed
2021/12/01 22:04:24 clusterSetup.go:405: App: TestCommentUnCommentOnDelete got un-deployed
2021/12/01 22:04:24 eventingSetup.go:46: No eventing-con process running
2021/12/01 22:04:24 eventingSetup.go:418: Function: TestCommentUnCommentOnDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCommentUnCommentOnDelete deleting in the background"
}
2021/12/01 22:04:25 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/01 22:04:30 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 22:04:32 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/01 22:04:37 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCommentUnCommentOnDelete (161.02s)
=== RUN   TestCPPWorkerCleanup
2021/12/01 22:04:47 clusterSetup.go:397: Waiting for app: TestCPPWorkerCleanup to get un-deployed
2021/12/01 22:04:47 clusterSetup.go:405: App: TestCPPWorkerCleanup got un-deployed
2021/12/01 22:04:47 eventingSetup.go:46: No eventing-con process running
2021/12/01 22:04:47 eventingSetup.go:418: Function: TestCPPWorkerCleanup delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestCPPWorkerCleanup not found"
 }
}
2021/12/01 22:04:48 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/01 22:04:53 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 22:04:55 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/01 22:05:00 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 22:05:00 eventingSetup.go:125: Deploying app: TestCPPWorkerCleanup. Item count src bucket: 0 dst bucket: 0
2021/12/01 22:05:01 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCPPWorkerCleanup' in metakv",
  "warnings": null
 }
}
2021/12/01 22:05:01 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/01 22:05:08 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 22:05:13 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 22:05:18 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 22:05:23 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 22:05:28 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 3537
2021/12/01 22:05:33 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 3537
2021/12/01 22:05:33 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/01 22:05:33 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 22:05:33 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 22:05:38 clusterSetup.go:397: Waiting for app: TestCPPWorkerCleanup to get un-deployed
2021/12/01 22:05:43 clusterSetup.go:397: Waiting for app: TestCPPWorkerCleanup to get un-deployed
2021/12/01 22:05:48 clusterSetup.go:397: Waiting for app: TestCPPWorkerCleanup to get un-deployed
2021/12/01 22:05:53 clusterSetup.go:397: Waiting for app: TestCPPWorkerCleanup to get un-deployed
2021/12/01 22:05:53 clusterSetup.go:405: App: TestCPPWorkerCleanup got un-deployed
2021/12/01 22:05:53 eventingSetup.go:46: No eventing-con process running
2021/12/01 22:05:53 eventingSetup.go:418: Function: TestCPPWorkerCleanup delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCPPWorkerCleanup deleting in the background"
}
2021/12/01 22:05:55 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/01 22:06:05 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/01 22:06:05 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 22:06:07 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/01 22:06:17 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/01 22:06:17 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCPPWorkerCleanup (129.87s)
=== RUN   TestWithUserXattrs
2021/12/01 22:06:57 clusterSetup.go:397: Waiting for app: TestWithUserXattrs to get un-deployed
2021/12/01 22:06:57 clusterSetup.go:405: App: TestWithUserXattrs got un-deployed
2021/12/01 22:06:57 eventingSetup.go:46: No eventing-con process running
2021/12/01 22:06:57 eventingSetup.go:418: Function: TestWithUserXattrs delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestWithUserXattrs not found"
 }
}
2021/12/01 22:06:58 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/01 22:07:03 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 22:07:05 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/01 22:07:10 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 22:07:10 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/01 22:07:12 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/01 22:07:14 eventingSetup.go:125: Deploying app: TestWithUserXattrs. Item count src bucket: 0 dst bucket: 0
2021/12/01 22:07:15 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestWithUserXattrs' in metakv",
  "warnings": null
 }
}
2021/12/01 22:07:20 clusterSetup.go:346: Waiting for app: TestWithUserXattrs to get deployed
2021/12/01 22:07:25 clusterSetup.go:346: Waiting for app: TestWithUserXattrs to get deployed
2021/12/01 22:07:30 clusterSetup.go:346: Waiting for app: TestWithUserXattrs to get deployed
2021/12/01 22:07:35 clusterSetup.go:346: Waiting for app: TestWithUserXattrs to get deployed
2021/12/01 22:07:35 clusterSetup.go:354: App: TestWithUserXattrs got deployed
2021/12/01 22:07:35 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/01 22:07:44 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 22:07:49 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 3886
2021/12/01 22:07:54 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 3886
2021/12/01 22:07:54 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/01 22:07:54 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 22:07:54 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 22:07:59 clusterSetup.go:397: Waiting for app: TestWithUserXattrs to get un-deployed
2021/12/01 22:08:04 clusterSetup.go:397: Waiting for app: TestWithUserXattrs to get un-deployed
2021/12/01 22:08:09 clusterSetup.go:397: Waiting for app: TestWithUserXattrs to get un-deployed
2021/12/01 22:08:14 clusterSetup.go:397: Waiting for app: TestWithUserXattrs to get un-deployed
2021/12/01 22:08:14 clusterSetup.go:405: App: TestWithUserXattrs got un-deployed
2021/12/01 22:08:14 eventingSetup.go:46: No eventing-con process running
2021/12/01 22:08:14 eventingSetup.go:418: Function: TestWithUserXattrs delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestWithUserXattrs deleting in the background"
}
2021/12/01 22:08:16 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/01 22:08:26 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/01 22:08:26 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 22:08:27 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/01 22:08:37 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/01 22:08:37 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestWithUserXattrs (110.87s)
=== RUN   TestEventProcessingPostBucketFlush
2021/12/01 22:08:47 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/01 22:08:47 clusterSetup.go:405: App: TestEventProcessingPostBucketFlush got un-deployed
2021/12/01 22:08:47 eventingSetup.go:46: No eventing-con process running
2021/12/01 22:08:47 eventingSetup.go:418: Function: TestEventProcessingPostBucketFlush delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventProcessingPostBucketFlush not found"
 }
}
2021/12/01 22:08:49 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/01 22:08:54 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 22:08:56 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/01 22:09:01 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 22:09:01 eventingSetup.go:125: Deploying app: TestEventProcessingPostBucketFlush. Item count src bucket: 0 dst bucket: 0
2021/12/01 22:09:02 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventProcessingPostBucketFlush' in metakv",
  "warnings": null
 }
}
2021/12/01 22:09:07 clusterSetup.go:346: Waiting for app: TestEventProcessingPostBucketFlush to get deployed
2021/12/01 22:09:12 clusterSetup.go:346: Waiting for app: TestEventProcessingPostBucketFlush to get deployed
2021/12/01 22:09:17 clusterSetup.go:346: Waiting for app: TestEventProcessingPostBucketFlush to get deployed
2021/12/01 22:09:22 clusterSetup.go:346: Waiting for app: TestEventProcessingPostBucketFlush to get deployed
2021/12/01 22:09:22 clusterSetup.go:354: App: TestEventProcessingPostBucketFlush got deployed
2021/12/01 22:09:22 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/01 22:09:31 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 22:09:36 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 22:09:36 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/01 22:09:36 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 22:09:40 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/01 22:09:50 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1562
2021/12/01 22:09:55 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1562
2021/12/01 22:09:55 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 22:09:56 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/01 22:10:07 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/01 22:10:07 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 22:10:07 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/01 22:10:33 eventingSetup.go:441: Waiting for dst bucket item count to get to: 25000 curr count: 13695
2021/12/01 22:10:38 eventingSetup.go:441: Waiting for dst bucket item count to get to: 25000 curr count: 24787
2021/12/01 22:10:43 eventingSetup.go:441: Waiting for dst bucket item count to get to: 25000 curr count: 24787
2021/12/01 22:10:43 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 25000 dst bucket count: 25000
2021/12/01 22:10:43 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 22:10:43 eventingSetup.go:270: Function: TestEventProcessingPostBucketFlush update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/01 22:10:48 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/01 22:10:53 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/01 22:10:58 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/01 22:11:03 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/01 22:11:03 clusterSetup.go:405: App: TestEventProcessingPostBucketFlush got un-deployed
2021/12/01 22:11:03 eventingSetup.go:46: No eventing-con process running
2021/12/01 22:11:08 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/01 22:11:08 clusterSetup.go:405: App: TestEventProcessingPostBucketFlush got un-deployed
2021/12/01 22:11:08 eventingSetup.go:46: No eventing-con process running
2021/12/01 22:11:08 eventingSetup.go:418: Function: TestEventProcessingPostBucketFlush delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventProcessingPostBucketFlush deleting in the background"
}
2021/12/01 22:11:09 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/01 22:11:19 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 25000
2021/12/01 22:11:24 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12500
2021/12/01 22:11:24 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 22:11:26 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/01 22:11:36 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 25000
2021/12/01 22:11:36 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventProcessingPostBucketFlush (178.54s)
=== RUN   TestMetaBucketDelete
2021/12/01 22:11:46 clusterSetup.go:397: Waiting for app: TestMetaBucketDelete to get un-deployed
2021/12/01 22:11:46 clusterSetup.go:405: App: TestMetaBucketDelete got un-deployed
2021/12/01 22:11:46 eventingSetup.go:46: No eventing-con process running
2021/12/01 22:11:46 eventingSetup.go:418: Function: TestMetaBucketDelete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMetaBucketDelete not found"
 }
}
2021/12/01 22:11:47 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/01 22:11:52 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 22:11:54 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/01 22:11:59 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 22:11:59 eventingSetup.go:125: Deploying app: TestMetaBucketDelete. Item count src bucket: 0 dst bucket: 0
2021/12/01 22:12:00 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMetaBucketDelete' in metakv",
  "warnings": null
 }
}
2021/12/01 22:12:05 clusterSetup.go:346: Waiting for app: TestMetaBucketDelete to get deployed
2021/12/01 22:12:10 clusterSetup.go:346: Waiting for app: TestMetaBucketDelete to get deployed
2021/12/01 22:12:15 clusterSetup.go:346: Waiting for app: TestMetaBucketDelete to get deployed
2021/12/01 22:12:20 clusterSetup.go:346: Waiting for app: TestMetaBucketDelete to get deployed
2021/12/01 22:12:20 clusterSetup.go:354: App: TestMetaBucketDelete got deployed
2021/12/01 22:12:20 handler_test.go:902: Deleting metadata bucket: eventing
2021/12/01 22:12:22 handler_test.go:904: Deleted metadata bucket: eventing
2021/12/01 22:12:27 clusterSetup.go:397: Waiting for app: TestMetaBucketDelete to get un-deployed
2021/12/01 22:12:27 clusterSetup.go:405: App: TestMetaBucketDelete got un-deployed
2021/12/01 22:12:42 clusterSetup.go:397: Waiting for app: TestMetaBucketDelete to get un-deployed
2021/12/01 22:12:42 clusterSetup.go:405: App: TestMetaBucketDelete got un-deployed
2021/12/01 22:12:42 eventingSetup.go:46: No eventing-con process running
2021/12/01 22:12:42 eventingSetup.go:418: Function: TestMetaBucketDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMetaBucketDelete deleting in the background"
}
2021/12/01 22:12:43 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/01 22:12:48 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 22:12:50 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/01 22:12:55 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestMetaBucketDelete (78.88s)
=== RUN   TestMetaBucketDeleteWithBootstrap
2021/12/01 22:13:05 clusterSetup.go:397: Waiting for app: TestMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/01 22:13:05 clusterSetup.go:405: App: TestMetaBucketDeleteWithBootstrap got un-deployed
2021/12/01 22:13:05 eventingSetup.go:46: No eventing-con process running
2021/12/01 22:13:05 eventingSetup.go:418: Function: TestMetaBucketDeleteWithBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMetaBucketDeleteWithBootstrap not found"
 }
}
2021/12/01 22:13:06 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/01 22:13:11 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 22:13:13 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/01 22:13:18 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 22:13:18 eventingSetup.go:125: Deploying app: TestMetaBucketDeleteWithBootstrap. Item count src bucket: 0 dst bucket: 0
2021/12/01 22:13:19 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMetaBucketDeleteWithBootstrap' in metakv",
  "warnings": null
 }
}
2021/12/01 22:13:38 handler_test.go:923: Deleting metadata bucket: eventing
2021/12/01 22:13:39 handler_test.go:925: Deleted metadata bucket: eventing
2021/12/01 22:13:54 clusterSetup.go:397: Waiting for app: TestMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/01 22:13:54 clusterSetup.go:405: App: TestMetaBucketDeleteWithBootstrap got un-deployed
2021/12/01 22:14:09 clusterSetup.go:397: Waiting for app: TestMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/01 22:14:09 clusterSetup.go:405: App: TestMetaBucketDeleteWithBootstrap got un-deployed
2021/12/01 22:14:09 eventingSetup.go:46: No eventing-con process running
2021/12/01 22:14:09 eventingSetup.go:418: Function: TestMetaBucketDeleteWithBootstrap delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMetaBucketDeleteWithBootstrap deleting in the background"
}
2021/12/01 22:14:10 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/01 22:14:15 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 22:14:17 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/01 22:14:22 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestMetaBucketDeleteWithBootstrap (86.97s)
=== RUN   TestSourceBucketDelete
2021/12/01 22:14:32 clusterSetup.go:397: Waiting for app: TestSourceBucketDelete to get un-deployed
2021/12/01 22:14:32 clusterSetup.go:405: App: TestSourceBucketDelete got un-deployed
2021/12/01 22:14:32 eventingSetup.go:46: No eventing-con process running
2021/12/01 22:14:32 eventingSetup.go:418: Function: TestSourceBucketDelete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestSourceBucketDelete not found"
 }
}
2021/12/01 22:14:33 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/01 22:14:38 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 22:14:40 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/01 22:14:45 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 22:14:45 eventingSetup.go:125: Deploying app: TestSourceBucketDelete. Item count src bucket: 0 dst bucket: 0
2021/12/01 22:14:46 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestSourceBucketDelete' in metakv",
  "warnings": null
 }
}
2021/12/01 22:14:51 clusterSetup.go:346: Waiting for app: TestSourceBucketDelete to get deployed
2021/12/01 22:14:56 clusterSetup.go:346: Waiting for app: TestSourceBucketDelete to get deployed
2021/12/01 22:15:01 clusterSetup.go:346: Waiting for app: TestSourceBucketDelete to get deployed
2021/12/01 22:15:06 clusterSetup.go:346: Waiting for app: TestSourceBucketDelete to get deployed
2021/12/01 22:15:06 clusterSetup.go:354: App: TestSourceBucketDelete got deployed
2021/12/01 22:15:06 handler_test.go:945: Deleting source bucket: default
2021/12/01 22:15:08 handler_test.go:947: Deleted source bucket: default
2021/12/01 22:15:13 clusterSetup.go:397: Waiting for app: TestSourceBucketDelete to get un-deployed
2021/12/01 22:15:13 clusterSetup.go:405: App: TestSourceBucketDelete got un-deployed
2021/12/01 22:15:28 clusterSetup.go:397: Waiting for app: TestSourceBucketDelete to get un-deployed
2021/12/01 22:15:28 clusterSetup.go:405: App: TestSourceBucketDelete got un-deployed
2021/12/01 22:15:28 eventingSetup.go:46: No eventing-con process running
2021/12/01 22:15:28 eventingSetup.go:418: Function: TestSourceBucketDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestSourceBucketDelete deleting in the background"
}
2021/12/01 22:15:29 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/01 22:15:34 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 22:15:36 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/01 22:15:41 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestSourceBucketDelete (78.99s)
=== RUN   TestSourceBucketDeleteWithBootstrap
2021/12/01 22:15:51 clusterSetup.go:397: Waiting for app: TestSourceBucketDeleteWithBootstrap to get un-deployed
2021/12/01 22:15:51 clusterSetup.go:405: App: TestSourceBucketDeleteWithBootstrap got un-deployed
2021/12/01 22:15:51 eventingSetup.go:46: No eventing-con process running
2021/12/01 22:15:51 eventingSetup.go:418: Function: TestSourceBucketDeleteWithBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestSourceBucketDeleteWithBootstrap not found"
 }
}
2021/12/01 22:15:52 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/01 22:15:57 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 22:15:59 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/01 22:16:04 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 22:16:04 eventingSetup.go:125: Deploying app: TestSourceBucketDeleteWithBootstrap. Item count src bucket: 0 dst bucket: 0
2021/12/01 22:16:05 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestSourceBucketDeleteWithBootstrap' in metakv",
  "warnings": null
 }
}
2021/12/01 22:16:24 handler_test.go:966: Deleting source bucket: default
2021/12/01 22:16:25 handler_test.go:968: Deleted source bucket: default
2021/12/01 22:16:40 clusterSetup.go:397: Waiting for app: TestSourceBucketDeleteWithBootstrap to get un-deployed
2021/12/01 22:16:40 clusterSetup.go:405: App: TestSourceBucketDeleteWithBootstrap got un-deployed
2021/12/01 22:16:55 clusterSetup.go:397: Waiting for app: TestSourceBucketDeleteWithBootstrap to get un-deployed
2021/12/01 22:16:55 clusterSetup.go:405: App: TestSourceBucketDeleteWithBootstrap got un-deployed
2021/12/01 22:16:55 eventingSetup.go:46: No eventing-con process running
2021/12/01 22:16:55 eventingSetup.go:418: Function: TestSourceBucketDeleteWithBootstrap delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestSourceBucketDeleteWithBootstrap deleting in the background"
}
2021/12/01 22:16:57 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/01 22:17:02 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 22:17:04 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/01 22:17:09 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestSourceBucketDeleteWithBootstrap (87.67s)
=== RUN   TestSourceAndMetaBucketDeleteWithBootstrap
2021/12/01 22:17:19 clusterSetup.go:397: Waiting for app: TestSourceAndMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/01 22:17:19 clusterSetup.go:405: App: TestSourceAndMetaBucketDeleteWithBootstrap got un-deployed
2021/12/01 22:17:19 eventingSetup.go:46: No eventing-con process running
2021/12/01 22:17:19 eventingSetup.go:418: Function: TestSourceAndMetaBucketDeleteWithBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestSourceAndMetaBucketDeleteWithBootstrap not found"
 }
}
2021/12/01 22:17:20 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/01 22:17:25 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 22:17:27 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/01 22:17:32 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 22:17:32 eventingSetup.go:125: Deploying app: TestSourceAndMetaBucketDeleteWithBootstrap. Item count src bucket: 0 dst bucket: 0
2021/12/01 22:17:33 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestSourceAndMetaBucketDeleteWithBootstrap' in metakv",
  "warnings": null
 }
}
2021/12/01 22:17:52 handler_test.go:988: Deleting source bucket: default
2021/12/01 22:17:53 handler_test.go:990: Deleted source bucket: default
2021/12/01 22:17:53 handler_test.go:992: Deleting metadata bucket: eventing
2021/12/01 22:17:54 handler_test.go:994: Deleted metadata bucket: eventing
2021/12/01 22:18:09 clusterSetup.go:397: Waiting for app: TestSourceAndMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/01 22:18:09 clusterSetup.go:405: App: TestSourceAndMetaBucketDeleteWithBootstrap got un-deployed
2021/12/01 22:18:24 clusterSetup.go:397: Waiting for app: TestSourceAndMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/01 22:18:24 clusterSetup.go:405: App: TestSourceAndMetaBucketDeleteWithBootstrap got un-deployed
2021/12/01 22:18:24 eventingSetup.go:46: No eventing-con process running
2021/12/01 22:18:24 eventingSetup.go:418: Function: TestSourceAndMetaBucketDeleteWithBootstrap delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestSourceAndMetaBucketDeleteWithBootstrap deleting in the background"
}
2021/12/01 22:18:25 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/01 22:18:30 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 22:18:32 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/01 22:18:37 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestSourceAndMetaBucketDeleteWithBootstrap (88.25s)
=== RUN   TestUndeployDuringBootstrap
2021/12/01 22:18:47 clusterSetup.go:397: Waiting for app: TestUndeployDuringBootstrap to get un-deployed
2021/12/01 22:18:47 clusterSetup.go:405: App: TestUndeployDuringBootstrap got un-deployed
2021/12/01 22:18:47 eventingSetup.go:46: No eventing-con process running
2021/12/01 22:18:47 eventingSetup.go:418: Function: TestUndeployDuringBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestUndeployDuringBootstrap not found"
 }
}
2021/12/01 22:18:48 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/01 22:18:53 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 22:18:55 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/01 22:19:00 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 22:19:00 eventingSetup.go:125: Deploying app: TestUndeployDuringBootstrap. Item count src bucket: 0 dst bucket: 0
2021/12/01 22:19:01 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestUndeployDuringBootstrap' in metakv",
  "warnings": null
 }
}
2021/12/01 22:19:01 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/01 22:19:13 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 22:19:13 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 22:19:13 clusterSetup.go:375: Apps undergoing bootstrap: true
2021/12/01 22:19:13 clusterSetup.go:378: App: TestUndeployDuringBootstrap started bootstrap
2021/12/01 22:19:13 clusterSetup.go:375: Apps undergoing bootstrap: true
2021/12/01 22:19:18 clusterSetup.go:375: Apps undergoing bootstrap: true
2021/12/01 22:19:23 clusterSetup.go:367: No apps undergoing bootstrap
2021/12/01 22:19:28 clusterSetup.go:397: Waiting for app: TestUndeployDuringBootstrap to get un-deployed
2021/12/01 22:19:33 clusterSetup.go:397: Waiting for app: TestUndeployDuringBootstrap to get un-deployed
2021/12/01 22:19:38 clusterSetup.go:397: Waiting for app: TestUndeployDuringBootstrap to get un-deployed
2021/12/01 22:19:38 clusterSetup.go:405: App: TestUndeployDuringBootstrap got un-deployed
2021/12/01 22:19:38 eventingSetup.go:46: No eventing-con process running
2021/12/01 22:20:03 clusterSetup.go:397: Waiting for app: TestUndeployDuringBootstrap to get un-deployed
2021/12/01 22:20:03 clusterSetup.go:405: App: TestUndeployDuringBootstrap got un-deployed
2021/12/01 22:20:03 eventingSetup.go:46: No eventing-con process running
2021/12/01 22:20:03 eventingSetup.go:418: Function: TestUndeployDuringBootstrap delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestUndeployDuringBootstrap deleting in the background"
}
2021/12/01 22:20:04 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/01 22:20:14 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/01 22:20:14 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 22:20:16 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/01 22:20:26 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/01 22:20:26 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestUndeployDuringBootstrap (109.47s)
=== RUN   TestDeleteBeforeUndeploy
2021/12/01 22:20:31 eventingSetup.go:125: Deploying app: TestDeleteBeforeUndeploy. Item count src bucket: 0 dst bucket: 0
2021/12/01 22:20:32 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeleteBeforeUndeploy' in metakv",
  "warnings": null
 }
}
2021/12/01 22:20:37 clusterSetup.go:346: Waiting for app: TestDeleteBeforeUndeploy to get deployed
2021/12/01 22:20:42 clusterSetup.go:346: Waiting for app: TestDeleteBeforeUndeploy to get deployed
2021/12/01 22:20:47 clusterSetup.go:346: Waiting for app: TestDeleteBeforeUndeploy to get deployed
2021/12/01 22:20:52 clusterSetup.go:346: Waiting for app: TestDeleteBeforeUndeploy to get deployed
2021/12/01 22:20:52 clusterSetup.go:354: App: TestDeleteBeforeUndeploy got deployed
2021/12/01 22:20:52 eventingSetup.go:418: Function: TestDeleteBeforeUndeploy delete from main store request response code: 422 dump: {
 "name": "ERR_APP_NOT_UNDEPLOYED",
 "code": 22,
 "description": "Function hasn't undeployed",
 "attributes": null,
 "runtime_info": {
  "code": 22,
  "info": "Function: TestDeleteBeforeUndeploy skipping delete request from primary store, as it hasn't been undeployed"
 }
}
2021/12/01 22:20:53 eventingSetup.go:270: Function: TestDeleteBeforeUndeploy update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/01 22:20:53 eventingSetup.go:418: Function: TestDeleteBeforeUndeploy delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDeleteBeforeUndeploy deleting in the background"
}
2021/12/01 22:20:58 clusterSetup.go:397: Waiting for app: TestDeleteBeforeUndeploy to get un-deployed
2021/12/01 22:20:58 clusterSetup.go:405: App: TestDeleteBeforeUndeploy got un-deployed
2021/12/01 22:20:58 eventingSetup.go:46: No eventing-con process running
2021/12/01 22:20:58 eventingSetup.go:418: Function: TestDeleteBeforeUndeploy delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDeleteBeforeUndeploy not found"
 }
}
2021/12/01 22:20:59 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/01 22:21:04 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 22:21:06 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/01 22:21:11 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDeleteBeforeUndeploy (44.63s)
=== RUN   TestUndeployWhenTimersAreFired
2021/12/01 22:21:21 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/01 22:21:21 clusterSetup.go:405: App: TestUndeployWhenTimersAreFired got un-deployed
2021/12/01 22:21:21 eventingSetup.go:46: No eventing-con process running
2021/12/01 22:21:21 eventingSetup.go:418: Function: TestUndeployWhenTimersAreFired delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestUndeployWhenTimersAreFired not found"
 }
}
2021/12/01 22:21:22 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/01 22:21:27 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 22:21:29 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/01 22:21:34 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 22:21:34 eventingSetup.go:125: Deploying app: TestUndeployWhenTimersAreFired. Item count src bucket: 0 dst bucket: 0
2021/12/01 22:21:36 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestUndeployWhenTimersAreFired' in metakv",
  "warnings": null
 }
}
2021/12/01 22:21:41 clusterSetup.go:346: Waiting for app: TestUndeployWhenTimersAreFired to get deployed
2021/12/01 22:21:46 clusterSetup.go:346: Waiting for app: TestUndeployWhenTimersAreFired to get deployed
2021/12/01 22:21:51 clusterSetup.go:346: Waiting for app: TestUndeployWhenTimersAreFired to get deployed
2021/12/01 22:21:56 clusterSetup.go:346: Waiting for app: TestUndeployWhenTimersAreFired to get deployed
2021/12/01 22:21:56 clusterSetup.go:354: App: TestUndeployWhenTimersAreFired got deployed
2021/12/01 22:21:56 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/01 22:22:26 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 22:22:31 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/01 22:22:36 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/01 22:22:41 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/01 22:22:46 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/01 22:22:46 clusterSetup.go:405: App: TestUndeployWhenTimersAreFired got un-deployed
2021/12/01 22:22:46 eventingSetup.go:46: No eventing-con process running
2021/12/01 22:24:31 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/01 22:24:31 clusterSetup.go:405: App: TestUndeployWhenTimersAreFired got un-deployed
2021/12/01 22:24:31 eventingSetup.go:46: No eventing-con process running
2021/12/01 22:24:31 eventingSetup.go:418: Function: TestUndeployWhenTimersAreFired delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestUndeployWhenTimersAreFired deleting in the background"
}
2021/12/01 22:24:32 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/01 22:24:42 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/01 22:24:47 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/01 22:24:47 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 22:24:49 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/01 22:24:59 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/01 22:25:04 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/01 22:25:04 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestUndeployWhenTimersAreFired (233.23s)
=== RUN   TestUndeployWithKVFailover
2021/12/01 22:25:14 clusterSetup.go:397: Waiting for app: TestUndeployWithKVFailover to get un-deployed
2021/12/01 22:25:14 clusterSetup.go:405: App: TestUndeployWithKVFailover got un-deployed
2021/12/01 22:25:14 eventingSetup.go:46: No eventing-con process running
2021/12/01 22:25:14 eventingSetup.go:418: Function: TestUndeployWithKVFailover delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestUndeployWithKVFailover not found"
 }
}
2021/12/01 22:25:15 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/01 22:25:21 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 22:25:22 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/01 22:25:27 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 22:25:27 eventingSetup.go:125: Deploying app: TestUndeployWithKVFailover. Item count src bucket: 0 dst bucket: 0
2021/12/01 22:25:28 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestUndeployWithKVFailover' in metakv",
  "warnings": null
 }
}
2021/12/01 22:25:28 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/01 22:25:35 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 22:25:40 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 22:25:45 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 22:25:50 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 22:25:55 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 22:26:00 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 758
2021/12/01 22:26:05 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 3258
2021/12/01 22:26:05 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/01 22:26:05 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: kv to the cluster
2021/12/01 22:26:06 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/01 22:26:11 Rebalance progress: 29.16666666666666
2021/12/01 22:26:16 Rebalance progress: 29.16666666666666
2021/12/01 22:26:21 Rebalance progress: 100
2021/12/01 22:26:21 Failing over: [127.0.0.1:19003]
2021/12/01 22:26:31 Function: TestUndeployWithKVFailover update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/01 22:27:31 Removing node(s): [] from the cluster
2021/12/01 22:27:36 Rebalance progress: 29.16666666666666
2021/12/01 22:27:41 Rebalance progress: 29.16666666666666
2021/12/01 22:27:46 Rebalance progress: 100
2021/12/01 22:27:51 Waiting for app: TestUndeployWithKVFailover to get un-deployed
2021/12/01 22:27:51 App: TestUndeployWithKVFailover got un-deployed
2021/12/01 22:27:51 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 22:27:56 Waiting for app: TestUndeployWithKVFailover to get un-deployed
2021/12/01 22:27:56 App: TestUndeployWithKVFailover got un-deployed
2021/12/01 22:27:56 No eventing-con process running
2021/12/01 22:27:56 Function: TestUndeployWithKVFailover delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestUndeployWithKVFailover deleting in the background"
}
2021/12/01 22:27:57 Context: Bucket flush, response code: 200 dump: 
2021/12/01 22:28:07 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/01 22:28:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 22:28:09 Context: Bucket flush, response code: 200 dump: 
2021/12/01 22:28:19 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/01 22:28:24 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/01 22:28:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestUndeployWithKVFailover (200.17s)
=== RUN   TestBucketFlushWhileFnDeployed
2021/12/01 22:28:34 Waiting for app: TestBucketFlushWhileFnDeployed to get un-deployed
2021/12/01 22:28:34 App: TestBucketFlushWhileFnDeployed got un-deployed
2021/12/01 22:28:34 No eventing-con process running
2021/12/01 22:28:34 Function: TestBucketFlushWhileFnDeployed delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestBucketFlushWhileFnDeployed not found"
 }
}
2021/12/01 22:28:36 Context: Bucket flush, response code: 200 dump: 
2021/12/01 22:28:41 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 22:28:43 Context: Bucket flush, response code: 200 dump: 
2021/12/01 22:28:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 22:28:48 Deploying app: TestBucketFlushWhileFnDeployed. Item count src bucket: 0 dst bucket: 0
2021/12/01 22:28:49 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestBucketFlushWhileFnDeployed' in metakv",
  "warnings": null
 }
}
2021/12/01 22:28:54 Waiting for app: TestBucketFlushWhileFnDeployed to get deployed
2021/12/01 22:28:59 Waiting for app: TestBucketFlushWhileFnDeployed to get deployed
2021/12/01 22:29:04 Waiting for app: TestBucketFlushWhileFnDeployed to get deployed
2021/12/01 22:29:09 Waiting for app: TestBucketFlushWhileFnDeployed to get deployed
2021/12/01 22:29:09 App: TestBucketFlushWhileFnDeployed got deployed
2021/12/01 22:29:09 Starting bucket ops to source bucket
2021/12/01 22:29:29 Waiting for dst bucket item count to get to: 20000 curr count: 12202
2021/12/01 22:29:34 Waiting for dst bucket item count to get to: 20000 curr count: 16784
2021/12/01 22:29:34 src & dst bucket item count matched up. src bucket count: 20000 dst bucket count: 20000
2021/12/01 22:29:34 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 22:29:37 Context: Bucket flush, response code: 200 dump: 
2021/12/01 22:29:47 Waiting for dst bucket item count to get to: 0 curr count: 20000
2021/12/01 22:29:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 22:29:49 Context: Bucket flush, response code: 200 dump: 
2021/12/01 22:29:59 Waiting for dst bucket item count to get to: 0 curr count: 18576
2021/12/01 22:30:04 Waiting for dst bucket item count to get to: 0 curr count: 8576
2021/12/01 22:30:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 22:30:04 Starting bucket ops to source bucket
2021/12/01 22:30:17 Waiting for dst bucket item count to get to: 10000 curr count: 2107
2021/12/01 22:30:22 Waiting for dst bucket item count to get to: 10000 curr count: 5581
2021/12/01 22:30:27 Waiting for dst bucket item count to get to: 10000 curr count: 8474
2021/12/01 22:30:27 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/01 22:30:27 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 22:30:27 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 22:30:32 Waiting for app: TestBucketFlushWhileFnDeployed to get un-deployed
2021/12/01 22:30:37 Waiting for app: TestBucketFlushWhileFnDeployed to get un-deployed
2021/12/01 22:30:42 Waiting for app: TestBucketFlushWhileFnDeployed to get un-deployed
2021/12/01 22:30:47 Waiting for app: TestBucketFlushWhileFnDeployed to get un-deployed
2021/12/01 22:30:47 App: TestBucketFlushWhileFnDeployed got un-deployed
2021/12/01 22:30:47 No eventing-con process running
2021/12/01 22:30:47 Function: TestBucketFlushWhileFnDeployed delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestBucketFlushWhileFnDeployed deleting in the background"
}
2021/12/01 22:30:48 Context: Bucket flush, response code: 200 dump: 
2021/12/01 22:30:58 Waiting for dst bucket item count to get to: 0 curr count: 9074
2021/12/01 22:31:03 Waiting for dst bucket item count to get to: 0 curr count: 4074
2021/12/01 22:31:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 22:31:05 Context: Bucket flush, response code: 200 dump: 
2021/12/01 22:31:15 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/01 22:31:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestBucketFlushWhileFnDeployed (170.70s)
=== RUN   TestTimerOverwrite
2021/12/01 22:31:25 Waiting for app: TestTimerOverwrite to get un-deployed
2021/12/01 22:31:25 App: TestTimerOverwrite got un-deployed
2021/12/01 22:31:25 No eventing-con process running
2021/12/01 22:31:25 Function: TestTimerOverwrite delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestTimerOverwrite not found"
 }
}
2021/12/01 22:31:26 Context: Bucket flush, response code: 200 dump: 
2021/12/01 22:31:31 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 22:31:33 Context: Bucket flush, response code: 200 dump: 
2021/12/01 22:31:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 22:31:38 Deploying app: TestTimerOverwrite. Item count src bucket: 0 dst bucket: 0
2021/12/01 22:31:39 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestTimerOverwrite' in metakv",
  "warnings": null
 }
}
2021/12/01 22:31:44 Waiting for app: TestTimerOverwrite to get deployed
2021/12/01 22:31:49 Waiting for app: TestTimerOverwrite to get deployed
2021/12/01 22:31:54 Waiting for app: TestTimerOverwrite to get deployed
2021/12/01 22:31:59 Waiting for app: TestTimerOverwrite to get deployed
2021/12/01 22:31:59 App: TestTimerOverwrite got deployed
2021/12/01 22:32:09 Starting bucket ops to source bucket
2021/12/01 22:32:19 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 22:32:24 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 22:32:29 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 22:32:34 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 22:32:39 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 22:32:44 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 22:32:49 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 22:32:54 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 22:32:59 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 22:33:04 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 22:33:09 Waiting for dst bucket item count to get to: 5000 curr count: 1097
2021/12/01 22:33:14 Waiting for dst bucket item count to get to: 5000 curr count: 1097
2021/12/01 22:33:14 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/01 22:33:24 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 22:33:24 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 22:33:29 Waiting for app: TestTimerOverwrite to get un-deployed
2021/12/01 22:33:34 Waiting for app: TestTimerOverwrite to get un-deployed
2021/12/01 22:33:39 Waiting for app: TestTimerOverwrite to get un-deployed
2021/12/01 22:33:44 Waiting for app: TestTimerOverwrite to get un-deployed
2021/12/01 22:33:44 App: TestTimerOverwrite got un-deployed
2021/12/01 22:33:44 No eventing-con process running
2021/12/01 22:33:44 Function: TestTimerOverwrite delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestTimerOverwrite deleting in the background"
}
2021/12/01 22:33:45 Context: Bucket flush, response code: 200 dump: 
2021/12/01 22:33:55 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/01 22:33:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 22:33:57 Context: Bucket flush, response code: 200 dump: 
2021/12/01 22:34:07 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/01 22:34:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestTimerOverwrite (172.25s)
=== RUN   TestUndeployBackdoorDuringBootstrap
2021/12/01 22:34:12 Adding node: https://127.0.0.1:19003 with role: kv to the cluster
2021/12/01 22:34:13 Removing node(s): [] from the cluster
2021/12/01 22:34:18 Rebalance progress: 100
2021/12/01 22:34:18 Failing over: [127.0.0.1:19003]
2021/12/01 22:34:23 Waiting for app: TestUndeployBackdoorDuringBootstrap to get un-deployed
2021/12/01 22:34:23 App: TestUndeployBackdoorDuringBootstrap got un-deployed
2021/12/01 22:34:23 No eventing-con process running
2021/12/01 22:34:23 Function: TestUndeployBackdoorDuringBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestUndeployBackdoorDuringBootstrap not found"
 }
}
2021/12/01 22:34:24 Context: Bucket flush, response code: 200 dump: 
2021/12/01 22:34:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 22:34:31 Context: Bucket flush, response code: 200 dump: 
2021/12/01 22:34:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 22:34:36 Deploying app: TestUndeployBackdoorDuringBootstrap. Item count src bucket: 0 dst bucket: 0
2021/12/01 22:34:37 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestUndeployBackdoorDuringBootstrap' in metakv",
  "warnings": null
 }
}
2021/12/01 22:34:37 Starting bucket ops to source bucket
2021/12/01 22:34:47 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 22:34:47 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 22:34:47 Context: Post to set retry counter, response code: 200 dump: 
2021/12/01 22:34:47 set retry counter, response &{[] }
2021/12/01 22:35:52 Waiting for app: TestUndeployBackdoorDuringBootstrap to get un-deployed
2021/12/01 22:35:52 App: TestUndeployBackdoorDuringBootstrap got un-deployed
2021/12/01 22:35:52 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 22:35:52 Function: TestUndeployBackdoorDuringBootstrap delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestUndeployBackdoorDuringBootstrap deleting in the background"
}
2021/12/01 22:35:57 Waiting for app: TestUndeployBackdoorDuringBootstrap to get un-deployed
2021/12/01 22:35:57 App: TestUndeployBackdoorDuringBootstrap got un-deployed
2021/12/01 22:35:57 No eventing-con process running
2021/12/01 22:35:57 Function: TestUndeployBackdoorDuringBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestUndeployBackdoorDuringBootstrap not found"
 }
}
2021/12/01 22:35:58 Context: Bucket flush, response code: 200 dump: 
2021/12/01 22:36:08 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/01 22:36:13 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/01 22:36:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 22:36:15 Context: Bucket flush, response code: 200 dump: 
2021/12/01 22:36:25 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/01 22:36:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 22:36:25 Removing node(s): [] from the cluster
2021/12/01 22:36:30 Rebalance progress: 100
--- PASS: TestUndeployBackdoorDuringBootstrap (143.10s)
=== RUN   TestOnUpdateSrcMutation
2021/12/01 22:36:40 Waiting for app: TestOnUpdateSrcMutation to get un-deployed
2021/12/01 22:36:40 App: TestOnUpdateSrcMutation got un-deployed
2021/12/01 22:36:40 No eventing-con process running
2021/12/01 22:36:40 Function: TestOnUpdateSrcMutation delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateSrcMutation not found"
 }
}
2021/12/01 22:36:42 Context: Bucket flush, response code: 200 dump: 
2021/12/01 22:36:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 22:36:49 Context: Bucket flush, response code: 200 dump: 
2021/12/01 22:36:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 22:36:54 Deploying app: TestOnUpdateSrcMutation. Item count src bucket: 0 dst bucket: 0
2021/12/01 22:36:55 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateSrcMutation' in metakv",
  "warnings": null
 }
}
2021/12/01 22:36:55 Starting bucket ops to source bucket
2021/12/01 22:37:22 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/01 22:37:22 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 22:37:22 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 22:37:27 Waiting for app: TestOnUpdateSrcMutation to get un-deployed
2021/12/01 22:37:32 Waiting for app: TestOnUpdateSrcMutation to get un-deployed
2021/12/01 22:37:37 Waiting for app: TestOnUpdateSrcMutation to get un-deployed
2021/12/01 22:37:42 Waiting for app: TestOnUpdateSrcMutation to get un-deployed
2021/12/01 22:37:42 App: TestOnUpdateSrcMutation got un-deployed
2021/12/01 22:37:42 No eventing-con process running
2021/12/01 22:37:42 Function: TestOnUpdateSrcMutation delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateSrcMutation deleting in the background"
}
2021/12/01 22:37:43 Context: Bucket flush, response code: 200 dump: 
2021/12/01 22:37:53 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/01 22:37:58 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/01 22:37:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 22:38:00 Context: Bucket flush, response code: 200 dump: 
2021/12/01 22:38:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateSrcMutation (94.71s)
=== RUN   TestOnDeleteSrcMutation
2021/12/01 22:38:15 Waiting for app: TestOnDeleteSrcMutation to get un-deployed
2021/12/01 22:38:15 App: TestOnDeleteSrcMutation got un-deployed
2021/12/01 22:38:15 No eventing-con process running
2021/12/01 22:38:15 Function: TestOnDeleteSrcMutation delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnDeleteSrcMutation not found"
 }
}
2021/12/01 22:38:16 Context: Bucket flush, response code: 200 dump: 
2021/12/01 22:38:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 22:38:23 Context: Bucket flush, response code: 200 dump: 
2021/12/01 22:38:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 22:38:28 Deploying app: TestOnDeleteSrcMutation. Item count src bucket: 0 dst bucket: 0
2021/12/01 22:38:30 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnDeleteSrcMutation' in metakv",
  "warnings": null
 }
}
2021/12/01 22:38:30 Starting bucket ops to source bucket
2021/12/01 22:38:58 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/01 22:38:58 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 22:38:58 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 22:39:03 Waiting for app: TestOnDeleteSrcMutation to get un-deployed
2021/12/01 22:39:08 Waiting for app: TestOnDeleteSrcMutation to get un-deployed
2021/12/01 22:39:13 Waiting for app: TestOnDeleteSrcMutation to get un-deployed
2021/12/01 22:39:18 Waiting for app: TestOnDeleteSrcMutation to get un-deployed
2021/12/01 22:39:18 App: TestOnDeleteSrcMutation got un-deployed
2021/12/01 22:39:18 No eventing-con process running
2021/12/01 22:39:19 Function: TestOnDeleteSrcMutation delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnDeleteSrcMutation deleting in the background"
}
2021/12/01 22:39:20 Context: Bucket flush, response code: 200 dump: 
2021/12/01 22:39:30 Waiting for dst bucket item count to get to: 0 curr count: 2542
2021/12/01 22:39:35 Waiting for dst bucket item count to get to: 0 curr count: 2542
2021/12/01 22:39:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 22:39:37 Context: Bucket flush, response code: 200 dump: 
2021/12/01 22:39:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnDeleteSrcMutation (96.56s)
=== RUN   TestOnUpdateSrcMutationWithTimer
2021/12/01 22:39:52 Waiting for app: TestOnUpdateSrcMutationWithTimer to get un-deployed
2021/12/01 22:39:52 App: TestOnUpdateSrcMutationWithTimer got un-deployed
2021/12/01 22:39:52 No eventing-con process running
2021/12/01 22:39:52 Function: TestOnUpdateSrcMutationWithTimer delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateSrcMutationWithTimer not found"
 }
}
2021/12/01 22:39:53 Context: Bucket flush, response code: 200 dump: 
2021/12/01 22:39:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 22:40:00 Context: Bucket flush, response code: 200 dump: 
2021/12/01 22:40:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 22:40:05 Deploying app: TestOnUpdateSrcMutationWithTimer. Item count src bucket: 0 dst bucket: 0
2021/12/01 22:40:06 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateSrcMutationWithTimer' in metakv",
  "warnings": null
 }
}
2021/12/01 22:40:06 Starting bucket ops to source bucket
2021/12/01 22:40:43 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/01 22:40:43 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 22:40:43 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 22:40:48 Waiting for app: TestOnUpdateSrcMutationWithTimer to get un-deployed
2021/12/01 22:40:53 Waiting for app: TestOnUpdateSrcMutationWithTimer to get un-deployed
2021/12/01 22:40:58 Waiting for app: TestOnUpdateSrcMutationWithTimer to get un-deployed
2021/12/01 22:41:03 Waiting for app: TestOnUpdateSrcMutationWithTimer to get un-deployed
2021/12/01 22:41:03 App: TestOnUpdateSrcMutationWithTimer got un-deployed
2021/12/01 22:41:03 No eventing-con process running
2021/12/01 22:41:03 Function: TestOnUpdateSrcMutationWithTimer delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateSrcMutationWithTimer deleting in the background"
}
2021/12/01 22:41:04 Context: Bucket flush, response code: 200 dump: 
2021/12/01 22:41:15 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/01 22:41:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 22:41:16 Context: Bucket flush, response code: 200 dump: 
2021/12/01 22:41:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateSrcMutationWithTimer (99.75s)
=== RUN   TestOnDeleteSrcMutationsWithTimer
2021/12/01 22:41:31 Waiting for app: TestOnDeleteSrcMutationsWithTimer to get un-deployed
2021/12/01 22:41:31 App: TestOnDeleteSrcMutationsWithTimer got un-deployed
2021/12/01 22:41:31 No eventing-con process running
2021/12/01 22:41:31 Function: TestOnDeleteSrcMutationsWithTimer delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnDeleteSrcMutationsWithTimer not found"
 }
}
2021/12/01 22:41:33 Context: Bucket flush, response code: 200 dump: 
2021/12/01 22:41:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 22:41:39 Context: Bucket flush, response code: 200 dump: 
2021/12/01 22:41:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 22:41:45 Deploying app: TestOnDeleteSrcMutationsWithTimer. Item count src bucket: 0 dst bucket: 0
2021/12/01 22:41:46 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnDeleteSrcMutationsWithTimer' in metakv",
  "warnings": null
 }
}
2021/12/01 22:41:46 Starting bucket ops to source bucket
2021/12/01 22:42:24 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/01 22:42:24 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 22:42:25 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 22:42:30 Waiting for app: TestOnDeleteSrcMutationsWithTimer to get un-deployed
2021/12/01 22:42:35 Waiting for app: TestOnDeleteSrcMutationsWithTimer to get un-deployed
2021/12/01 22:42:40 Waiting for app: TestOnDeleteSrcMutationsWithTimer to get un-deployed
2021/12/01 22:42:45 Waiting for app: TestOnDeleteSrcMutationsWithTimer to get un-deployed
2021/12/01 22:42:45 App: TestOnDeleteSrcMutationsWithTimer got un-deployed
2021/12/01 22:42:45 No eventing-con process running
2021/12/01 22:42:45 Function: TestOnDeleteSrcMutationsWithTimer delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnDeleteSrcMutationsWithTimer deleting in the background"
}
2021/12/01 22:42:46 Context: Bucket flush, response code: 200 dump: 
2021/12/01 22:42:56 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/01 22:42:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 22:42:58 Context: Bucket flush, response code: 200 dump: 
2021/12/01 22:43:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnDeleteSrcMutationsWithTimer (101.52s)
=== RUN   TestInterHandlerRecursion
2021/12/01 22:43:13 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/01 22:43:13 App: TestInterHandlerRecursion_source_bucket_update_op got un-deployed
2021/12/01 22:43:13 No eventing-con process running
2021/12/01 22:43:13 Function: TestInterHandlerRecursion_source_bucket_update_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestInterHandlerRecursion_source_bucket_update_op not found"
 }
}
2021/12/01 22:43:14 Context: Bucket flush, response code: 200 dump: 
2021/12/01 22:43:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 22:43:21 Context: Bucket flush, response code: 200 dump: 
2021/12/01 22:43:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 22:43:31 Waiting for app: TestInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/01 22:43:31 App: TestInterHandlerRecursion_src_bucket_op_on_delete got un-deployed
2021/12/01 22:43:31 No eventing-con process running
2021/12/01 22:43:31 Function: TestInterHandlerRecursion_src_bucket_op_on_delete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestInterHandlerRecursion_src_bucket_op_on_delete not found"
 }
}
2021/12/01 22:43:32 Context: Bucket flush, response code: 200 dump: 
2021/12/01 22:43:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 22:43:39 Context: Bucket flush, response code: 200 dump: 
2021/12/01 22:43:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 22:43:44 Deploying app: TestInterHandlerRecursion_source_bucket_update_op. Item count src bucket: 0 dst bucket: 0
2021/12/01 22:43:46 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterHandlerRecursion_source_bucket_update_op' in metakv",
  "warnings": null
 }
}
2021/12/01 22:43:46 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterHandlerRecursion_source_bucket_update_op' in metakv",
  "warnings": null
 }
} err 
2021/12/01 22:43:51 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/01 22:43:56 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/01 22:44:01 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/01 22:44:06 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/01 22:44:06 App: TestInterHandlerRecursion_source_bucket_update_op got deployed
2021/12/01 22:44:06 Deploying app: TestInterHandlerRecursion_src_bucket_op_on_delete. Item count src bucket: 0 dst bucket: 0
2021/12/01 22:44:06 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_INTER_FUNCTION_RECURSION",
 "code": 50,
 "description": "Inter function recursion error, only one function is allowed to do source bucket mutation/delete on a bucket",
 "attributes": null,
 "runtime_info": {
  "code": 50,
  "info": "Inter handler recursion error"
 }
}
2021/12/01 22:44:06 response body {
 "name": "ERR_INTER_FUNCTION_RECURSION",
 "code": 50,
 "description": "Inter function recursion error, only one function is allowed to do source bucket mutation/delete on a bucket",
 "attributes": null,
 "runtime_info": {
  "code": 50,
  "info": "Inter handler recursion error"
 }
} err 
2021/12/01 22:44:06 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 22:44:11 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/01 22:44:16 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/01 22:44:21 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/01 22:44:26 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/01 22:44:26 App: TestInterHandlerRecursion_source_bucket_update_op got un-deployed
2021/12/01 22:44:26 No eventing-con process running
2021/12/01 22:44:26 Function: TestInterHandlerRecursion_source_bucket_update_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestInterHandlerRecursion_source_bucket_update_op deleting in the background"
}
2021/12/01 22:44:27 Context: Bucket flush, response code: 200 dump: 
2021/12/01 22:44:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 22:44:34 Context: Bucket flush, response code: 200 dump: 
2021/12/01 22:44:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 22:44:44 Waiting for app: TestInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/01 22:44:44 App: TestInterHandlerRecursion_src_bucket_op_on_delete got un-deployed
2021/12/01 22:44:44 No eventing-con process running
2021/12/01 22:44:44 Function: TestInterHandlerRecursion_src_bucket_op_on_delete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestInterHandlerRecursion_src_bucket_op_on_delete not found"
 }
}
2021/12/01 22:44:45 Context: Bucket flush, response code: 200 dump: 
2021/12/01 22:44:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 22:44:52 Context: Bucket flush, response code: 200 dump: 
2021/12/01 22:44:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestInterHandlerRecursion (114.53s)
=== RUN   TestInterBucketRecursion
2021/12/01 22:45:02 Deploying app: TestInterBucketRecursion_function1. Item count src bucket: 0 dst bucket: 0
2021/12/01 22:45:04 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterBucketRecursion_function1' in metakv",
  "warnings": null
 }
}
2021/12/01 22:45:04 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterBucketRecursion_function1' in metakv",
  "warnings": null
 }
} err 
2021/12/01 22:45:04 Deploying app: TestInterBucketRecursion_function2. Item count src bucket: 0 dst bucket: 0
2021/12/01 22:45:04 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_INTER_BUCKET_RECURSION",
 "code": 51,
 "description": "Inter bucket recursion error, deployment of current handler will cause inter bucket recursion",
 "attributes": null,
 "runtime_info": {
  "code": 51,
  "info": "Inter bucket recursion error; function: TestInterBucketRecursion_function2 causes a cycle involving functions: [TestInterBucketRecursion_function1], hence deployment is disallowed"
 }
}
2021/12/01 22:45:09 Waiting for app: TestInterBucketRecursion_function1 to get deployed
2021/12/01 22:45:14 Waiting for app: TestInterBucketRecursion_function1 to get deployed
2021/12/01 22:45:19 Waiting for app: TestInterBucketRecursion_function1 to get deployed
2021/12/01 22:45:24 Waiting for app: TestInterBucketRecursion_function1 to get deployed
2021/12/01 22:45:24 App: TestInterBucketRecursion_function1 got deployed
2021/12/01 22:45:24 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 22:45:29 Waiting for app: TestInterBucketRecursion_function1 to get un-deployed
2021/12/01 22:45:34 Waiting for app: TestInterBucketRecursion_function1 to get un-deployed
2021/12/01 22:45:39 Waiting for app: TestInterBucketRecursion_function1 to get un-deployed
2021/12/01 22:45:44 Waiting for app: TestInterBucketRecursion_function1 to get un-deployed
2021/12/01 22:45:44 App: TestInterBucketRecursion_function1 got un-deployed
2021/12/01 22:45:44 No eventing-con process running
2021/12/01 22:45:44 Function: TestInterBucketRecursion_function1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestInterBucketRecursion_function1 deleting in the background"
}
2021/12/01 22:45:45 Context: Bucket flush, response code: 200 dump: 
2021/12/01 22:45:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 22:45:52 Context: Bucket flush, response code: 200 dump: 
2021/12/01 22:45:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 22:46:02 Waiting for app: TestInterBucketRecursion_function2 to get un-deployed
2021/12/01 22:46:02 App: TestInterBucketRecursion_function2 got un-deployed
2021/12/01 22:46:02 No eventing-con process running
2021/12/01 22:46:02 Function: TestInterBucketRecursion_function2 delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestInterBucketRecursion_function2 not found"
 }
}
2021/12/01 22:46:03 Context: Bucket flush, response code: 200 dump: 
2021/12/01 22:46:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 22:46:10 Context: Bucket flush, response code: 200 dump: 
2021/12/01 22:46:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestInterBucketRecursion (77.73s)
=== RUN   TestLargeHandler
2021/12/01 22:46:15 Changed force_compress value to false
2021/12/01 22:46:15 Deploying app: TestLargeHandler_function. Item count src bucket: 0 dst bucket: 0
2021/12/01 22:46:15 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_APPCODE_SIZE",
 "code": 39,
 "description": "Function Code size is more than the configured limit",
 "attributes": null,
 "runtime_info": {
  "code": 39,
  "info": "Function: TestLargeHandler_function handler Code size is more than 131072. Code Size: 131768"
 }
}
2021/12/01 22:46:15 Changed force_compress value to true
2021/12/01 22:46:15 Deploying app: TestLargeHandler_function. Item count src bucket: 0 dst bucket: 0
2021/12/01 22:46:17 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestLargeHandler_function' in metakv",
  "warnings": null
 }
}
2021/12/01 22:46:22 Waiting for app: TestLargeHandler_function to get deployed
2021/12/01 22:46:27 Waiting for app: TestLargeHandler_function to get deployed
2021/12/01 22:46:32 Waiting for app: TestLargeHandler_function to get deployed
2021/12/01 22:46:37 Waiting for app: TestLargeHandler_function to get deployed
2021/12/01 22:46:37 App: TestLargeHandler_function got deployed
2021/12/01 22:46:37 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 22:46:42 Waiting for app: TestLargeHandler_function to get un-deployed
2021/12/01 22:46:47 Waiting for app: TestLargeHandler_function to get un-deployed
2021/12/01 22:46:52 Waiting for app: TestLargeHandler_function to get un-deployed
2021/12/01 22:46:57 Waiting for app: TestLargeHandler_function to get un-deployed
2021/12/01 22:46:57 App: TestLargeHandler_function got un-deployed
2021/12/01 22:46:57 No eventing-con process running
2021/12/01 22:46:57 Function: TestLargeHandler_function delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestLargeHandler_function deleting in the background"
}
2021/12/01 22:46:59 Context: Bucket flush, response code: 200 dump: 
2021/12/01 22:47:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 22:47:05 Context: Bucket flush, response code: 200 dump: 
2021/12/01 22:47:10 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestLargeHandler (55.35s)
=== RUN   TestAllowInterHandlerRecursion
2021/12/01 22:47:20 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/01 22:47:20 App: TestAllowInterHandlerRecursion_source_bucket_update_op got un-deployed
2021/12/01 22:47:20 No eventing-con process running
2021/12/01 22:47:20 Function: TestAllowInterHandlerRecursion_source_bucket_update_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestAllowInterHandlerRecursion_source_bucket_update_op not found"
 }
}
2021/12/01 22:47:22 Context: Bucket flush, response code: 200 dump: 
2021/12/01 22:47:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 22:47:29 Context: Bucket flush, response code: 200 dump: 
2021/12/01 22:47:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 22:47:39 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/01 22:47:39 App: TestAllowInterHandlerRecursion_src_bucket_op_on_delete got un-deployed
2021/12/01 22:47:39 No eventing-con process running
2021/12/01 22:47:39 Function: TestAllowInterHandlerRecursion_src_bucket_op_on_delete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestAllowInterHandlerRecursion_src_bucket_op_on_delete not found"
 }
}
2021/12/01 22:47:40 Context: Bucket flush, response code: 200 dump: 
2021/12/01 22:47:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 22:47:47 Context: Bucket flush, response code: 200 dump: 
2021/12/01 22:47:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 22:47:52 Deploying app: TestAllowInterHandlerRecursion_source_bucket_update_op. Item count src bucket: 0 dst bucket: 0
2021/12/01 22:47:53 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterHandlerRecursion_source_bucket_update_op' in metakv",
  "warnings": null
 }
}
2021/12/01 22:47:53 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterHandlerRecursion_source_bucket_update_op' in metakv",
  "warnings": null
 }
} err 
2021/12/01 22:47:58 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/01 22:48:03 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/01 22:48:08 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/01 22:48:13 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/01 22:48:13 App: TestAllowInterHandlerRecursion_source_bucket_update_op got deployed
2021/12/01 22:48:13 Deploying app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete. Item count src bucket: 0 dst bucket: 0
2021/12/01 22:48:14 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterHandlerRecursion_src_bucket_op_on_delete' in metakv",
  "warnings": null
 }
}
2021/12/01 22:48:14 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterHandlerRecursion_src_bucket_op_on_delete' in metakv",
  "warnings": null
 }
} err 
2021/12/01 22:48:19 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get deployed
2021/12/01 22:48:24 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get deployed
2021/12/01 22:48:29 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get deployed
2021/12/01 22:48:34 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get deployed
2021/12/01 22:48:34 App: TestAllowInterHandlerRecursion_src_bucket_op_on_delete got deployed
2021/12/01 22:48:34 TestAllowInterHandlerRecursion_src_bucket_op_on_delete is deployed
2021/12/01 22:48:34 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 22:48:39 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/01 22:48:44 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/01 22:48:50 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/01 22:48:55 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/01 22:48:55 App: TestAllowInterHandlerRecursion_source_bucket_update_op got un-deployed
2021/12/01 22:48:55 No eventing-con process running
2021/12/01 22:48:55 Function: TestAllowInterHandlerRecursion_source_bucket_update_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAllowInterHandlerRecursion_source_bucket_update_op deleting in the background"
}
2021/12/01 22:48:57 Context: Bucket flush, response code: 200 dump: 
2021/12/01 22:49:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 22:49:04 Context: Bucket flush, response code: 200 dump: 
2021/12/01 22:49:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 22:49:09 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 22:49:14 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/01 22:49:19 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/01 22:49:24 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/01 22:49:29 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/01 22:49:29 App: TestAllowInterHandlerRecursion_src_bucket_op_on_delete got un-deployed
2021/12/01 22:49:29 No eventing-con process running
2021/12/01 22:49:29 Function: TestAllowInterHandlerRecursion_src_bucket_op_on_delete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAllowInterHandlerRecursion_src_bucket_op_on_delete deleting in the background"
}
2021/12/01 22:49:30 Context: Bucket flush, response code: 200 dump: 
2021/12/01 22:49:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 22:49:37 Context: Bucket flush, response code: 200 dump: 
2021/12/01 22:49:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAllowInterHandlerRecursion (151.71s)
=== RUN   TestAllowInterBucketRecursion
2021/12/01 22:49:47 Deploying app: TestAllowInterBucketRecursion_function1. Item count src bucket: 0 dst bucket: 0
2021/12/01 22:49:48 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterBucketRecursion_function1' in metakv",
  "warnings": null
 }
}
2021/12/01 22:49:48 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterBucketRecursion_function1' in metakv",
  "warnings": null
 }
} err 
2021/12/01 22:49:53 Waiting for app: TestAllowInterBucketRecursion_function1 to get deployed
2021/12/01 22:49:58 Waiting for app: TestAllowInterBucketRecursion_function1 to get deployed
2021/12/01 22:50:03 Waiting for app: TestAllowInterBucketRecursion_function1 to get deployed
2021/12/01 22:50:08 Waiting for app: TestAllowInterBucketRecursion_function1 to get deployed
2021/12/01 22:50:08 App: TestAllowInterBucketRecursion_function1 got deployed
2021/12/01 22:50:08 Deploying app: TestAllowInterBucketRecursion_function2. Item count src bucket: 0 dst bucket: 0
2021/12/01 22:50:10 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterBucketRecursion_function2' in metakv",
  "warnings": null
 }
}
2021/12/01 22:50:10 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterBucketRecursion_function2' in metakv",
  "warnings": null
 }
} err 
2021/12/01 22:50:15 Waiting for app: TestAllowInterBucketRecursion_function2 to get deployed
2021/12/01 22:50:20 Waiting for app: TestAllowInterBucketRecursion_function2 to get deployed
2021/12/01 22:50:25 Waiting for app: TestAllowInterBucketRecursion_function2 to get deployed
2021/12/01 22:50:30 Waiting for app: TestAllowInterBucketRecursion_function2 to get deployed
2021/12/01 22:50:30 App: TestAllowInterBucketRecursion_function2 got deployed
2021/12/01 22:50:30 TestAllowInterBucketRecursion_function2 is deployed
2021/12/01 22:50:30 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 22:50:35 Waiting for app: TestAllowInterBucketRecursion_function1 to get un-deployed
2021/12/01 22:50:40 Waiting for app: TestAllowInterBucketRecursion_function1 to get un-deployed
2021/12/01 22:50:45 Waiting for app: TestAllowInterBucketRecursion_function1 to get un-deployed
2021/12/01 22:50:50 Waiting for app: TestAllowInterBucketRecursion_function1 to get un-deployed
2021/12/01 22:50:50 App: TestAllowInterBucketRecursion_function1 got un-deployed
2021/12/01 22:50:50 No eventing-con process running
2021/12/01 22:50:50 Function: TestAllowInterBucketRecursion_function1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAllowInterBucketRecursion_function1 deleting in the background"
}
2021/12/01 22:50:51 Context: Bucket flush, response code: 200 dump: 
2021/12/01 22:50:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 22:50:59 Context: Bucket flush, response code: 200 dump: 
2021/12/01 22:51:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 22:51:05 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 22:51:10 Waiting for app: TestAllowInterBucketRecursion_function2 to get un-deployed
2021/12/01 22:51:15 Waiting for app: TestAllowInterBucketRecursion_function2 to get un-deployed
2021/12/01 22:51:20 Waiting for app: TestAllowInterBucketRecursion_function2 to get un-deployed
2021/12/01 22:51:25 Waiting for app: TestAllowInterBucketRecursion_function2 to get un-deployed
2021/12/01 22:51:25 App: TestAllowInterBucketRecursion_function2 got un-deployed
2021/12/01 22:51:25 No eventing-con process running
2021/12/01 22:51:25 Function: TestAllowInterBucketRecursion_function2 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAllowInterBucketRecursion_function2 deleting in the background"
}
2021/12/01 22:51:26 Context: Bucket flush, response code: 200 dump: 
2021/12/01 22:51:31 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 22:51:33 Context: Bucket flush, response code: 200 dump: 
2021/12/01 22:51:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAllowInterBucketRecursion (115.64s)
=== RUN   TestInterBucketRecursion2
2021/12/01 22:51:44 Context: Bucket flush, response code: 200 dump: 
2021/12/01 22:51:50 Context: Bucket flush, response code: 200 dump: 
2021/12/01 22:51:56 Context: Bucket flush, response code: 200 dump: 
2021/12/01 22:52:02 Context: Bucket flush, response code: 200 dump: 
2021/12/01 22:52:07 Deploying app: TestInterBucketRecursion2_function1. Item count src bucket: 0 dst bucket: 0
2021/12/01 22:52:08 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterBucketRecursion2_function1' in metakv",
  "warnings": null
 }
}
2021/12/01 22:52:13 Waiting for app: TestInterBucketRecursion2_function1 to get deployed
2021/12/01 22:52:18 Waiting for app: TestInterBucketRecursion2_function1 to get deployed
2021/12/01 22:52:23 Waiting for app: TestInterBucketRecursion2_function1 to get deployed
2021/12/01 22:52:28 Waiting for app: TestInterBucketRecursion2_function1 to get deployed
2021/12/01 22:52:28 App: TestInterBucketRecursion2_function1 got deployed
2021/12/01 22:52:28 Deploying app: TestInterBucketRecursion2_function2. Item count src bucket: 0 dst bucket: 0
2021/12/01 22:52:29 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterBucketRecursion2_function2' in metakv",
  "warnings": null
 }
}
2021/12/01 22:52:34 Waiting for app: TestInterBucketRecursion2_function2 to get deployed
2021/12/01 22:52:39 Waiting for app: TestInterBucketRecursion2_function2 to get deployed
2021/12/01 22:52:44 Waiting for app: TestInterBucketRecursion2_function2 to get deployed
2021/12/01 22:52:49 Waiting for app: TestInterBucketRecursion2_function2 to get deployed
2021/12/01 22:52:49 App: TestInterBucketRecursion2_function2 got deployed
2021/12/01 22:52:49 Deploying app: TestInterBucketRecursion2_function3. Item count src bucket: 0 dst bucket: 0
2021/12/01 22:52:50 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterBucketRecursion2_function3' in metakv",
  "warnings": null
 }
}
2021/12/01 22:52:55 Waiting for app: TestInterBucketRecursion2_function3 to get deployed
2021/12/01 22:53:00 Waiting for app: TestInterBucketRecursion2_function3 to get deployed
2021/12/01 22:53:05 Waiting for app: TestInterBucketRecursion2_function3 to get deployed
2021/12/01 22:53:10 Waiting for app: TestInterBucketRecursion2_function3 to get deployed
2021/12/01 22:53:10 App: TestInterBucketRecursion2_function3 got deployed
2021/12/01 22:53:11 Deploying app: TestInterBucketRecursion2_function4. Item count src bucket: 0 dst bucket: 0
2021/12/01 22:53:11 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_INTER_BUCKET_RECURSION",
 "code": 51,
 "description": "Inter bucket recursion error, deployment of current handler will cause inter bucket recursion",
 "attributes": null,
 "runtime_info": {
  "code": 51,
  "info": "Inter bucket recursion error; function: TestInterBucketRecursion2_function4 causes a cycle involving functions: [TestInterBucketRecursion2_function1 TestInterBucketRecursion2_function2 TestInterBucketRecursion2_function3], hence deployment is disallowed"
 }
}
2021/12/01 22:53:11 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 22:53:16 Waiting for app: TestInterBucketRecursion2_function1 to get un-deployed
2021/12/01 22:53:21 Waiting for app: TestInterBucketRecursion2_function1 to get un-deployed
2021/12/01 22:53:26 Waiting for app: TestInterBucketRecursion2_function1 to get un-deployed
2021/12/01 22:53:31 Waiting for app: TestInterBucketRecursion2_function1 to get un-deployed
2021/12/01 22:53:31 App: TestInterBucketRecursion2_function1 got un-deployed
2021/12/01 22:53:31 No eventing-con process running
2021/12/01 22:53:31 Function: TestInterBucketRecursion2_function1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestInterBucketRecursion2_function1 deleting in the background"
}
2021/12/01 22:53:32 Context: Bucket flush, response code: 200 dump: 
2021/12/01 22:53:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 22:53:39 Context: Bucket flush, response code: 200 dump: 
2021/12/01 22:53:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 22:53:44 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 22:53:49 Waiting for app: TestInterBucketRecursion2_function2 to get un-deployed
2021/12/01 22:53:54 Waiting for app: TestInterBucketRecursion2_function2 to get un-deployed
2021/12/01 22:53:59 Waiting for app: TestInterBucketRecursion2_function2 to get un-deployed
2021/12/01 22:54:04 Waiting for app: TestInterBucketRecursion2_function2 to get un-deployed
2021/12/01 22:54:04 App: TestInterBucketRecursion2_function2 got un-deployed
2021/12/01 22:54:04 No eventing-con process running
2021/12/01 22:54:04 Function: TestInterBucketRecursion2_function2 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestInterBucketRecursion2_function2 deleting in the background"
}
2021/12/01 22:54:06 Context: Bucket flush, response code: 200 dump: 
2021/12/01 22:54:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 22:54:13 Context: Bucket flush, response code: 200 dump: 
2021/12/01 22:54:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 22:54:18 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 22:54:23 Waiting for app: TestInterBucketRecursion2_function3 to get un-deployed
2021/12/01 22:54:28 Waiting for app: TestInterBucketRecursion2_function3 to get un-deployed
2021/12/01 22:54:33 Waiting for app: TestInterBucketRecursion2_function3 to get un-deployed
2021/12/01 22:54:38 Waiting for app: TestInterBucketRecursion2_function3 to get un-deployed
2021/12/01 22:54:38 App: TestInterBucketRecursion2_function3 got un-deployed
2021/12/01 22:54:38 No eventing-con process running
2021/12/01 22:54:38 Function: TestInterBucketRecursion2_function3 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestInterBucketRecursion2_function3 deleting in the background"
}
2021/12/01 22:54:39 Context: Bucket flush, response code: 200 dump: 
2021/12/01 22:54:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 22:54:46 Context: Bucket flush, response code: 200 dump: 
2021/12/01 22:54:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 22:54:56 Waiting for app: TestInterBucketRecursion2_function4 to get un-deployed
2021/12/01 22:54:56 App: TestInterBucketRecursion2_function4 got un-deployed
2021/12/01 22:54:56 No eventing-con process running
2021/12/01 22:54:56 Function: TestInterBucketRecursion2_function4 delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestInterBucketRecursion2_function4 not found"
 }
}
2021/12/01 22:54:57 Context: Bucket flush, response code: 200 dump: 
2021/12/01 22:55:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 22:55:04 Context: Bucket flush, response code: 200 dump: 
2021/12/01 22:55:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestInterBucketRecursion2 (213.39s)
=== RUN   TestBucketDeleteWithRebOut
2021/12/01 22:55:16 Removing node(s): [127.0.0.1:19001 127.0.0.1:19002 127.0.0.1:19003] from the cluster
2021/12/01 22:55:21 Rebalance progress: 100

VB distribution across eventing nodes and their workers::

2021/12/01 22:55:24 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:58} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/01 22:55:39 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:58} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/01 22:55:54 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:58} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/01 22:56:09 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:58} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/01 22:56:24 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:58} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/01 22:56:39 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:58} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/01 22:56:54 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:58} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/01 22:57:09 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:58} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/01 22:57:24 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:58} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/01 22:57:39 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:58} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/01 22:57:54 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:58} {Name:#primary Bucket:hello-world Completion:100}]

DCP Stream statuses:

vbucket curr owner:


2021/12/01 22:58:09 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 22:58:09 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/01 22:58:10 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/01 22:58:15 Rebalance progress: 100

VB distribution across eventing nodes and their workers::


DCP Stream statuses:

vbucket curr owner:


2021/12/01 22:58:15 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 22:58:15 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/01 22:58:15 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/01 22:58:20 Rebalance progress: 100

VB distribution across eventing nodes and their workers::


DCP Stream statuses:

vbucket curr owner:


2021/12/01 22:58:20 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 22:58:25 clusterSetup.go:397: Waiting for app: TestBucketDeleteWithRebOut to get un-deployed
2021/12/01 22:58:25 clusterSetup.go:405: App: TestBucketDeleteWithRebOut got un-deployed
2021/12/01 22:58:25 eventingSetup.go:46: No eventing-con process running
2021/12/01 22:58:25 eventingSetup.go:418: Function: TestBucketDeleteWithRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestBucketDeleteWithRebOut not found"
 }
}
2021/12/01 22:58:26 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/01 22:58:31 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 22:58:33 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/01 22:58:38 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 22:58:43 eventingSetup.go:125: Deploying app: TestBucketDeleteWithRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/01 22:58:44 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestBucketDeleteWithRebOut' in metakv",
  "warnings": null
 }
}
2021/12/01 22:58:49 clusterSetup.go:346: Waiting for app: TestBucketDeleteWithRebOut to get deployed
2021/12/01 22:58:54 clusterSetup.go:346: Waiting for app: TestBucketDeleteWithRebOut to get deployed
2021/12/01 22:58:59 clusterSetup.go:346: Waiting for app: TestBucketDeleteWithRebOut to get deployed
2021/12/01 22:59:04 clusterSetup.go:346: Waiting for app: TestBucketDeleteWithRebOut to get deployed
2021/12/01 22:59:04 clusterSetup.go:354: App: TestBucketDeleteWithRebOut got deployed

VB distribution across eventing nodes and their workers::


DCP Stream statuses:
	stream status: running
	len: 1024
	vb list dump: [0-1023]

vbucket curr owner:
Producer node: 127.0.0.1:9302	Node UUID: 8fba8ff49cea46a70914fc3ca45a5ed0
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [0-113]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [114-227]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 114
	v: [228-341]
Producer node: 172.31.2.134:9300	Node UUID: 21889364c47a615566e201ad2e8a9316
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 113
	v: [911-1023]
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [683-796]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [797-910]
Producer node: 127.0.0.1:9303	Node UUID: 60cefba8cf95278cc87f2b36a423c038
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [342-455]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [456-569]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 113
	v: [570-682]


2021/12/01 22:59:05 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 22:59:05 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/01 22:59:10 Rebalance progress: 29.16666666666666
2021/12/01 22:59:15 Rebalance progress: 29.16666666666666
2021/12/01 22:59:20 Rebalance progress: 100

VB distribution across eventing nodes and their workers::


DCP Stream statuses:
	stream status: running
	len: 1024
	vb list dump: [0-1023]

vbucket curr owner:
Producer node: 172.31.2.134:9300	Node UUID: 21889364c47a615566e201ad2e8a9316
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [683-796]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [797-910]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 113
	v: [911-1023]
Producer node: 127.0.0.1:9303	Node UUID: 60cefba8cf95278cc87f2b36a423c038
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [342-455]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [456-569]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 113
	v: [570-682]
Producer node: 127.0.0.1:9302	Node UUID: 8fba8ff49cea46a70914fc3ca45a5ed0
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 114
	v: [228-341]
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [0-113]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [114-227]


2021/12/01 22:59:20 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 22:59:20 handler_test.go:1674: Deleting source bucket: default
2021/12/01 22:59:22 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/01 22:59:22 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/01 22:59:27 Rebalance progress: 29.16666666666666
2021/12/01 22:59:32 Rebalance progress: 29.16666666666666
2021/12/01 22:59:37 Rebalance progress: 100

VB distribution across eventing nodes and their workers::


DCP Stream statuses:
	stream status: stopped
	len: 132
	vb list dump: [0-1, 4, 6, 10, 15, 26, 29, 31, 39, 43, 52, 64-65, 83, 94, 114, 117, 128, 140, 143, 149, 172-173, 185, 192, 197, 217, 221, 234, 245, 248, 253, 262, 265, 270, 274, 283, 285, 295, 300, 305, 325, 329, 343, 346, 354, 365, 373, 377, 380, 383, 388, 390, 411, 439, 455, 513, 519, 521, 545, 547, 555, 557-558, 561-562, 566-569, 571, 579, 594-595, 598, 603, 611, 641, 647, 671-672, 675, 680, 691, 705, 713, 715, 729, 731-732, 738-739, 746-747, 774-775, 783, 790-791, 793, 809, 811, 818, 826, 843, 845, 849-850, 858-859, 865, 867, 873, 883, 891, 897, 920, 928, 930, 939, 948-949, 958, 969, 979, 983-985, 994, 1004, 1010]
	stream status: running
	len: 892
	vb list dump: [2-3, 5, 7-9, 11-14, 16-25, 27-28, 30, 32-38, 40-42, 44-51, 53-63, 66-82, 84-93, 95-113, 115-116, 118-127, 129-139, 141-142, 144-148, 150-171, 174-184, 186-191, 193-196, 198-216, 218-220, 222-233, 235-244, 246-247, 249-252, 254-261, 263-264, 266-269, 271-273, 275-282, 284, 286-294, 296-299, 301-304, 306-324, 326-328, 330-342, 344-345, 347-353, 355-364, 366-372, 374-376, 378-379, 381-382, 384-387, 389, 391-410, 412-438, 440-454, 456-512, 514-518, 520, 522-544, 546, 548-554, 556, 559-560, 563-565, 570, 572-578, 580-593, 596-597, 599-602, 604-610, 612-640, 642-646, 648-670, 673-674, 676-679, 681-690, 692-704, 706-712, 714, 716-728, 730, 733-737, 740-745, 748-773, 776-782, 784-789, 792, 794-808, 810, 812-817, 819-825, 827-842, 844, 846-848, 851-857, 860-864, 866, 868-872, 874-882, 884-890, 892-896, 898-919, 921-927, 929, 931-938, 940-947, 950-957, 959-968, 970-978, 980-982, 986-993, 995-1003, 1005-1009, 1011-1023]

vbucket curr owner:
Producer node: 127.0.0.1:9302	Node UUID: 8fba8ff49cea46a70914fc3ca45a5ed0
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 98
	v: [2-3, 5, 7-9, 11-14, 16-25, 27-28, 30, 32-38, 40-42, 44-51, 53-63, 66-82, 84-93, 95-113]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 101
	v: [115-116, 118-127, 129-139, 141-142, 144-148, 150-171, 174-184, 186-191, 193-196, 198-216, 218-220, 222-227]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 99
	v: [228-233, 235-244, 246-247, 249-252, 254-261, 263-264, 266-269, 271-273, 275-282, 284, 286-294, 296-299, 301-304, 306-324, 326-328, 330-341]
Producer node: 172.31.2.134:9300	Node UUID: 21889364c47a615566e201ad2e8a9316
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 97
	v: [683-690, 692-704, 706-712, 714, 716-728, 730, 733-737, 740-745, 748-773, 776-782, 784-789, 792, 794-796]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 98
	v: [797-808, 810, 812-817, 819-825, 827-842, 844, 846-848, 851-857, 860-864, 866, 868-872, 874-882, 884-890, 892-896, 898-910]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 98
	v: [911-919, 921-927, 929, 931-938, 940-947, 950-957, 959-968, 970-978, 980-982, 986-993, 995-1003, 1005-1009, 1011-1023]
Producer node: 127.0.0.1:9303	Node UUID: 60cefba8cf95278cc87f2b36a423c038
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 101
	v: [342, 344-345, 347-353, 355-364, 366-372, 374-376, 378-379, 381-382, 384-387, 389, 391-410, 412-438, 440-454]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 100
	v: [456-512, 514-518, 520, 522-544, 546, 548-554, 556, 559-560, 563-565]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 100
	v: [570, 572-578, 580-593, 596-597, 599-602, 604-610, 612-640, 642-646, 648-670, 673-674, 676-679, 681-682]


2021/12/01 22:59:37 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 22:59:42 eventingSetup.go:719: Waiting for function: TestBucketDeleteWithRebOut status to change to undeployed
2021/12/01 22:59:42 eventingSetup.go:727: Function: TestBucketDeleteWithRebOut status changed to undeployed
2021/12/01 22:59:48 clusterSetup.go:397: Waiting for app: TestBucketDeleteWithRebOut to get un-deployed
2021/12/01 22:59:48 clusterSetup.go:405: App: TestBucketDeleteWithRebOut got un-deployed
2021/12/01 22:59:48 eventingSetup.go:46: No eventing-con process running
2021/12/01 22:59:48 eventingSetup.go:418: Function: TestBucketDeleteWithRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestBucketDeleteWithRebOut deleting in the background"
}
2021/12/01 22:59:49 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/01 22:59:54 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 22:59:56 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/01 23:00:01 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 23:00:01 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv,index,n1ql to the cluster
2021/12/01 23:00:01 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/01 23:00:06 Rebalance progress: 100
--- PASS: TestBucketDeleteWithRebOut (297.70s)
=== RUN   TestN1QLRecursion
2021/12/01 23:00:09 Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/01 23:00:09 Error in adding nodes : , response: ["Prepare join failed. Node is already part of cluster."]
2021/12/01 23:00:09 Removing node(s): [] from the cluster
2021/12/01 23:00:14 Rebalance progress: 100
2021/12/01 23:00:20 Context: Bucket flush, response code: 200 dump: 
2021/12/01 23:00:26 Context: Bucket flush, response code: 200 dump: 
2021/12/01 23:00:32 Context: Bucket flush, response code: 200 dump: 
2021/12/01 23:00:37 Deploying app: TestN1QLRecursion_function1. Item count src bucket: 0 dst bucket: 0
2021/12/01 23:00:39 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLRecursion_function1' in metakv",
  "warnings": null
 }
}
2021/12/01 23:00:39 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLRecursion_function1' in metakv",
  "warnings": null
 }
} err 
2021/12/01 23:00:44 Waiting for app: TestN1QLRecursion_function1 to get deployed
2021/12/01 23:00:49 Waiting for app: TestN1QLRecursion_function1 to get deployed
2021/12/01 23:00:54 Waiting for app: TestN1QLRecursion_function1 to get deployed
2021/12/01 23:00:59 Waiting for app: TestN1QLRecursion_function1 to get deployed
2021/12/01 23:01:04 Waiting for app: TestN1QLRecursion_function1 to get deployed
2021/12/01 23:01:09 Waiting for app: TestN1QLRecursion_function1 to get deployed
2021/12/01 23:01:14 Waiting for app: TestN1QLRecursion_function1 to get deployed
2021/12/01 23:01:14 App: TestN1QLRecursion_function1 got deployed
2021/12/01 23:01:14 TestN1QLRecursion_function1 is deployed
2021/12/01 23:01:14 Deploying app: TestN1QLRecursion_function3. Item count src bucket: 0 dst bucket: 0
2021/12/01 23:01:15 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLRecursion_function3' in metakv",
  "warnings": null
 }
}
2021/12/01 23:01:15 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLRecursion_function3' in metakv",
  "warnings": null
 }
} err 
2021/12/01 23:01:20 Waiting for app: TestN1QLRecursion_function3 to get deployed
2021/12/01 23:01:25 Waiting for app: TestN1QLRecursion_function3 to get deployed
2021/12/01 23:01:30 Waiting for app: TestN1QLRecursion_function3 to get deployed
2021/12/01 23:01:35 Waiting for app: TestN1QLRecursion_function3 to get deployed
2021/12/01 23:01:35 App: TestN1QLRecursion_function3 got deployed
2021/12/01 23:01:35 TestN1QLRecursion_function3 is deployed
2021/12/01 23:01:35 Deploying app: TestN1QLRecursion_function2. Item count src bucket: 0 dst bucket: 0
2021/12/01 23:01:35 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_INTER_BUCKET_RECURSION",
 "code": 51,
 "description": "Inter bucket recursion error, deployment of current handler will cause inter bucket recursion",
 "attributes": null,
 "runtime_info": {
  "code": 51,
  "info": "Inter bucket recursion error; function: TestN1QLRecursion_function2 causes a cycle involving functions: [TestN1QLRecursion_function3 TestN1QLRecursion_function1], hence deployment is disallowed"
 }
}
2021/12/01 23:01:35 response body {
 "name": "ERR_INTER_BUCKET_RECURSION",
 "code": 51,
 "description": "Inter bucket recursion error, deployment of current handler will cause inter bucket recursion",
 "attributes": null,
 "runtime_info": {
  "code": 51,
  "info": "Inter bucket recursion error; function: TestN1QLRecursion_function2 causes a cycle involving functions: [TestN1QLRecursion_function3 TestN1QLRecursion_function1], hence deployment is disallowed"
 }
} err 
2021/12/01 23:01:35 Success: TestN1QLRecursion_function2 is not deployed
2021/12/01 23:01:35 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 23:01:40 Waiting for app: TestN1QLRecursion_function1 to get un-deployed
2021/12/01 23:01:45 Waiting for app: TestN1QLRecursion_function1 to get un-deployed
2021/12/01 23:01:50 Waiting for app: TestN1QLRecursion_function1 to get un-deployed
2021/12/01 23:01:55 Waiting for app: TestN1QLRecursion_function1 to get un-deployed
2021/12/01 23:01:55 App: TestN1QLRecursion_function1 got un-deployed
2021/12/01 23:01:55 No eventing-con process running
2021/12/01 23:01:55 Function: TestN1QLRecursion_function1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLRecursion_function1 deleting in the background"
}
2021/12/01 23:01:57 Context: Bucket flush, response code: 200 dump: 
2021/12/01 23:02:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 23:02:04 Context: Bucket flush, response code: 200 dump: 
2021/12/01 23:02:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 23:02:09 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 23:02:14 Waiting for app: TestN1QLRecursion_function3 to get un-deployed
2021/12/01 23:02:19 Waiting for app: TestN1QLRecursion_function3 to get un-deployed
2021/12/01 23:02:24 Waiting for app: TestN1QLRecursion_function3 to get un-deployed
2021/12/01 23:02:29 Waiting for app: TestN1QLRecursion_function3 to get un-deployed
2021/12/01 23:02:29 App: TestN1QLRecursion_function3 got un-deployed
2021/12/01 23:02:29 No eventing-con process running
2021/12/01 23:02:29 Function: TestN1QLRecursion_function3 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLRecursion_function3 deleting in the background"
}
2021/12/01 23:02:31 Context: Bucket flush, response code: 200 dump: 
2021/12/01 23:02:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 23:02:38 Context: Bucket flush, response code: 200 dump: 
2021/12/01 23:02:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 23:02:44 Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/01 23:02:49 Rebalance progress: 100
--- PASS: TestN1QLRecursion (160.58s)
=== RUN   TestN1QLAllowRecursion
2021/12/01 23:02:49 Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/01 23:02:50 Error in adding nodes : , response: ["Prepare join failed. Node is already part of cluster."]
2021/12/01 23:02:50 Removing node(s): [] from the cluster
2021/12/01 23:02:55 Rebalance progress: 100
2021/12/01 23:03:00 Context: Bucket flush, response code: 500 dump: {"_":"Flush failed with unexpected error. Check server logs for details."}
2021/12/01 23:03:06 Context: Bucket flush, response code: 200 dump: 
2021/12/01 23:03:12 Context: Bucket flush, response code: 200 dump: 
2021/12/01 23:03:17 Deploying app: TestN1QLAllowRecursion_function1. Item count src bucket: 0 dst bucket: 0
2021/12/01 23:03:18 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function1' in metakv",
  "warnings": null
 }
}
2021/12/01 23:03:18 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function1' in metakv",
  "warnings": null
 }
} err 
2021/12/01 23:03:23 Waiting for app: TestN1QLAllowRecursion_function1 to get deployed
2021/12/01 23:03:28 Waiting for app: TestN1QLAllowRecursion_function1 to get deployed
2021/12/01 23:03:33 Waiting for app: TestN1QLAllowRecursion_function1 to get deployed
2021/12/01 23:03:39 Waiting for app: TestN1QLAllowRecursion_function1 to get deployed
2021/12/01 23:03:39 App: TestN1QLAllowRecursion_function1 got deployed
2021/12/01 23:03:39 TestN1QLAllowRecursion_function1 is deployed
2021/12/01 23:03:39 Deploying app: TestN1QLAllowRecursion_function3. Item count src bucket: 0 dst bucket: 0
2021/12/01 23:03:40 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function3' in metakv",
  "warnings": null
 }
}
2021/12/01 23:03:40 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function3' in metakv",
  "warnings": null
 }
} err 
2021/12/01 23:03:45 Waiting for app: TestN1QLAllowRecursion_function3 to get deployed
2021/12/01 23:03:50 Waiting for app: TestN1QLAllowRecursion_function3 to get deployed
2021/12/01 23:03:55 Waiting for app: TestN1QLAllowRecursion_function3 to get deployed
2021/12/01 23:04:00 Waiting for app: TestN1QLAllowRecursion_function3 to get deployed
2021/12/01 23:04:00 App: TestN1QLAllowRecursion_function3 got deployed
2021/12/01 23:04:00 TestN1QLAllowRecursion_function3 is deployed
2021/12/01 23:04:00 Deploying app: TestN1QLAllowRecursion_function2. Item count src bucket: 0 dst bucket: 0
2021/12/01 23:04:01 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function2' in metakv",
  "warnings": null
 }
}
2021/12/01 23:04:01 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function2' in metakv",
  "warnings": null
 }
} err 
2021/12/01 23:04:06 Waiting for app: TestN1QLAllowRecursion_function2 to get deployed
2021/12/01 23:04:11 Waiting for app: TestN1QLAllowRecursion_function2 to get deployed
2021/12/01 23:04:16 Waiting for app: TestN1QLAllowRecursion_function2 to get deployed
2021/12/01 23:04:21 Waiting for app: TestN1QLAllowRecursion_function2 to get deployed
2021/12/01 23:04:21 App: TestN1QLAllowRecursion_function2 got deployed
2021/12/01 23:04:21 Success: TestN1QLAllowRecursion_function2 is deployed after allow_interbucket_recursion
2021/12/01 23:04:21 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 23:04:26 Waiting for app: TestN1QLAllowRecursion_function1 to get un-deployed
2021/12/01 23:04:31 Waiting for app: TestN1QLAllowRecursion_function1 to get un-deployed
2021/12/01 23:04:36 Waiting for app: TestN1QLAllowRecursion_function1 to get un-deployed
2021/12/01 23:04:41 Waiting for app: TestN1QLAllowRecursion_function1 to get un-deployed
2021/12/01 23:04:41 App: TestN1QLAllowRecursion_function1 got un-deployed
2021/12/01 23:04:41 No eventing-con process running
2021/12/01 23:04:41 Function: TestN1QLAllowRecursion_function1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLAllowRecursion_function1 deleting in the background"
}
2021/12/01 23:04:43 Context: Bucket flush, response code: 200 dump: 
2021/12/01 23:04:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 23:04:50 Context: Bucket flush, response code: 200 dump: 
2021/12/01 23:04:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 23:04:55 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 23:05:00 Waiting for app: TestN1QLAllowRecursion_function2 to get un-deployed
2021/12/01 23:05:05 Waiting for app: TestN1QLAllowRecursion_function2 to get un-deployed
2021/12/01 23:05:10 Waiting for app: TestN1QLAllowRecursion_function2 to get un-deployed
2021/12/01 23:05:15 Waiting for app: TestN1QLAllowRecursion_function2 to get un-deployed
2021/12/01 23:05:15 App: TestN1QLAllowRecursion_function2 got un-deployed
2021/12/01 23:05:15 No eventing-con process running
2021/12/01 23:05:15 Function: TestN1QLAllowRecursion_function2 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLAllowRecursion_function2 deleting in the background"
}
2021/12/01 23:05:17 Context: Bucket flush, response code: 200 dump: 
2021/12/01 23:05:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 23:05:24 Context: Bucket flush, response code: 200 dump: 
2021/12/01 23:05:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 23:05:29 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 23:05:34 Waiting for app: TestN1QLAllowRecursion_function3 to get un-deployed
2021/12/01 23:05:39 Waiting for app: TestN1QLAllowRecursion_function3 to get un-deployed
2021/12/01 23:05:44 Waiting for app: TestN1QLAllowRecursion_function3 to get un-deployed
2021/12/01 23:05:49 Waiting for app: TestN1QLAllowRecursion_function3 to get un-deployed
2021/12/01 23:05:49 App: TestN1QLAllowRecursion_function3 got un-deployed
2021/12/01 23:05:49 No eventing-con process running
2021/12/01 23:05:49 Function: TestN1QLAllowRecursion_function3 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLAllowRecursion_function3 deleting in the background"
}
2021/12/01 23:05:51 Context: Bucket flush, response code: 200 dump: 
2021/12/01 23:05:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 23:05:58 Context: Bucket flush, response code: 200 dump: 
2021/12/01 23:06:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 23:06:04 Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/01 23:06:09 Rebalance progress: 100
--- PASS: TestN1QLAllowRecursion (200.00s)
=== RUN   TestOnDeleteExpiryBucketOp
2021/12/01 23:06:09 Sleeping for some time
2021/12/01 23:06:19 Waiting for app: TestOnDeleteExpiryBucketOp to get un-deployed
2021/12/01 23:06:19 App: TestOnDeleteExpiryBucketOp got un-deployed
2021/12/01 23:06:20 No eventing-con process running
2021/12/01 23:06:20 Function: TestOnDeleteExpiryBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnDeleteExpiryBucketOp not found"
 }
}
2021/12/01 23:06:21 Context: Bucket flush, response code: 200 dump: 
2021/12/01 23:06:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 23:06:28 Context: Bucket flush, response code: 200 dump: 
2021/12/01 23:06:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 23:06:33 Deploying app: TestOnDeleteExpiryBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/01 23:06:34 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnDeleteExpiryBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/01 23:06:34 Starting bucket ops to source bucket
2021/12/01 23:07:24 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 23:07:29 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 23:07:34 Waiting for dst bucket item count to get to: 5000 curr count: 2500
2021/12/01 23:07:34 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/01 23:07:34 Starting bucket ops to source bucket
2021/12/01 23:07:35 Starting bucket ops to source bucket
2021/12/01 23:08:41 Waiting for dst bucket item count to get to: 7000 curr count: 5000
2021/12/01 23:08:41 src & dst bucket item count matched up. src bucket count: 7000 dst bucket count: 7000
2021/12/01 23:08:41 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 23:08:41 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 23:08:46 Waiting for app: TestOnDeleteExpiryBucketOp to get un-deployed
2021/12/01 23:08:51 Waiting for app: TestOnDeleteExpiryBucketOp to get un-deployed
2021/12/01 23:08:56 Waiting for app: TestOnDeleteExpiryBucketOp to get un-deployed
2021/12/01 23:09:01 Waiting for app: TestOnDeleteExpiryBucketOp to get un-deployed
2021/12/01 23:09:01 App: TestOnDeleteExpiryBucketOp got un-deployed
2021/12/01 23:09:01 No eventing-con process running
2021/12/01 23:09:01 Function: TestOnDeleteExpiryBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnDeleteExpiryBucketOp deleting in the background"
}
2021/12/01 23:09:03 Context: Bucket flush, response code: 200 dump: 
2021/12/01 23:09:13 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/01 23:09:18 Waiting for dst bucket item count to get to: 0 curr count: 501
2021/12/01 23:09:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 23:09:20 Context: Bucket flush, response code: 200 dump: 
2021/12/01 23:09:30 Waiting for dst bucket item count to get to: 0 curr count: 3500
2021/12/01 23:09:35 Waiting for dst bucket item count to get to: 0 curr count: 3500
2021/12/01 23:09:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnDeleteExpiryBucketOp (205.06s)
=== RUN   TestTimerOverWriteSameReference
2021/12/01 23:09:40 Waiting for app: TestTimerOverWriteSameReference to get un-deployed
2021/12/01 23:09:40 App: TestTimerOverWriteSameReference got un-deployed
2021/12/01 23:09:40 No eventing-con process running
2021/12/01 23:09:40 Function: TestTimerOverWriteSameReference delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestTimerOverWriteSameReference not found"
 }
}
2021/12/01 23:09:41 Context: Bucket flush, response code: 200 dump: 
2021/12/01 23:09:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 23:09:48 Context: Bucket flush, response code: 200 dump: 
2021/12/01 23:09:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 23:09:53 Deploying app: TestTimerOverWriteSameReference. Item count src bucket: 0 dst bucket: 0
2021/12/01 23:09:54 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestTimerOverWriteSameReference' in metakv",
  "warnings": null
 }
}
2021/12/01 23:09:59 Waiting for app: TestTimerOverWriteSameReference to get deployed
2021/12/01 23:10:04 Waiting for app: TestTimerOverWriteSameReference to get deployed
2021/12/01 23:10:09 Waiting for app: TestTimerOverWriteSameReference to get deployed
2021/12/01 23:10:15 Waiting for app: TestTimerOverWriteSameReference to get deployed
2021/12/01 23:10:15 App: TestTimerOverWriteSameReference got deployed
2021/12/01 23:10:25 Starting bucket ops to source bucket
2021/12/01 23:11:26 src & dst bucket item count matched up. src bucket count: 1 dst bucket count: 1
2021/12/01 23:11:36 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 23:11:36 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 23:11:41 Waiting for app: TestTimerOverWriteSameReference to get un-deployed
2021/12/01 23:11:46 Waiting for app: TestTimerOverWriteSameReference to get un-deployed
2021/12/01 23:11:51 Waiting for app: TestTimerOverWriteSameReference to get un-deployed
2021/12/01 23:11:56 Waiting for app: TestTimerOverWriteSameReference to get un-deployed
2021/12/01 23:11:56 App: TestTimerOverWriteSameReference got un-deployed
2021/12/01 23:11:56 No eventing-con process running
2021/12/01 23:11:56 Function: TestTimerOverWriteSameReference delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestTimerOverWriteSameReference deleting in the background"
}
2021/12/01 23:11:58 Context: Bucket flush, response code: 200 dump: 
2021/12/01 23:12:08 Waiting for dst bucket item count to get to: 0 curr count: 500
2021/12/01 23:12:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 23:12:09 Context: Bucket flush, response code: 200 dump: 
2021/12/01 23:12:19 Waiting for dst bucket item count to get to: 0 curr count: 1
2021/12/01 23:12:24 Waiting for dst bucket item count to get to: 0 curr count: 1
2021/12/01 23:12:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestTimerOverWriteSameReference (169.94s)
=== RUN   TestCancelTimerBucketop
2021/12/01 23:12:29 Waiting for app: TestCancelTimerBucketop to get un-deployed
2021/12/01 23:12:29 App: TestCancelTimerBucketop got un-deployed
2021/12/01 23:12:30 No eventing-con process running
2021/12/01 23:12:30 Function: TestCancelTimerBucketop delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestCancelTimerBucketop not found"
 }
}
2021/12/01 23:12:31 Context: Bucket flush, response code: 200 dump: 
2021/12/01 23:12:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 23:12:38 Context: Bucket flush, response code: 200 dump: 
2021/12/01 23:12:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 23:12:43 Deploying app: TestCancelTimerBucketop. Item count src bucket: 0 dst bucket: 0
2021/12/01 23:12:44 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCancelTimerBucketop' in metakv",
  "warnings": null
 }
}
2021/12/01 23:12:49 Waiting for app: TestCancelTimerBucketop to get deployed
2021/12/01 23:12:54 Waiting for app: TestCancelTimerBucketop to get deployed
2021/12/01 23:12:59 Waiting for app: TestCancelTimerBucketop to get deployed
2021/12/01 23:13:04 Waiting for app: TestCancelTimerBucketop to get deployed
2021/12/01 23:13:04 App: TestCancelTimerBucketop got deployed
2021/12/01 23:13:04 Starting bucket ops to source bucket
2021/12/01 23:13:12 Starting bucket ops to source bucket
2021/12/01 23:14:14 src & dst bucket item count matched up. src bucket count: 1000 dst bucket count: 1000
2021/12/01 23:14:14 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 23:14:14 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 23:14:19 Waiting for app: TestCancelTimerBucketop to get un-deployed
2021/12/01 23:14:24 Waiting for app: TestCancelTimerBucketop to get un-deployed
2021/12/01 23:14:29 Waiting for app: TestCancelTimerBucketop to get un-deployed
2021/12/01 23:14:34 Waiting for app: TestCancelTimerBucketop to get un-deployed
2021/12/01 23:14:34 App: TestCancelTimerBucketop got un-deployed
2021/12/01 23:14:34 No eventing-con process running
2021/12/01 23:14:34 Function: TestCancelTimerBucketop delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCancelTimerBucketop deleting in the background"
}
2021/12/01 23:14:36 Context: Bucket flush, response code: 200 dump: 
2021/12/01 23:14:46 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/01 23:14:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 23:14:48 Context: Bucket flush, response code: 200 dump: 
2021/12/01 23:14:58 Waiting for dst bucket item count to get to: 0 curr count: 712
2021/12/01 23:15:03 Waiting for dst bucket item count to get to: 0 curr count: 212
2021/12/01 23:15:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCancelTimerBucketop (158.55s)
=== RUN   TestJSExpiryDate
2021/12/01 23:15:08 Starting bucket ops to source bucket
2021/12/01 23:15:26 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/01 23:15:26 Deploying app: TestJSExpiryDate. Item count src bucket: 5000 dst bucket: 0
2021/12/01 23:15:27 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestJSExpiryDate' in metakv",
  "warnings": null
 }
}
2021/12/01 23:15:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 23:15:57 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 23:15:57 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 23:16:02 Waiting for app: TestJSExpiryDate to get un-deployed
2021/12/01 23:16:08 Waiting for app: TestJSExpiryDate to get un-deployed
2021/12/01 23:16:13 Waiting for app: TestJSExpiryDate to get un-deployed
2021/12/01 23:16:18 Waiting for app: TestJSExpiryDate to get un-deployed
2021/12/01 23:16:18 App: TestJSExpiryDate got un-deployed
2021/12/01 23:16:18 No eventing-con process running
2021/12/01 23:16:18 Function: TestJSExpiryDate delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestJSExpiryDate deleting in the background"
}
2021/12/01 23:16:19 Context: Bucket flush, response code: 200 dump: 
2021/12/01 23:16:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 23:16:27 Context: Bucket flush, response code: 200 dump: 
2021/12/01 23:16:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestJSExpiryDate (88.77s)
=== RUN   TestBinaryDoc
2021/12/01 23:16:37 Starting bucket ops to source bucket
2021/12/01 23:16:37 Deploying app: TestBinaryDoc. Item count src bucket: 0 dst bucket: 0
2021/12/01 23:16:38 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestBinaryDoc' in metakv",
  "warnings": null
 }
}
2021/12/01 23:16:38 Starting bucket ops to source bucket
2021/12/01 23:16:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 23:16:38 Deleting source bucket docs now
2021/12/01 23:16:38 Starting bucket ops to source bucket
2021/12/01 23:17:15 Waiting for dst bucket item count to get to: 32 curr count: 25
2021/12/01 23:17:20 Waiting for dst bucket item count to get to: 32 curr count: 30
2021/12/01 23:17:20 src & dst bucket item count matched up. src bucket count: 32 dst bucket count: 32
2021/12/01 23:17:20 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 23:17:20 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 23:17:25 Waiting for app: TestBinaryDoc to get un-deployed
2021/12/01 23:17:31 Waiting for app: TestBinaryDoc to get un-deployed
2021/12/01 23:17:36 Waiting for app: TestBinaryDoc to get un-deployed
2021/12/01 23:17:41 Waiting for app: TestBinaryDoc to get un-deployed
2021/12/01 23:17:41 App: TestBinaryDoc got un-deployed
2021/12/01 23:17:41 No eventing-con process running
2021/12/01 23:17:41 Function: TestBinaryDoc delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestBinaryDoc deleting in the background"
}
2021/12/01 23:17:42 Context: Bucket flush, response code: 200 dump: 
2021/12/01 23:17:52 Waiting for dst bucket item count to get to: 0 curr count: 32
2021/12/01 23:17:57 Waiting for dst bucket item count to get to: 0 curr count: 18
2021/12/01 23:17:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 23:17:59 Context: Bucket flush, response code: 200 dump: 
2021/12/01 23:18:09 Waiting for dst bucket item count to get to: 0 curr count: 32
2021/12/01 23:18:14 Waiting for dst bucket item count to get to: 0 curr count: 16
2021/12/01 23:18:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestBinaryDoc (102.44s)
=== RUN   TestConstantBindings
2021/12/01 23:18:14 Deploying app: TestConstantBindings. Item count src bucket: 0 dst bucket: 0
2021/12/01 23:18:15 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestConstantBindings' in metakv",
  "warnings": null
 }
}
2021/12/01 23:18:15 Starting bucket ops to source bucket
2021/12/01 23:18:21 Waiting for dst bucket item count to get to: 6000 curr count: 0
2021/12/01 23:18:26 Waiting for dst bucket item count to get to: 6000 curr count: 0
2021/12/01 23:18:31 Waiting for dst bucket item count to get to: 6000 curr count: 0
2021/12/01 23:18:36 Waiting for dst bucket item count to get to: 6000 curr count: 2498
2021/12/01 23:18:41 Waiting for dst bucket item count to get to: 6000 curr count: 2498
2021/12/01 23:18:41 src & dst bucket item count matched up. src bucket count: 6000 dst bucket count: 6000
2021/12/01 23:18:41 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 23:18:41 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 23:18:46 Waiting for app: TestConstantBindings to get un-deployed
2021/12/01 23:18:51 Waiting for app: TestConstantBindings to get un-deployed
2021/12/01 23:18:56 Waiting for app: TestConstantBindings to get un-deployed
2021/12/01 23:19:01 Waiting for app: TestConstantBindings to get un-deployed
2021/12/01 23:19:01 App: TestConstantBindings got un-deployed
2021/12/01 23:19:01 No eventing-con process running
2021/12/01 23:19:01 Function: TestConstantBindings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestConstantBindings deleting in the background"
}
2021/12/01 23:19:03 Context: Bucket flush, response code: 200 dump: 
2021/12/01 23:19:13 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/01 23:19:18 Waiting for dst bucket item count to get to: 0 curr count: 500
2021/12/01 23:19:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 23:19:20 Context: Bucket flush, response code: 200 dump: 
2021/12/01 23:19:30 Waiting for dst bucket item count to get to: 0 curr count: 3458
2021/12/01 23:19:35 Waiting for dst bucket item count to get to: 0 curr count: 3458
2021/12/01 23:19:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestConstantBindings (80.63s)
=== RUN   TestCodeInsight
2021/12/01 23:19:40 Waiting for app: TestCodeInsight to get un-deployed
2021/12/01 23:19:40 App: TestCodeInsight got un-deployed
2021/12/01 23:19:40 No eventing-con process running
2021/12/01 23:19:40 Function: TestCodeInsight delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestCodeInsight not found"
 }
}
2021/12/01 23:19:42 Context: Bucket flush, response code: 200 dump: 
2021/12/01 23:19:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 23:19:49 Context: Bucket flush, response code: 200 dump: 
2021/12/01 23:19:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 23:19:54 Starting bucket ops to source bucket
2021/12/01 23:19:54 Deploying app: TestCodeInsight. Item count src bucket: 0 dst bucket: 0
2021/12/01 23:19:56 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCodeInsight' in metakv",
  "warnings": null
 }
}
2021/12/01 23:20:01 Waiting for app: TestCodeInsight to get deployed
2021/12/01 23:20:06 Waiting for app: TestCodeInsight to get deployed
2021/12/01 23:20:11 Waiting for app: TestCodeInsight to get deployed
2021/12/01 23:20:16 Waiting for app: TestCodeInsight to get deployed
2021/12/01 23:20:16 App: TestCodeInsight got deployed
2021/12/01 23:20:21 Waiting for dst bucket item count to get to: 1000 curr count: 597
2021/12/01 23:20:21 src & dst bucket item count matched up. src bucket count: 1000 dst bucket count: 1000
2021/12/01 23:20:21 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 23:20:26 Waiting for app: TestCodeInsight to get un-deployed
2021/12/01 23:20:31 Waiting for app: TestCodeInsight to get un-deployed
2021/12/01 23:20:36 Waiting for app: TestCodeInsight to get un-deployed
2021/12/01 23:20:41 Waiting for app: TestCodeInsight to get un-deployed
2021/12/01 23:20:41 App: TestCodeInsight got un-deployed
2021/12/01 23:20:41 No eventing-con process running
2021/12/01 23:20:41 Function: TestCodeInsight delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCodeInsight deleting in the background"
}
2021/12/01 23:20:43 Context: Bucket flush, response code: 200 dump: 
2021/12/01 23:20:53 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/01 23:20:58 Waiting for dst bucket item count to get to: 0 curr count: 500
2021/12/01 23:20:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 23:21:00 Context: Bucket flush, response code: 200 dump: 
2021/12/01 23:21:10 Waiting for dst bucket item count to get to: 0 curr count: 500
2021/12/01 23:21:15 Waiting for dst bucket item count to get to: 0 curr count: 500
2021/12/01 23:21:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCodeInsight (99.69s)
=== RUN   TestParserTransform
--- PASS: TestParserTransform (0.04s)
=== RUN   TestParserDetect
--- PASS: TestParserDetect (0.00s)
PASS
ok  	github.com/couchbase/eventing/tests/functional_tests	13008.967s

Functional tests took 3 hours, 36 minutes and 55 seconds.

N1ql tests

2021/12/01 23:21:23 Started n1ql
Post "http://127.0.0.1:9000/pools/default": read tcp 127.0.0.1:60146->127.0.0.1:9000: read: connection reset by peer
Quota setup Post "http://127.0.0.1:9000/pools/default": read tcp 127.0.0.1:60146->127.0.0.1:9000: read: connection reset by peer
2021/12/01 23:21:33 Adding node: https://127.0.0.1:19001 with role: kv,index,n1ql to the cluster
2021/12/01 23:21:33 Error in adding nodes : , response: ["Prepare join failed. Node is already part of cluster."]
2021/12/01 23:21:33 Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/01 23:21:34 Removing node(s): [] from the cluster
2021/12/01 23:21:39 Rebalance progress: 100
=== RUN   TestRecursiveMutationN1QL
2021/12/01 23:21:44 Waiting for app: TestRecursiveMutationN1QL to get un-deployed
2021/12/01 23:21:44 App: TestRecursiveMutationN1QL got un-deployed
2021/12/01 23:21:44 No eventing-con process running
2021/12/01 23:21:44 Function: TestRecursiveMutationN1QL delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestRecursiveMutationN1QL not found"
 }
}
2021/12/01 23:21:46 Context: Bucket flush, response code: 200 dump: 
2021/12/01 23:21:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 23:21:53 Context: Bucket flush, response code: 200 dump: 
2021/12/01 23:21:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 23:21:58 Deploying app: TestRecursiveMutationN1QL. Item count src bucket: 0 dst bucket: 0
2021/12/01 23:21:58 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_INTER_BUCKET_RECURSION",
 "code": 51,
 "description": "Inter bucket recursion error, deployment of current handler will cause inter bucket recursion",
 "attributes": null,
 "runtime_info": {
  "code": 51,
  "info": "Function: TestRecursiveMutationN1QL N1QL DML to source keyspace default:default"
 }
}
--- PASS: TestRecursiveMutationN1QL (18.64s)
=== RUN   TestFlexReset1
2021/12/01 23:22:03 Waiting for app: TestFlexReset1 to get un-deployed
2021/12/01 23:22:03 App: TestFlexReset1 got un-deployed
2021/12/01 23:22:03 No eventing-con process running
2021/12/01 23:22:03 Function: TestFlexReset1 delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestFlexReset1 not found"
 }
}
2021/12/01 23:22:04 Context: Bucket flush, response code: 200 dump: 
2021/12/01 23:22:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 23:22:11 Context: Bucket flush, response code: 200 dump: 
2021/12/01 23:22:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 23:22:16 Starting bucket ops to source bucket
2021/12/01 23:22:17 Deploying app: TestFlexReset1. Item count src bucket: 0 dst bucket: 0
2021/12/01 23:22:18 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestFlexReset1' in metakv",
  "warnings": null
 }
}
2021/12/01 23:22:23 Waiting for app: TestFlexReset1 to get deployed
2021/12/01 23:22:28 Waiting for app: TestFlexReset1 to get deployed
2021/12/01 23:22:33 Waiting for app: TestFlexReset1 to get deployed
2021/12/01 23:22:38 Waiting for app: TestFlexReset1 to get deployed
2021/12/01 23:22:38 App: TestFlexReset1 got deployed
2021/12/01 23:22:43 Waiting for dst bucket item count to get to: 1000 curr count: 11
2021/12/01 23:22:43 src & dst bucket item count matched up. src bucket count: 1000 dst bucket count: 1000
2021/12/01 23:22:43 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 23:22:43 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 23:22:48 Waiting for app: TestFlexReset1 to get un-deployed
2021/12/01 23:22:53 Waiting for app: TestFlexReset1 to get un-deployed
2021/12/01 23:22:58 Waiting for app: TestFlexReset1 to get un-deployed
2021/12/01 23:23:03 Waiting for app: TestFlexReset1 to get un-deployed
2021/12/01 23:23:03 App: TestFlexReset1 got un-deployed
2021/12/01 23:23:03 No eventing-con process running
2021/12/01 23:23:03 Function: TestFlexReset1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestFlexReset1 deleting in the background"
}
2021/12/01 23:23:05 Context: Bucket flush, response code: 200 dump: 
2021/12/01 23:23:15 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/01 23:23:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 23:23:17 Context: Bucket flush, response code: 200 dump: 
2021/12/01 23:23:27 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/01 23:23:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestFlexReset1 (89.49s)
=== RUN   TestFlexReset2
2021/12/01 23:23:32 Waiting for app: TestFlexReset2 to get un-deployed
2021/12/01 23:23:32 App: TestFlexReset2 got un-deployed
2021/12/01 23:23:32 No eventing-con process running
2021/12/01 23:23:32 Function: TestFlexReset2 delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestFlexReset2 not found"
 }
}
2021/12/01 23:23:34 Context: Bucket flush, response code: 200 dump: 
2021/12/01 23:23:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 23:23:41 Context: Bucket flush, response code: 200 dump: 
2021/12/01 23:23:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 23:23:46 Starting bucket ops to source bucket
2021/12/01 23:23:46 Deploying app: TestFlexReset2. Item count src bucket: 0 dst bucket: 0
2021/12/01 23:23:48 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestFlexReset2' in metakv",
  "warnings": null
 }
}
2021/12/01 23:23:53 Waiting for app: TestFlexReset2 to get deployed
2021/12/01 23:23:58 Waiting for app: TestFlexReset2 to get deployed
2021/12/01 23:24:03 Waiting for app: TestFlexReset2 to get deployed
2021/12/01 23:24:08 Waiting for app: TestFlexReset2 to get deployed
2021/12/01 23:24:08 App: TestFlexReset2 got deployed
2021/12/01 23:24:13 Waiting for dst bucket item count to get to: 1000 curr count: 0
2021/12/01 23:24:13 src & dst bucket item count matched up. src bucket count: 1000 dst bucket count: 1000
2021/12/01 23:24:13 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 23:24:13 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 23:24:18 Waiting for app: TestFlexReset2 to get un-deployed
2021/12/01 23:24:23 Waiting for app: TestFlexReset2 to get un-deployed
2021/12/01 23:24:28 Waiting for app: TestFlexReset2 to get un-deployed
2021/12/01 23:24:33 Waiting for app: TestFlexReset2 to get un-deployed
2021/12/01 23:24:33 App: TestFlexReset2 got un-deployed
2021/12/01 23:24:33 No eventing-con process running
2021/12/01 23:24:33 Function: TestFlexReset2 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestFlexReset2 deleting in the background"
}
2021/12/01 23:24:35 Context: Bucket flush, response code: 200 dump: 
2021/12/01 23:24:45 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/01 23:24:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 23:24:46 Context: Bucket flush, response code: 200 dump: 
2021/12/01 23:24:56 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/01 23:24:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestFlexReset2 (89.39s)
=== RUN   TestN1QLLabelledBreak
2021/12/01 23:25:01 Waiting for app: TestN1QLLabelledBreak to get un-deployed
2021/12/01 23:25:01 App: TestN1QLLabelledBreak got un-deployed
2021/12/01 23:25:01 No eventing-con process running
2021/12/01 23:25:01 Function: TestN1QLLabelledBreak delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLLabelledBreak not found"
 }
}
2021/12/01 23:25:03 Context: Bucket flush, response code: 200 dump: 
2021/12/01 23:25:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 23:25:10 Context: Bucket flush, response code: 200 dump: 
2021/12/01 23:25:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 23:25:15 Deploying app: TestN1QLLabelledBreak. Item count src bucket: 0 dst bucket: 0
2021/12/01 23:25:16 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLLabelledBreak' in metakv",
  "warnings": null
 }
}
2021/12/01 23:25:21 Waiting for app: TestN1QLLabelledBreak to get deployed
2021/12/01 23:25:26 Waiting for app: TestN1QLLabelledBreak to get deployed
2021/12/01 23:25:31 Waiting for app: TestN1QLLabelledBreak to get deployed
2021/12/01 23:25:36 Waiting for app: TestN1QLLabelledBreak to get deployed
2021/12/01 23:25:36 App: TestN1QLLabelledBreak got deployed
2021/12/01 23:25:36 Starting bucket ops to source bucket
2021/12/01 23:25:46 Waiting for dst bucket item count to get to: 10000 curr count: 0
2021/12/01 23:25:51 Waiting for dst bucket item count to get to: 10000 curr count: 243
2021/12/01 23:25:56 Waiting for dst bucket item count to get to: 10000 curr count: 243
2021/12/01 23:26:01 Waiting for dst bucket item count to get to: 10000 curr count: 3548
2021/12/01 23:26:06 Waiting for dst bucket item count to get to: 10000 curr count: 3548
2021/12/01 23:26:11 Waiting for dst bucket item count to get to: 10000 curr count: 7458
2021/12/01 23:26:16 Waiting for dst bucket item count to get to: 10000 curr count: 7458
2021/12/01 23:26:16 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/01 23:26:16 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 23:26:16 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 23:26:21 Waiting for app: TestN1QLLabelledBreak to get un-deployed
2021/12/01 23:26:26 Waiting for app: TestN1QLLabelledBreak to get un-deployed
2021/12/01 23:26:31 Waiting for app: TestN1QLLabelledBreak to get un-deployed
2021/12/01 23:26:36 Waiting for app: TestN1QLLabelledBreak to get un-deployed
2021/12/01 23:26:36 App: TestN1QLLabelledBreak got un-deployed
2021/12/01 23:26:36 No eventing-con process running
2021/12/01 23:26:36 Function: TestN1QLLabelledBreak delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLLabelledBreak deleting in the background"
}
2021/12/01 23:26:38 Context: Bucket flush, response code: 200 dump: 
2021/12/01 23:26:48 Waiting for dst bucket item count to get to: 0 curr count: 3284
2021/12/01 23:26:53 Waiting for dst bucket item count to get to: 0 curr count: 784
2021/12/01 23:26:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 23:26:55 Context: Bucket flush, response code: 200 dump: 
2021/12/01 23:27:05 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/01 23:27:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLLabelledBreak (128.53s)
=== RUN   TestN1QLUnlabelledBreak
2021/12/01 23:27:10 Waiting for app: TestN1QLUnlabelledBreak to get un-deployed
2021/12/01 23:27:10 App: TestN1QLUnlabelledBreak got un-deployed
2021/12/01 23:27:10 No eventing-con process running
2021/12/01 23:27:10 Function: TestN1QLUnlabelledBreak delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLUnlabelledBreak not found"
 }
}
2021/12/01 23:27:12 Context: Bucket flush, response code: 200 dump: 
2021/12/01 23:27:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 23:27:19 Context: Bucket flush, response code: 200 dump: 
2021/12/01 23:27:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 23:27:24 Deploying app: TestN1QLUnlabelledBreak. Item count src bucket: 0 dst bucket: 0
2021/12/01 23:27:25 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLUnlabelledBreak' in metakv",
  "warnings": null
 }
}
2021/12/01 23:27:30 Waiting for app: TestN1QLUnlabelledBreak to get deployed
2021/12/01 23:27:35 Waiting for app: TestN1QLUnlabelledBreak to get deployed
2021/12/01 23:27:40 Waiting for app: TestN1QLUnlabelledBreak to get deployed
2021/12/01 23:27:45 Waiting for app: TestN1QLUnlabelledBreak to get deployed
2021/12/01 23:27:45 App: TestN1QLUnlabelledBreak got deployed
2021/12/01 23:27:45 Starting bucket ops to source bucket
2021/12/01 23:27:55 Waiting for dst bucket item count to get to: 10000 curr count: 214
2021/12/01 23:28:00 Waiting for dst bucket item count to get to: 10000 curr count: 667
2021/12/01 23:28:05 Waiting for dst bucket item count to get to: 10000 curr count: 3018
2021/12/01 23:28:10 Waiting for dst bucket item count to get to: 10000 curr count: 3018
2021/12/01 23:28:15 Waiting for dst bucket item count to get to: 10000 curr count: 5752
2021/12/01 23:28:15 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/01 23:28:15 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 23:28:15 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 23:28:20 Waiting for app: TestN1QLUnlabelledBreak to get un-deployed
2021/12/01 23:28:25 Waiting for app: TestN1QLUnlabelledBreak to get un-deployed
2021/12/01 23:28:30 Waiting for app: TestN1QLUnlabelledBreak to get un-deployed
2021/12/01 23:28:35 Waiting for app: TestN1QLUnlabelledBreak to get un-deployed
2021/12/01 23:28:35 App: TestN1QLUnlabelledBreak got un-deployed
2021/12/01 23:28:35 No eventing-con process running
2021/12/01 23:28:35 Function: TestN1QLUnlabelledBreak delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLUnlabelledBreak deleting in the background"
}
2021/12/01 23:28:37 Context: Bucket flush, response code: 200 dump: 
2021/12/01 23:28:47 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/01 23:28:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 23:28:49 Context: Bucket flush, response code: 200 dump: 
2021/12/01 23:28:59 Waiting for dst bucket item count to get to: 0 curr count: 8775
2021/12/01 23:29:04 Waiting for dst bucket item count to get to: 0 curr count: 3775
2021/12/01 23:29:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLUnlabelledBreak (118.68s)
=== RUN   TestN1QLThrowStatement
2021/12/01 23:29:09 Waiting for app: TestN1QLThrowStatement to get un-deployed
2021/12/01 23:29:09 App: TestN1QLThrowStatement got un-deployed
2021/12/01 23:29:09 No eventing-con process running
2021/12/01 23:29:09 Function: TestN1QLThrowStatement delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLThrowStatement not found"
 }
}
2021/12/01 23:29:10 Context: Bucket flush, response code: 200 dump: 
2021/12/01 23:29:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 23:29:17 Context: Bucket flush, response code: 200 dump: 
2021/12/01 23:29:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 23:29:22 Deploying app: TestN1QLThrowStatement. Item count src bucket: 0 dst bucket: 0
2021/12/01 23:29:23 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLThrowStatement' in metakv",
  "warnings": null
 }
}
2021/12/01 23:29:28 Waiting for app: TestN1QLThrowStatement to get deployed
2021/12/01 23:29:33 Waiting for app: TestN1QLThrowStatement to get deployed
2021/12/01 23:29:38 Waiting for app: TestN1QLThrowStatement to get deployed
2021/12/01 23:29:43 Waiting for app: TestN1QLThrowStatement to get deployed
2021/12/01 23:29:43 App: TestN1QLThrowStatement got deployed
2021/12/01 23:29:43 Starting bucket ops to source bucket
2021/12/01 23:29:53 Waiting for dst bucket item count to get to: 10000 curr count: 0
2021/12/01 23:29:58 Waiting for dst bucket item count to get to: 10000 curr count: 241
2021/12/01 23:30:03 Waiting for dst bucket item count to get to: 10000 curr count: 743
2021/12/01 23:30:08 Waiting for dst bucket item count to get to: 10000 curr count: 2582
2021/12/01 23:30:13 Waiting for dst bucket item count to get to: 10000 curr count: 4592
2021/12/01 23:30:18 Waiting for dst bucket item count to get to: 10000 curr count: 6600
2021/12/01 23:30:23 Waiting for dst bucket item count to get to: 10000 curr count: 8580
2021/12/01 23:30:28 Waiting for dst bucket item count to get to: 10000 curr count: 9492
2021/12/01 23:30:28 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/01 23:30:28 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 23:30:28 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 23:30:33 Waiting for app: TestN1QLThrowStatement to get un-deployed
2021/12/01 23:30:38 Waiting for app: TestN1QLThrowStatement to get un-deployed
2021/12/01 23:30:43 Waiting for app: TestN1QLThrowStatement to get un-deployed
2021/12/01 23:30:48 Waiting for app: TestN1QLThrowStatement to get un-deployed
2021/12/01 23:30:48 App: TestN1QLThrowStatement got un-deployed
2021/12/01 23:30:48 No eventing-con process running
2021/12/01 23:30:48 Function: TestN1QLThrowStatement delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLThrowStatement deleting in the background"
}
2021/12/01 23:30:50 Context: Bucket flush, response code: 200 dump: 
2021/12/01 23:31:00 Waiting for dst bucket item count to get to: 0 curr count: 3359
2021/12/01 23:31:05 Waiting for dst bucket item count to get to: 0 curr count: 3359
2021/12/01 23:31:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 23:31:07 Context: Bucket flush, response code: 200 dump: 
2021/12/01 23:31:17 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/01 23:31:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLThrowStatement (133.27s)
=== RUN   TestN1QLNestedForLoop
2021/12/01 23:31:22 Waiting for app: TestN1QLNestedForLoop to get un-deployed
2021/12/01 23:31:22 App: TestN1QLNestedForLoop got un-deployed
2021/12/01 23:31:22 No eventing-con process running
2021/12/01 23:31:22 Function: TestN1QLNestedForLoop delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLNestedForLoop not found"
 }
}
2021/12/01 23:31:24 Context: Bucket flush, response code: 200 dump: 
2021/12/01 23:31:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 23:31:30 Context: Bucket flush, response code: 200 dump: 
2021/12/01 23:31:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 23:31:36 Deploying app: TestN1QLNestedForLoop. Item count src bucket: 0 dst bucket: 0
2021/12/01 23:31:37 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLNestedForLoop' in metakv",
  "warnings": null
 }
}
2021/12/01 23:31:42 Waiting for app: TestN1QLNestedForLoop to get deployed
2021/12/01 23:31:47 Waiting for app: TestN1QLNestedForLoop to get deployed
2021/12/01 23:31:52 Waiting for app: TestN1QLNestedForLoop to get deployed
2021/12/01 23:31:57 Waiting for app: TestN1QLNestedForLoop to get deployed
2021/12/01 23:31:57 App: TestN1QLNestedForLoop got deployed
2021/12/01 23:31:57 Starting bucket ops to source bucket
2021/12/01 23:32:06 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 23:32:11 Waiting for dst bucket item count to get to: 5000 curr count: 90
2021/12/01 23:32:16 Waiting for dst bucket item count to get to: 5000 curr count: 90
2021/12/01 23:32:21 Waiting for dst bucket item count to get to: 5000 curr count: 1991
2021/12/01 23:32:26 Waiting for dst bucket item count to get to: 5000 curr count: 1991
2021/12/01 23:32:31 Waiting for dst bucket item count to get to: 5000 curr count: 4393
2021/12/01 23:32:36 Waiting for dst bucket item count to get to: 5000 curr count: 4393
2021/12/01 23:32:36 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/01 23:32:36 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 23:32:36 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 23:32:41 Waiting for app: TestN1QLNestedForLoop to get un-deployed
2021/12/01 23:32:46 Waiting for app: TestN1QLNestedForLoop to get un-deployed
2021/12/01 23:32:51 Waiting for app: TestN1QLNestedForLoop to get un-deployed
2021/12/01 23:32:56 Waiting for app: TestN1QLNestedForLoop to get un-deployed
2021/12/01 23:32:56 App: TestN1QLNestedForLoop got un-deployed
2021/12/01 23:32:56 No eventing-con process running
2021/12/01 23:32:56 Function: TestN1QLNestedForLoop delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLNestedForLoop deleting in the background"
}
2021/12/01 23:32:58 Context: Bucket flush, response code: 200 dump: 
2021/12/01 23:33:08 Waiting for dst bucket item count to get to: 0 curr count: 3548
2021/12/01 23:33:13 Waiting for dst bucket item count to get to: 0 curr count: 1048
2021/12/01 23:33:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 23:33:15 Context: Bucket flush, response code: 200 dump: 
2021/12/01 23:33:25 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/01 23:33:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLNestedForLoop (128.16s)
=== RUN   TestN1QLAutoClose
2021/12/01 23:33:30 Waiting for app: TestN1QLAutoClose to get un-deployed
2021/12/01 23:33:30 App: TestN1QLAutoClose got un-deployed
2021/12/01 23:33:30 No eventing-con process running
2021/12/01 23:33:30 Function: TestN1QLAutoClose delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLAutoClose not found"
 }
}
2021/12/01 23:33:32 Context: Bucket flush, response code: 200 dump: 
2021/12/01 23:33:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 23:33:39 Context: Bucket flush, response code: 200 dump: 
2021/12/01 23:33:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 23:33:44 Deploying app: TestN1QLAutoClose. Item count src bucket: 0 dst bucket: 0
2021/12/01 23:33:45 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAutoClose' in metakv",
  "warnings": null
 }
}
2021/12/01 23:33:50 Waiting for app: TestN1QLAutoClose to get deployed
2021/12/01 23:33:55 Waiting for app: TestN1QLAutoClose to get deployed
2021/12/01 23:34:00 Waiting for app: TestN1QLAutoClose to get deployed
2021/12/01 23:34:05 Waiting for app: TestN1QLAutoClose to get deployed
2021/12/01 23:34:05 App: TestN1QLAutoClose got deployed
2021/12/01 23:34:05 Starting bucket ops to source bucket
2021/12/01 23:34:11 Waiting for dst bucket item count to get to: 500 curr count: 0
2021/12/01 23:34:16 Waiting for dst bucket item count to get to: 500 curr count: 2
2021/12/01 23:34:21 Waiting for dst bucket item count to get to: 500 curr count: 5
2021/12/01 23:34:26 Waiting for dst bucket item count to get to: 500 curr count: 28
2021/12/01 23:34:31 Waiting for dst bucket item count to get to: 500 curr count: 28
2021/12/01 23:34:36 Waiting for dst bucket item count to get to: 500 curr count: 49
2021/12/01 23:34:41 Waiting for dst bucket item count to get to: 500 curr count: 49
2021/12/01 23:34:46 Waiting for dst bucket item count to get to: 500 curr count: 72
2021/12/01 23:34:51 Waiting for dst bucket item count to get to: 500 curr count: 72
2021/12/01 23:34:56 Waiting for dst bucket item count to get to: 500 curr count: 92
2021/12/01 23:35:01 Waiting for dst bucket item count to get to: 500 curr count: 92
2021/12/01 23:35:06 Waiting for dst bucket item count to get to: 500 curr count: 92
2021/12/01 23:35:11 Waiting for dst bucket item count to get to: 500 curr count: 113
2021/12/01 23:35:16 Waiting for dst bucket item count to get to: 500 curr count: 113
2021/12/01 23:35:21 Waiting for dst bucket item count to get to: 500 curr count: 134
2021/12/01 23:35:26 Waiting for dst bucket item count to get to: 500 curr count: 134
2021/12/01 23:35:31 Waiting for dst bucket item count to get to: 500 curr count: 156
2021/12/01 23:35:36 Waiting for dst bucket item count to get to: 500 curr count: 156
2021/12/01 23:35:41 Waiting for dst bucket item count to get to: 500 curr count: 179
2021/12/01 23:35:46 Waiting for dst bucket item count to get to: 500 curr count: 179
2021/12/01 23:35:51 Waiting for dst bucket item count to get to: 500 curr count: 202
2021/12/01 23:35:56 Waiting for dst bucket item count to get to: 500 curr count: 202
2021/12/01 23:36:01 Waiting for dst bucket item count to get to: 500 curr count: 223
2021/12/01 23:36:06 Waiting for dst bucket item count to get to: 500 curr count: 223
2021/12/01 23:36:11 Waiting for dst bucket item count to get to: 500 curr count: 245
2021/12/01 23:36:16 Waiting for dst bucket item count to get to: 500 curr count: 245
2021/12/01 23:36:21 Waiting for dst bucket item count to get to: 500 curr count: 263
2021/12/01 23:36:26 Waiting for dst bucket item count to get to: 500 curr count: 263
2021/12/01 23:36:31 Waiting for dst bucket item count to get to: 500 curr count: 285
2021/12/01 23:36:36 Waiting for dst bucket item count to get to: 500 curr count: 297
2021/12/01 23:36:41 Waiting for dst bucket item count to get to: 500 curr count: 306
2021/12/01 23:36:46 Waiting for dst bucket item count to get to: 500 curr count: 319
2021/12/01 23:36:51 Waiting for dst bucket item count to get to: 500 curr count: 329
2021/12/01 23:36:56 Waiting for dst bucket item count to get to: 500 curr count: 341
2021/12/01 23:37:01 Waiting for dst bucket item count to get to: 500 curr count: 352
2021/12/01 23:37:06 Waiting for dst bucket item count to get to: 500 curr count: 362
2021/12/01 23:37:11 Waiting for dst bucket item count to get to: 500 curr count: 371
2021/12/01 23:37:16 Waiting for dst bucket item count to get to: 500 curr count: 382
2021/12/01 23:37:21 Waiting for dst bucket item count to get to: 500 curr count: 392
2021/12/01 23:37:26 Waiting for dst bucket item count to get to: 500 curr count: 403
2021/12/01 23:37:31 Waiting for dst bucket item count to get to: 500 curr count: 413
2021/12/01 23:37:36 Waiting for dst bucket item count to get to: 500 curr count: 427
2021/12/01 23:37:41 Waiting for dst bucket item count to get to: 500 curr count: 437
2021/12/01 23:37:46 Waiting for dst bucket item count to get to: 500 curr count: 449
2021/12/01 23:37:51 Waiting for dst bucket item count to get to: 500 curr count: 458
2021/12/01 23:37:56 Waiting for dst bucket item count to get to: 500 curr count: 468
2021/12/01 23:38:01 Waiting for dst bucket item count to get to: 500 curr count: 480
2021/12/01 23:38:06 Waiting for dst bucket item count to get to: 500 curr count: 490
2021/12/01 23:38:11 Waiting for dst bucket item count to get to: 500 curr count: 499
2021/12/01 23:38:11 src & dst bucket item count matched up. src bucket count: 500 dst bucket count: 500
2021/12/01 23:38:11 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 23:38:11 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 23:38:16 Waiting for app: TestN1QLAutoClose to get un-deployed
2021/12/01 23:38:21 Waiting for app: TestN1QLAutoClose to get un-deployed
2021/12/01 23:38:26 Waiting for app: TestN1QLAutoClose to get un-deployed
2021/12/01 23:38:31 Waiting for app: TestN1QLAutoClose to get un-deployed
2021/12/01 23:38:31 App: TestN1QLAutoClose got un-deployed
2021/12/01 23:38:31 No eventing-con process running
2021/12/01 23:38:31 Function: TestN1QLAutoClose delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLAutoClose deleting in the background"
}
2021/12/01 23:38:33 Context: Bucket flush, response code: 200 dump: 
2021/12/01 23:38:43 Waiting for dst bucket item count to get to: 0 curr count: 500
2021/12/01 23:38:48 Waiting for dst bucket item count to get to: 0 curr count: 250
2021/12/01 23:38:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 23:38:50 Context: Bucket flush, response code: 200 dump: 
2021/12/01 23:39:00 Waiting for dst bucket item count to get to: 0 curr count: 250
2021/12/01 23:39:05 Waiting for dst bucket item count to get to: 0 curr count: 250
2021/12/01 23:39:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLAutoClose (339.65s)
=== RUN   TestN1QLPosParams
2021/12/01 23:39:10 Waiting for app: TestN1QLPosParams to get un-deployed
2021/12/01 23:39:10 App: TestN1QLPosParams got un-deployed
2021/12/01 23:39:10 No eventing-con process running
2021/12/01 23:39:10 Function: TestN1QLPosParams delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLPosParams not found"
 }
}
2021/12/01 23:39:12 Context: Bucket flush, response code: 200 dump: 
2021/12/01 23:39:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 23:39:18 Context: Bucket flush, response code: 200 dump: 
2021/12/01 23:39:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 23:39:23 Deploying app: TestN1QLPosParams. Item count src bucket: 0 dst bucket: 0
2021/12/01 23:39:24 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLPosParams' in metakv",
  "warnings": null
 }
}
2021/12/01 23:39:29 Waiting for app: TestN1QLPosParams to get deployed
2021/12/01 23:39:35 Waiting for app: TestN1QLPosParams to get deployed
2021/12/01 23:39:40 Waiting for app: TestN1QLPosParams to get deployed
2021/12/01 23:39:45 Waiting for app: TestN1QLPosParams to get deployed
2021/12/01 23:39:45 App: TestN1QLPosParams got deployed
2021/12/01 23:39:45 Starting bucket ops to source bucket
2021/12/01 23:39:54 Waiting for dst bucket item count to get to: 5000 curr count: 416
2021/12/01 23:39:59 Waiting for dst bucket item count to get to: 5000 curr count: 1219
2021/12/01 23:40:04 Waiting for dst bucket item count to get to: 5000 curr count: 1219
2021/12/01 23:40:09 Waiting for dst bucket item count to get to: 5000 curr count: 3303
2021/12/01 23:40:09 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/01 23:40:09 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 23:40:09 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 23:40:14 Waiting for app: TestN1QLPosParams to get un-deployed
2021/12/01 23:40:19 Waiting for app: TestN1QLPosParams to get un-deployed
2021/12/01 23:40:24 Waiting for app: TestN1QLPosParams to get un-deployed
2021/12/01 23:40:29 Waiting for app: TestN1QLPosParams to get un-deployed
2021/12/01 23:40:29 App: TestN1QLPosParams got un-deployed
2021/12/01 23:40:29 No eventing-con process running
2021/12/01 23:40:29 Function: TestN1QLPosParams delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLPosParams deleting in the background"
}
2021/12/01 23:40:31 Context: Bucket flush, response code: 200 dump: 
2021/12/01 23:40:41 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/01 23:40:46 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/01 23:40:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 23:40:48 Context: Bucket flush, response code: 200 dump: 
2021/12/01 23:40:58 Waiting for dst bucket item count to get to: 0 curr count: 3513
2021/12/01 23:41:03 Waiting for dst bucket item count to get to: 0 curr count: 1013
2021/12/01 23:41:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLPosParams (118.30s)
=== RUN   TestN1QLExhaustConnPool
2021/12/01 23:41:08 Waiting for app: TestN1QLExhaustConnPool to get un-deployed
2021/12/01 23:41:08 App: TestN1QLExhaustConnPool got un-deployed
2021/12/01 23:41:08 No eventing-con process running
2021/12/01 23:41:08 Function: TestN1QLExhaustConnPool delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLExhaustConnPool not found"
 }
}
2021/12/01 23:41:10 Context: Bucket flush, response code: 200 dump: 
2021/12/01 23:41:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 23:41:17 Context: Bucket flush, response code: 200 dump: 
2021/12/01 23:41:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 23:41:22 Deploying app: TestN1QLExhaustConnPool. Item count src bucket: 0 dst bucket: 0
2021/12/01 23:41:23 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLExhaustConnPool' in metakv",
  "warnings": null
 }
}
2021/12/01 23:41:28 Waiting for app: TestN1QLExhaustConnPool to get deployed
2021/12/01 23:41:33 Waiting for app: TestN1QLExhaustConnPool to get deployed
2021/12/01 23:41:38 Waiting for app: TestN1QLExhaustConnPool to get deployed
2021/12/01 23:41:43 Waiting for app: TestN1QLExhaustConnPool to get deployed
2021/12/01 23:41:43 App: TestN1QLExhaustConnPool got deployed
2021/12/01 23:41:43 Starting bucket ops to source bucket
2021/12/01 23:41:48 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/01 23:41:53 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/01 23:41:53 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/01 23:41:53 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 23:41:53 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 23:41:58 Waiting for app: TestN1QLExhaustConnPool to get un-deployed
2021/12/01 23:42:03 Waiting for app: TestN1QLExhaustConnPool to get un-deployed
2021/12/01 23:42:08 Waiting for app: TestN1QLExhaustConnPool to get un-deployed
2021/12/01 23:42:13 Waiting for app: TestN1QLExhaustConnPool to get un-deployed
2021/12/01 23:42:13 App: TestN1QLExhaustConnPool got un-deployed
2021/12/01 23:42:13 No eventing-con process running
2021/12/01 23:42:13 Function: TestN1QLExhaustConnPool delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLExhaustConnPool deleting in the background"
}
2021/12/01 23:42:15 Context: Bucket flush, response code: 200 dump: 
2021/12/01 23:42:25 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/01 23:42:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 23:42:27 Context: Bucket flush, response code: 200 dump: 
2021/12/01 23:42:37 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/01 23:42:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLExhaustConnPool (93.95s)
=== RUN   TestN1QLGraceSufficientTimeOut
2021/12/01 23:42:42 Waiting for app: TestN1QLGraceSufficientTimeOut to get un-deployed
2021/12/01 23:42:42 App: TestN1QLGraceSufficientTimeOut got un-deployed
2021/12/01 23:42:42 No eventing-con process running
2021/12/01 23:42:42 Function: TestN1QLGraceSufficientTimeOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLGraceSufficientTimeOut not found"
 }
}
2021/12/01 23:42:44 Context: Bucket flush, response code: 200 dump: 
2021/12/01 23:42:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 23:42:51 Context: Bucket flush, response code: 200 dump: 
2021/12/01 23:42:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 23:42:56 Starting bucket ops to source bucket
2021/12/01 23:43:13 Starting bucket ops to source bucket
2021/12/01 23:43:31 Deploying app: TestN1QLGraceSufficientTimeOut. Item count src bucket: 30000 dst bucket: 8463
2021/12/01 23:43:32 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLGraceSufficientTimeOut' in metakv",
  "warnings": null
 }
}
2021/12/01 23:43:37 Waiting for app: TestN1QLGraceSufficientTimeOut to get deployed
2021/12/01 23:43:42 Waiting for app: TestN1QLGraceSufficientTimeOut to get deployed
2021/12/01 23:43:47 Waiting for app: TestN1QLGraceSufficientTimeOut to get deployed
2021/12/01 23:43:52 Waiting for app: TestN1QLGraceSufficientTimeOut to get deployed
2021/12/01 23:43:52 App: TestN1QLGraceSufficientTimeOut got deployed
2021/12/01 23:43:52 Starting bucket ops to source bucket
2021/12/01 23:43:57 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/01 23:44:02 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/01 23:44:07 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/01 23:44:12 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/01 23:44:17 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/01 23:44:22 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/01 23:44:27 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/01 23:44:32 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/01 23:44:37 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/01 23:44:42 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/01 23:44:47 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/01 23:44:47 src & dst bucket item count matched up. src bucket count: 30001 dst bucket count: 30001
2021/12/01 23:44:47 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 23:44:47 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 23:44:52 Waiting for app: TestN1QLGraceSufficientTimeOut to get un-deployed
2021/12/01 23:44:57 Waiting for app: TestN1QLGraceSufficientTimeOut to get un-deployed
2021/12/01 23:45:02 Waiting for app: TestN1QLGraceSufficientTimeOut to get un-deployed
2021/12/01 23:45:07 Waiting for app: TestN1QLGraceSufficientTimeOut to get un-deployed
2021/12/01 23:45:07 App: TestN1QLGraceSufficientTimeOut got un-deployed
2021/12/01 23:45:07 No eventing-con process running
2021/12/01 23:45:07 Function: TestN1QLGraceSufficientTimeOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLGraceSufficientTimeOut deleting in the background"
}
2021/12/01 23:45:09 Context: Bucket flush, response code: 200 dump: 
2021/12/01 23:45:19 Waiting for dst bucket item count to get to: 0 curr count: 30000
2021/12/01 23:45:24 Waiting for dst bucket item count to get to: 0 curr count: 15000
2021/12/01 23:45:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 23:45:26 Context: Bucket flush, response code: 200 dump: 
2021/12/01 23:45:36 Waiting for dst bucket item count to get to: 0 curr count: 30001
2021/12/01 23:45:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLGraceSufficientTimeOut (179.09s)
=== RUN   TestN1QLQueryConsistency
2021/12/01 23:45:41 Waiting for app: n1qlquery_consistency to get un-deployed
2021/12/01 23:45:41 App: n1qlquery_consistency got un-deployed
2021/12/01 23:45:41 No eventing-con process running
2021/12/01 23:45:41 Function: n1qlquery_consistency delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: n1qlquery_consistency not found"
 }
}
2021/12/01 23:45:43 Context: Bucket flush, response code: 200 dump: 
2021/12/01 23:45:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 23:45:50 Context: Bucket flush, response code: 200 dump: 
2021/12/01 23:45:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 23:45:55 Starting bucket ops to source bucket
2021/12/01 23:45:58 Deploying app: n1qlquery_consistency. Item count src bucket: 0 dst bucket: 0
2021/12/01 23:45:59 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'n1qlquery_consistency' in metakv",
  "warnings": null
 }
}
2021/12/01 23:46:04 Waiting for app: n1qlquery_consistency to get deployed
2021/12/01 23:46:09 Waiting for app: n1qlquery_consistency to get deployed
2021/12/01 23:46:14 Waiting for app: n1qlquery_consistency to get deployed
2021/12/01 23:46:19 Waiting for app: n1qlquery_consistency to get deployed
2021/12/01 23:46:19 App: n1qlquery_consistency got deployed
2021/12/01 23:46:24 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 23:46:29 Waiting for dst bucket item count to get to: 5000 curr count: 4306
2021/12/01 23:46:34 Waiting for dst bucket item count to get to: 5000 curr count: 4306
2021/12/01 23:46:34 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/01 23:46:34 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 23:46:34 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 23:46:39 Waiting for app: n1qlquery_consistency to get un-deployed
2021/12/01 23:46:44 Waiting for app: n1qlquery_consistency to get un-deployed
2021/12/01 23:46:49 Waiting for app: n1qlquery_consistency to get un-deployed
2021/12/01 23:46:54 Waiting for app: n1qlquery_consistency to get un-deployed
2021/12/01 23:46:54 App: n1qlquery_consistency got un-deployed
2021/12/01 23:46:54 No eventing-con process running
2021/12/01 23:46:54 Function: n1qlquery_consistency delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: n1qlquery_consistency deleting in the background"
}
2021/12/01 23:46:56 Context: Bucket flush, response code: 200 dump: 
2021/12/01 23:47:06 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/01 23:47:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 23:47:08 Context: Bucket flush, response code: 200 dump: 
2021/12/01 23:47:18 Waiting for dst bucket item count to get to: 0 curr count: 3628
2021/12/01 23:47:23 Waiting for dst bucket item count to get to: 0 curr count: 1128
2021/12/01 23:47:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLQueryConsistency (106.86s)
PASS
ok  	github.com/couchbase/eventing/tests/functional_tests	1555.858s

N1ql tests took 0 hours, 26 minutes and 0 seconds.

Starting server
Starting server: attempt 1

Rebalance tests

2021/12/01 23:48:58 Started rebalance
Post "http://127.0.0.1:9000/pools/default": EOF
Quota setup Post "http://127.0.0.1:9000/pools/default": EOF
=== RUN   TestDuplicateWithEventingRebIn
2021/12/01 23:49:16 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
2021/12/01 23:49:16 App: TestDuplicateWithEventingRebIn got un-deployed
2021/12/01 23:49:17 No eventing-con process running
2021/12/01 23:49:17 Function: TestDuplicateWithEventingRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithEventingRebIn not found"
 }
}
2021/12/01 23:49:18 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 23:49:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 23:49:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 23:49:25 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 23:49:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 23:49:30 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 23:49:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 23:49:35 Deploying app: TestDuplicateWithEventingRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/01 23:49:36 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithEventingRebIn' in metakv",
  "warnings": null
 }
}
2021/12/01 23:49:41 Waiting for app: TestDuplicateWithEventingRebIn to get deployed
2021/12/01 23:49:46 Waiting for app: TestDuplicateWithEventingRebIn to get deployed
2021/12/01 23:49:51 Waiting for app: TestDuplicateWithEventingRebIn to get deployed
2021/12/01 23:49:56 Waiting for app: TestDuplicateWithEventingRebIn to get deployed
2021/12/01 23:49:56 App: TestDuplicateWithEventingRebIn got deployed
2021/12/01 23:49:56 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/01 23:49:56 Starting bucket ops to source bucket
2021/12/01 23:50:11 Removing node(s): [] from the cluster
2021/12/01 23:50:16 Rebalance progress: 25
2021/12/01 23:50:21 Rebalance progress: 25
2021/12/01 23:50:26 Rebalance progress: 25
2021/12/01 23:50:31 Rebalance progress: 25
2021/12/01 23:50:36 Rebalance progress: 75
2021/12/01 23:50:41 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 23:50:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 23:50:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 23:50:41 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9302/api/v1/stats": dial tcp 127.0.0.1:9302: connect: connection refused
2021/12/01 23:50:41 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/01 23:50:41 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/01 23:50:46 Rebalance progress: 25
2021/12/01 23:50:51 Rebalance progress: 25
2021/12/01 23:50:56 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 23:50:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 23:50:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 23:50:56 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9302/api/v1/stats": dial tcp 127.0.0.1:9302: connect: connection refused
2021/12/01 23:50:56 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/01 23:50:56 Waiting for src bucket item count to get to: 200000 curr count: 82033
2021/12/01 23:51:01 Waiting for src bucket item count to get to: 200000 curr count: 82033
2021/12/01 23:51:06 Waiting for src bucket item count to get to: 200000 curr count: 98077
2021/12/01 23:51:11 Waiting for src bucket item count to get to: 200000 curr count: 98077
2021/12/01 23:51:16 Waiting for src bucket item count to get to: 200000 curr count: 114484
2021/12/01 23:51:21 Waiting for src bucket item count to get to: 200000 curr count: 114484
2021/12/01 23:51:26 Waiting for src bucket item count to get to: 200000 curr count: 130834
2021/12/01 23:51:31 Waiting for src bucket item count to get to: 200000 curr count: 130834
2021/12/01 23:51:36 Waiting for src bucket item count to get to: 200000 curr count: 147412
2021/12/01 23:51:41 Waiting for src bucket item count to get to: 200000 curr count: 147412
2021/12/01 23:51:46 Waiting for src bucket item count to get to: 200000 curr count: 163957
2021/12/01 23:51:51 Waiting for src bucket item count to get to: 200000 curr count: 163957
2021/12/01 23:51:56 Waiting for src bucket item count to get to: 200000 curr count: 180285
2021/12/01 23:52:01 Waiting for src bucket item count to get to: 200000 curr count: 180285
2021/12/01 23:52:06 Waiting for src bucket item count to get to: 200000 curr count: 196435
2021/12/01 23:52:11 Waiting for src bucket item count to get to: 200000 curr count: 196435
2021/12/01 23:52:21 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/01 23:52:26 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/01 23:52:31 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/01 23:52:36 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/01 23:52:41 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/01 23:52:46 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/01 23:52:51 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/01 23:52:56 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/01 23:53:01 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/01 23:53:06 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/01 23:53:06 Post SET ops. Expected item count: 200000 got 200001
2021/12/01 23:53:06 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 23:53:11 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
2021/12/01 23:53:16 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
2021/12/01 23:53:21 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
2021/12/01 23:53:26 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
2021/12/01 23:53:26 App: TestDuplicateWithEventingRebIn got un-deployed
2021/12/01 23:53:26 No eventing-con process running
2021/12/01 23:53:26 Function: TestDuplicateWithEventingRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithEventingRebIn deleting in the background"
}
2021/12/01 23:53:28 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 23:53:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 23:53:38 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/01 23:53:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 23:53:40 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 23:53:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 23:53:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithEventingRebIn (278.30s)
=== RUN   TestDuplicateWithEventingRebOut
2021/12/01 23:53:50 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/01 23:53:50 Removing node(s): [] from the cluster
2021/12/01 23:53:55 Rebalance progress: 100
2021/12/01 23:54:00 Waiting for app: TestDuplicateWithEventingRebOut to get un-deployed
2021/12/01 23:54:00 App: TestDuplicateWithEventingRebOut got un-deployed
2021/12/01 23:54:00 No eventing-con process running
2021/12/01 23:54:00 Function: TestDuplicateWithEventingRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithEventingRebOut not found"
 }
}
2021/12/01 23:54:02 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 23:54:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 23:54:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 23:54:09 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 23:54:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 23:54:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 23:54:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 23:54:19 Deploying app: TestDuplicateWithEventingRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/01 23:54:20 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithEventingRebOut' in metakv",
  "warnings": null
 }
}
2021/12/01 23:54:25 Waiting for app: TestDuplicateWithEventingRebOut to get deployed
2021/12/01 23:54:30 Waiting for app: TestDuplicateWithEventingRebOut to get deployed
2021/12/01 23:54:35 Waiting for app: TestDuplicateWithEventingRebOut to get deployed
2021/12/01 23:54:40 Waiting for app: TestDuplicateWithEventingRebOut to get deployed
2021/12/01 23:54:40 App: TestDuplicateWithEventingRebOut got deployed
2021/12/01 23:54:40 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/01 23:54:40 Starting bucket ops to source bucket
2021/12/01 23:54:45 Rebalance progress: 25
2021/12/01 23:54:50 Rebalance progress: 25
2021/12/01 23:54:55 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 23:54:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 23:54:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 23:54:55 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9302/api/v1/stats": dial tcp 127.0.0.1:9302: connect: connection refused
2021/12/01 23:54:55 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/01 23:54:55 Waiting for src bucket item count to get to: 200000 curr count: 14298
2021/12/01 23:55:00 Waiting for src bucket item count to get to: 200000 curr count: 30537
2021/12/01 23:55:05 Waiting for src bucket item count to get to: 200000 curr count: 30537
2021/12/01 23:55:10 Waiting for src bucket item count to get to: 200000 curr count: 46865
2021/12/01 23:55:15 Waiting for src bucket item count to get to: 200000 curr count: 46865
2021/12/01 23:55:20 Waiting for src bucket item count to get to: 200000 curr count: 63085
2021/12/01 23:55:25 Waiting for src bucket item count to get to: 200000 curr count: 63085
2021/12/01 23:55:30 Waiting for src bucket item count to get to: 200000 curr count: 63085
2021/12/01 23:55:35 Waiting for src bucket item count to get to: 200000 curr count: 79101
2021/12/01 23:55:40 Waiting for src bucket item count to get to: 200000 curr count: 79101
2021/12/01 23:55:45 Waiting for src bucket item count to get to: 200000 curr count: 95331
2021/12/01 23:55:50 Waiting for src bucket item count to get to: 200000 curr count: 95331
2021/12/01 23:55:55 Waiting for src bucket item count to get to: 200000 curr count: 111695
2021/12/01 23:56:00 Waiting for src bucket item count to get to: 200000 curr count: 111695
2021/12/01 23:56:05 Waiting for src bucket item count to get to: 200000 curr count: 128041
2021/12/01 23:56:10 Waiting for src bucket item count to get to: 200000 curr count: 128041
2021/12/01 23:56:15 Waiting for src bucket item count to get to: 200000 curr count: 144324
2021/12/01 23:56:20 Waiting for src bucket item count to get to: 200000 curr count: 144324
2021/12/01 23:56:25 Waiting for src bucket item count to get to: 200000 curr count: 160589
2021/12/01 23:56:30 Waiting for src bucket item count to get to: 200000 curr count: 160589
2021/12/01 23:56:35 Waiting for src bucket item count to get to: 200000 curr count: 176245
2021/12/01 23:56:40 Waiting for src bucket item count to get to: 200000 curr count: 176245
2021/12/01 23:56:45 Waiting for src bucket item count to get to: 200000 curr count: 192325
2021/12/01 23:56:50 Waiting for src bucket item count to get to: 200000 curr count: 192325
2021/12/01 23:56:55 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/01 23:56:55 Post SET ops. Expected item count: 200000 got 200000
2021/12/01 23:56:55 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 23:57:00 Waiting for app: TestDuplicateWithEventingRebOut to get un-deployed
2021/12/01 23:57:05 Waiting for app: TestDuplicateWithEventingRebOut to get un-deployed
2021/12/01 23:57:10 Waiting for app: TestDuplicateWithEventingRebOut to get un-deployed
2021/12/01 23:57:15 Waiting for app: TestDuplicateWithEventingRebOut to get un-deployed
2021/12/01 23:57:15 App: TestDuplicateWithEventingRebOut got un-deployed
2021/12/01 23:57:15 No eventing-con process running
2021/12/01 23:57:15 Function: TestDuplicateWithEventingRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithEventingRebOut deleting in the background"
}
2021/12/01 23:57:17 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 23:57:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 23:57:27 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/01 23:57:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 23:57:29 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 23:57:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 23:57:39 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/01 23:57:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithEventingRebOut (234.18s)
=== RUN   TestDuplicateWithEventingSwapReb
2021/12/01 23:57:49 Waiting for app: TestDuplicateWithEventingSwapReb to get un-deployed
2021/12/01 23:57:49 App: TestDuplicateWithEventingSwapReb got un-deployed
2021/12/01 23:57:49 No eventing-con process running
2021/12/01 23:57:49 Function: TestDuplicateWithEventingSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithEventingSwapReb not found"
 }
}
2021/12/01 23:57:51 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 23:57:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 23:57:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 23:57:58 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 23:58:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 23:58:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 23:58:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 23:58:08 Deploying app: TestDuplicateWithEventingSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/01 23:58:09 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithEventingSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/01 23:58:14 Waiting for app: TestDuplicateWithEventingSwapReb to get deployed
2021/12/01 23:58:19 Waiting for app: TestDuplicateWithEventingSwapReb to get deployed
2021/12/01 23:58:24 Waiting for app: TestDuplicateWithEventingSwapReb to get deployed
2021/12/01 23:58:29 Waiting for app: TestDuplicateWithEventingSwapReb to get deployed
2021/12/01 23:58:29 App: TestDuplicateWithEventingSwapReb got deployed
2021/12/01 23:58:29 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/01 23:58:29 Starting bucket ops to source bucket
2021/12/01 23:58:29 Removing node(s): [] from the cluster
2021/12/01 23:58:34 Rebalance progress: 25
2021/12/01 23:58:39 Rebalance progress: 25
2021/12/01 23:58:44 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 23:58:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 23:58:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 23:58:44 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9302/api/v1/stats": dial tcp 127.0.0.1:9302: connect: connection refused
2021/12/01 23:58:44 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/01 23:58:44 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/01 23:58:51 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/01 23:58:56 Rebalance progress: 16.66666666666667
2021/12/01 23:59:01 Rebalance progress: 16.66666666666667
2021/12/01 23:59:06 Rebalance progress: 16.66666666666667
2021/12/01 23:59:11 Rebalance progress: 72.22222222222223
2021/12/01 23:59:16 Rebalance progress: 72.22222222222223
2021/12/01 23:59:21 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 23:59:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 23:59:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 23:59:22 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/01 23:59:22 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/01 23:59:27 Rebalance progress: 16.66666666666667
2021/12/01 23:59:32 Rebalance progress: 16.66666666666667
2021/12/01 23:59:37 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 23:59:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 23:59:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 23:59:37 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/01 23:59:37 Waiting for src bucket item count to get to: 200000 curr count: 90736
2021/12/01 23:59:42 Waiting for src bucket item count to get to: 200000 curr count: 90736
2021/12/01 23:59:47 Waiting for src bucket item count to get to: 200000 curr count: 106853
2021/12/01 23:59:52 Waiting for src bucket item count to get to: 200000 curr count: 106853
2021/12/01 23:59:57 Waiting for src bucket item count to get to: 200000 curr count: 123096
2021/12/02 00:00:02 Waiting for src bucket item count to get to: 200000 curr count: 123096
2021/12/02 00:00:07 Waiting for src bucket item count to get to: 200000 curr count: 138862
2021/12/02 00:00:12 Waiting for src bucket item count to get to: 200000 curr count: 154531
2021/12/02 00:00:17 Waiting for src bucket item count to get to: 200000 curr count: 154531
2021/12/02 00:00:22 Waiting for src bucket item count to get to: 200000 curr count: 170074
2021/12/02 00:00:27 Waiting for src bucket item count to get to: 200000 curr count: 170074
2021/12/02 00:00:32 Waiting for src bucket item count to get to: 200000 curr count: 185927
2021/12/02 00:00:37 Waiting for src bucket item count to get to: 200000 curr count: 185927
2021/12/02 00:00:42 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/02 00:00:42 Post SET ops. Expected item count: 200000 got 200000
2021/12/02 00:00:42 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 00:00:47 Waiting for app: TestDuplicateWithEventingSwapReb to get un-deployed
2021/12/02 00:00:52 Waiting for app: TestDuplicateWithEventingSwapReb to get un-deployed
2021/12/02 00:00:57 Waiting for app: TestDuplicateWithEventingSwapReb to get un-deployed
2021/12/02 00:01:02 Waiting for app: TestDuplicateWithEventingSwapReb to get un-deployed
2021/12/02 00:01:02 App: TestDuplicateWithEventingSwapReb got un-deployed
2021/12/02 00:01:02 No eventing-con process running
2021/12/02 00:01:02 Function: TestDuplicateWithEventingSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithEventingSwapReb deleting in the background"
}
2021/12/02 00:01:04 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:01:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 00:01:14 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/02 00:01:19 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/02 00:01:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 00:01:20 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:01:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 00:01:30 Waiting for dst bucket item count to get to: 0 curr count: 97586
2021/12/02 00:01:35 Waiting for dst bucket item count to get to: 0 curr count: 97586
2021/12/02 00:01:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithEventingSwapReb (236.45s)
=== RUN   TestDuplicateWithKVRebIn
2021/12/02 00:01:45 Waiting for app: TestDuplicateWithKVRebIn to get un-deployed
2021/12/02 00:01:45 App: TestDuplicateWithKVRebIn got un-deployed
2021/12/02 00:01:45 No eventing-con process running
2021/12/02 00:01:45 Function: TestDuplicateWithKVRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithKVRebIn not found"
 }
}
2021/12/02 00:01:47 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:01:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 00:01:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 00:01:54 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:01:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 00:01:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:02:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 00:02:04 Deploying app: TestDuplicateWithKVRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/02 00:02:05 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithKVRebIn' in metakv",
  "warnings": null
 }
}
2021/12/02 00:02:10 Waiting for app: TestDuplicateWithKVRebIn to get deployed
2021/12/02 00:02:15 Waiting for app: TestDuplicateWithKVRebIn to get deployed
2021/12/02 00:02:20 Waiting for app: TestDuplicateWithKVRebIn to get deployed
2021/12/02 00:02:25 Waiting for app: TestDuplicateWithKVRebIn to get deployed
2021/12/02 00:02:25 App: TestDuplicateWithKVRebIn got deployed
2021/12/02 00:02:25 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/02 00:02:25 Starting bucket ops to source bucket
2021/12/02 00:02:25 Removing node(s): [] from the cluster
2021/12/02 00:02:30 Rebalance progress: 16.66666666666667
2021/12/02 00:02:35 Rebalance progress: 16.66666666666667
2021/12/02 00:02:40 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:02:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:02:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 00:02:41 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/02 00:02:41 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/02 00:02:46 Rebalance progress: 16.66666666666667
2021/12/02 00:02:51 Rebalance progress: 16.66666666666667
2021/12/02 00:02:56 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:02:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:02:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 00:02:56 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/02 00:02:56 Waiting for src bucket item count to get to: 200000 curr count: 37423
2021/12/02 00:03:01 Waiting for src bucket item count to get to: 200000 curr count: 53577
2021/12/02 00:03:06 Waiting for src bucket item count to get to: 200000 curr count: 53577
2021/12/02 00:03:11 Waiting for src bucket item count to get to: 200000 curr count: 69770
2021/12/02 00:03:16 Waiting for src bucket item count to get to: 200000 curr count: 69770
2021/12/02 00:03:21 Waiting for src bucket item count to get to: 200000 curr count: 85424
2021/12/02 00:03:26 Waiting for src bucket item count to get to: 200000 curr count: 85424
2021/12/02 00:03:31 Waiting for src bucket item count to get to: 200000 curr count: 101439
2021/12/02 00:03:36 Waiting for src bucket item count to get to: 200000 curr count: 101439
2021/12/02 00:03:41 Waiting for src bucket item count to get to: 200000 curr count: 101439
2021/12/02 00:03:46 Waiting for src bucket item count to get to: 200000 curr count: 117563
2021/12/02 00:03:51 Waiting for src bucket item count to get to: 200000 curr count: 117563
2021/12/02 00:03:56 Waiting for src bucket item count to get to: 200000 curr count: 133581
2021/12/02 00:04:01 Waiting for src bucket item count to get to: 200000 curr count: 133581
2021/12/02 00:04:06 Waiting for src bucket item count to get to: 200000 curr count: 149925
2021/12/02 00:04:11 Waiting for src bucket item count to get to: 200000 curr count: 149925
2021/12/02 00:04:16 Waiting for src bucket item count to get to: 200000 curr count: 166231
2021/12/02 00:04:21 Waiting for src bucket item count to get to: 200000 curr count: 166231
2021/12/02 00:04:26 Waiting for src bucket item count to get to: 200000 curr count: 181697
2021/12/02 00:04:31 Waiting for src bucket item count to get to: 200000 curr count: 181697
2021/12/02 00:04:36 Waiting for src bucket item count to get to: 200000 curr count: 197789
2021/12/02 00:04:41 Waiting for src bucket item count to get to: 200000 curr count: 197789
2021/12/02 00:04:46 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/02 00:04:46 Post SET ops. Expected item count: 200000 got 200000
2021/12/02 00:04:46 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 00:04:51 Waiting for app: TestDuplicateWithKVRebIn to get un-deployed
2021/12/02 00:04:56 Waiting for app: TestDuplicateWithKVRebIn to get un-deployed
2021/12/02 00:05:01 Waiting for app: TestDuplicateWithKVRebIn to get un-deployed
2021/12/02 00:05:06 Waiting for app: TestDuplicateWithKVRebIn to get un-deployed
2021/12/02 00:05:06 App: TestDuplicateWithKVRebIn got un-deployed
2021/12/02 00:05:06 No eventing-con process running
2021/12/02 00:05:06 Function: TestDuplicateWithKVRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithKVRebIn deleting in the background"
}
2021/12/02 00:05:08 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:05:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 00:05:18 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/02 00:05:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 00:05:19 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:05:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 00:05:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithKVRebIn (229.09s)
=== RUN   TestDuplicateWithKVRebOut
2021/12/02 00:05:30 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/02 00:05:30 Removing node(s): [] from the cluster
2021/12/02 00:05:35 Rebalance progress: 100
2021/12/02 00:05:40 Waiting for app: TestDuplicateWithKVRebOut to get un-deployed
2021/12/02 00:05:40 App: TestDuplicateWithKVRebOut got un-deployed
2021/12/02 00:05:40 No eventing-con process running
2021/12/02 00:05:40 Function: TestDuplicateWithKVRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithKVRebOut not found"
 }
}
2021/12/02 00:05:42 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:05:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 00:05:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 00:05:49 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:05:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 00:05:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:05:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 00:05:59 Deploying app: TestDuplicateWithKVRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/02 00:06:00 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithKVRebOut' in metakv",
  "warnings": null
 }
}
2021/12/02 00:06:05 Waiting for app: TestDuplicateWithKVRebOut to get deployed
2021/12/02 00:06:10 Waiting for app: TestDuplicateWithKVRebOut to get deployed
2021/12/02 00:06:15 Waiting for app: TestDuplicateWithKVRebOut to get deployed
2021/12/02 00:06:20 Waiting for app: TestDuplicateWithKVRebOut to get deployed
2021/12/02 00:06:20 App: TestDuplicateWithKVRebOut got deployed
2021/12/02 00:06:20 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/02 00:06:20 Starting bucket ops to source bucket
2021/12/02 00:06:25 Rebalance progress: 16.66666666666667
2021/12/02 00:06:30 Rebalance progress: 16.66666666666667
2021/12/02 00:06:35 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:06:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:06:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 00:06:35 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/02 00:06:35 Waiting for src bucket item count to get to: 200000 curr count: 14081
2021/12/02 00:06:40 Waiting for src bucket item count to get to: 200000 curr count: 30131
2021/12/02 00:06:45 Waiting for src bucket item count to get to: 200000 curr count: 30131
2021/12/02 00:06:50 Waiting for src bucket item count to get to: 200000 curr count: 46178
2021/12/02 00:06:55 Waiting for src bucket item count to get to: 200000 curr count: 46178
2021/12/02 00:07:00 Waiting for src bucket item count to get to: 200000 curr count: 62347
2021/12/02 00:07:05 Waiting for src bucket item count to get to: 200000 curr count: 62347
2021/12/02 00:07:10 Waiting for src bucket item count to get to: 200000 curr count: 62347
2021/12/02 00:07:15 Waiting for src bucket item count to get to: 200000 curr count: 77914
2021/12/02 00:07:20 Waiting for src bucket item count to get to: 200000 curr count: 77914
2021/12/02 00:07:25 Waiting for src bucket item count to get to: 200000 curr count: 93953
2021/12/02 00:07:30 Waiting for src bucket item count to get to: 200000 curr count: 93953
2021/12/02 00:07:35 Waiting for src bucket item count to get to: 200000 curr count: 110031
2021/12/02 00:07:40 Waiting for src bucket item count to get to: 200000 curr count: 110031
2021/12/02 00:07:45 Waiting for src bucket item count to get to: 200000 curr count: 126228
2021/12/02 00:07:50 Waiting for src bucket item count to get to: 200000 curr count: 126228
2021/12/02 00:07:55 Waiting for src bucket item count to get to: 200000 curr count: 142657
2021/12/02 00:08:00 Waiting for src bucket item count to get to: 200000 curr count: 142657
2021/12/02 00:08:05 Waiting for src bucket item count to get to: 200000 curr count: 158975
2021/12/02 00:08:10 Waiting for src bucket item count to get to: 200000 curr count: 158975
2021/12/02 00:08:15 Waiting for src bucket item count to get to: 200000 curr count: 174546
2021/12/02 00:08:20 Waiting for src bucket item count to get to: 200000 curr count: 174546
2021/12/02 00:08:25 Waiting for src bucket item count to get to: 200000 curr count: 190225
2021/12/02 00:08:30 Waiting for src bucket item count to get to: 200000 curr count: 190225
2021/12/02 00:08:35 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/02 00:08:35 Post SET ops. Expected item count: 200000 got 200000
2021/12/02 00:08:35 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 00:08:40 Waiting for app: TestDuplicateWithKVRebOut to get un-deployed
2021/12/02 00:08:45 Waiting for app: TestDuplicateWithKVRebOut to get un-deployed
2021/12/02 00:08:50 Waiting for app: TestDuplicateWithKVRebOut to get un-deployed
2021/12/02 00:08:55 Waiting for app: TestDuplicateWithKVRebOut to get un-deployed
2021/12/02 00:08:55 App: TestDuplicateWithKVRebOut got un-deployed
2021/12/02 00:08:55 No eventing-con process running
2021/12/02 00:08:55 Function: TestDuplicateWithKVRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithKVRebOut deleting in the background"
}
2021/12/02 00:08:57 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:09:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 00:09:07 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/02 00:09:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 00:09:09 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:09:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 00:09:19 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/02 00:09:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithKVRebOut (234.42s)
=== RUN   TestDuplicateWithKVSwapReb
2021/12/02 00:09:29 Waiting for app: TestDuplicateWithKVSwapReb to get un-deployed
2021/12/02 00:09:29 App: TestDuplicateWithKVSwapReb got un-deployed
2021/12/02 00:09:29 No eventing-con process running
2021/12/02 00:09:29 Function: TestDuplicateWithKVSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithKVSwapReb not found"
 }
}
2021/12/02 00:09:31 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:09:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 00:09:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 00:09:38 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:09:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 00:09:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:09:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 00:09:48 Deploying app: TestDuplicateWithKVSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/02 00:09:49 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithKVSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/02 00:09:54 Waiting for app: TestDuplicateWithKVSwapReb to get deployed
2021/12/02 00:09:59 Waiting for app: TestDuplicateWithKVSwapReb to get deployed
2021/12/02 00:10:04 Waiting for app: TestDuplicateWithKVSwapReb to get deployed
2021/12/02 00:10:09 Waiting for app: TestDuplicateWithKVSwapReb to get deployed
2021/12/02 00:10:09 App: TestDuplicateWithKVSwapReb got deployed
2021/12/02 00:10:09 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/02 00:10:09 Starting bucket ops to source bucket
2021/12/02 00:10:09 Removing node(s): [] from the cluster
2021/12/02 00:10:14 Rebalance progress: 16.66666666666667
2021/12/02 00:10:19 Rebalance progress: 16.66666666666667
2021/12/02 00:10:24 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:10:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:10:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 00:10:24 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/02 00:10:24 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/02 00:10:25 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/02 00:10:30 Rebalance progress: 16.66666666666667
2021/12/02 00:10:35 Rebalance progress: 16.66666666666667
2021/12/02 00:10:40 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:10:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:10:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 00:10:40 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/02 00:10:40 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/02 00:10:45 Rebalance progress: 16.66666666666667
2021/12/02 00:10:50 Rebalance progress: 16.66666666666667
2021/12/02 00:10:55 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:10:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:10:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 00:10:55 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/02 00:10:55 Waiting for src bucket item count to get to: 200000 curr count: 63242
2021/12/02 00:11:00 Waiting for src bucket item count to get to: 200000 curr count: 63242
2021/12/02 00:11:05 Waiting for src bucket item count to get to: 200000 curr count: 78535
2021/12/02 00:11:10 Waiting for src bucket item count to get to: 200000 curr count: 78535
2021/12/02 00:11:15 Waiting for src bucket item count to get to: 200000 curr count: 94318
2021/12/02 00:11:20 Waiting for src bucket item count to get to: 200000 curr count: 94318
2021/12/02 00:11:25 Waiting for src bucket item count to get to: 200000 curr count: 110174
2021/12/02 00:11:30 Waiting for src bucket item count to get to: 200000 curr count: 110174
2021/12/02 00:11:35 Waiting for src bucket item count to get to: 200000 curr count: 126069
2021/12/02 00:11:40 Waiting for src bucket item count to get to: 200000 curr count: 126069
2021/12/02 00:11:45 Waiting for src bucket item count to get to: 200000 curr count: 142443
2021/12/02 00:11:50 Waiting for src bucket item count to get to: 200000 curr count: 158546
2021/12/02 00:11:55 Waiting for src bucket item count to get to: 200000 curr count: 158546
2021/12/02 00:12:00 Waiting for src bucket item count to get to: 200000 curr count: 173780
2021/12/02 00:12:05 Waiting for src bucket item count to get to: 200000 curr count: 173780
2021/12/02 00:12:10 Waiting for src bucket item count to get to: 200000 curr count: 173780
2021/12/02 00:12:15 Waiting for src bucket item count to get to: 200000 curr count: 189399
2021/12/02 00:12:20 Waiting for src bucket item count to get to: 200000 curr count: 189399
2021/12/02 00:12:25 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/02 00:12:25 Post SET ops. Expected item count: 200000 got 200000
2021/12/02 00:12:25 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 00:12:30 Waiting for app: TestDuplicateWithKVSwapReb to get un-deployed
2021/12/02 00:12:35 Waiting for app: TestDuplicateWithKVSwapReb to get un-deployed
2021/12/02 00:12:40 Waiting for app: TestDuplicateWithKVSwapReb to get un-deployed
2021/12/02 00:12:45 Waiting for app: TestDuplicateWithKVSwapReb to get un-deployed
2021/12/02 00:12:45 App: TestDuplicateWithKVSwapReb got un-deployed
2021/12/02 00:12:45 No eventing-con process running
2021/12/02 00:12:45 Function: TestDuplicateWithKVSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithKVSwapReb deleting in the background"
}
2021/12/02 00:12:47 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:12:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 00:12:57 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/02 00:12:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 00:12:59 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:13:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 00:13:09 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/02 00:13:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithKVSwapReb (229.73s)
=== RUN   TestEventingRebKVOpsOnUpdateBucketOpOneByOne
2021/12/02 00:13:19 Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/02 00:13:19 App: TestEventingRebKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/02 00:13:19 No eventing-con process running
2021/12/02 00:13:19 Function: TestEventingRebKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpOneByOne not found"
 }
}
2021/12/02 00:13:20 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:13:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 00:13:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 00:13:27 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:13:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 00:13:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:13:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 00:13:37 Deploying app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne. Item count src bucket: 0 dst bucket: 0
2021/12/02 00:13:38 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebKVOpsOnUpdateBucketOpOneByOne' in metakv",
  "warnings": null
 }
}
2021/12/02 00:13:43 Starting bucket ops to source bucket
2021/12/02 00:13:48 Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/02 00:13:53 Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/02 00:13:58 Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/02 00:13:58 App: TestEventingRebKVOpsOnUpdateBucketOpOneByOne got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:13:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:13:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 00:13:58 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/02 00:13:58 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/02 00:13:59 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/02 00:14:04 Rebalance progress: 16.66666666666667
2021/12/02 00:14:09 Rebalance progress: 16.66666666666667
2021/12/02 00:14:14 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:14:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:14:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 00:14:14 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/02 00:14:14 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/02 00:14:14 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/02 00:14:19 Rebalance progress: 16.66666666666667
2021/12/02 00:14:24 Rebalance progress: 16.66666666666667
2021/12/02 00:14:29 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:14:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:14:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 00:14:29 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/02 00:14:29 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/02 00:14:34 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/02 00:14:39 Rebalance progress: 12.5
2021/12/02 00:14:44 Rebalance progress: 12.5
2021/12/02 00:14:49 Rebalance progress: 12.5
2021/12/02 00:14:54 Rebalance progress: 70.83333333333334
2021/12/02 00:14:59 Rebalance progress: 70.83333333333334
2021/12/02 00:15:04 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:15:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:15:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 00:15:04 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/02 00:15:09 Rebalance progress: 12.5
2021/12/02 00:15:14 Rebalance progress: 12.5
2021/12/02 00:15:19 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:15:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:15:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 00:15:19 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/02 00:15:24 Rebalance progress: 12.5
2021/12/02 00:15:29 Rebalance progress: 12.5
2021/12/02 00:15:34 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:15:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:15:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 00:15:34 clusterSetup.go:253: Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/02 00:15:40 Rebalance progress: 12.5
2021/12/02 00:15:45 Rebalance progress: 12.5
2021/12/02 00:15:50 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:15:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:15:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 00:15:50 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 00:15:55 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/02 00:16:00 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/02 00:16:05 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/02 00:16:10 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/02 00:16:10 clusterSetup.go:405: App: TestEventingRebKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/02 00:16:10 eventingSetup.go:46: No eventing-con process running
2021/12/02 00:16:10 eventingSetup.go:418: Function: TestEventingRebKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpOneByOne deleting in the background"
}
2021/12/02 00:16:12 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:16:17 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 00:16:22 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12619
2021/12/02 00:16:27 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12619
2021/12/02 00:16:27 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 00:16:28 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:16:33 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 00:16:38 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12619
2021/12/02 00:16:38 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebKVOpsOnUpdateBucketOpOneByOne (209.77s)
=== RUN   TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce
2021/12/02 00:16:48 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get un-deployed
2021/12/02 00:16:48 clusterSetup.go:405: App: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce got un-deployed
2021/12/02 00:16:48 eventingSetup.go:46: No eventing-con process running
2021/12/02 00:16:48 eventingSetup.go:418: Function: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce not found"
 }
}
2021/12/02 00:16:50 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:16:55 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 00:16:55 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 00:16:57 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:17:02 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 00:17:02 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:17:07 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 00:17:07 eventingSetup.go:125: Deploying app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce. Item count src bucket: 0 dst bucket: 0
2021/12/02 00:17:08 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce' in metakv",
  "warnings": null
 }
}
2021/12/02 00:17:13 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/02 00:17:18 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get deployed
2021/12/02 00:17:23 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get deployed
2021/12/02 00:17:28 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get deployed
2021/12/02 00:17:28 clusterSetup.go:354: App: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:17:28 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:17:28 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 00:17:28 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/02 00:17:29 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/02 00:17:29 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/02 00:17:30 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/02 00:17:35 Rebalance progress: 12.5
2021/12/02 00:17:40 Rebalance progress: 12.5
2021/12/02 00:17:45 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:17:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:17:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 00:17:45 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002 127.0.0.1:19003] from the cluster
2021/12/02 00:17:50 Rebalance progress: 12.5
2021/12/02 00:17:55 Rebalance progress: 12.5
2021/12/02 00:18:00 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:18:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:18:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 00:18:00 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 00:18:05 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get un-deployed
2021/12/02 00:18:10 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get un-deployed
2021/12/02 00:18:15 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get un-deployed
2021/12/02 00:18:20 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get un-deployed
2021/12/02 00:18:20 clusterSetup.go:405: App: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce got un-deployed
2021/12/02 00:18:20 eventingSetup.go:46: No eventing-con process running
2021/12/02 00:18:20 eventingSetup.go:418: Function: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce deleting in the background"
}
2021/12/02 00:18:22 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:18:27 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 00:18:32 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4639
2021/12/02 00:18:37 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4639
2021/12/02 00:18:37 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 00:18:38 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:18:43 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 00:18:48 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4639
2021/12/02 00:18:48 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce (130.02s)
=== RUN   TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings
2021/12/02 00:18:58 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/02 00:18:58 clusterSetup.go:405: App: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings got un-deployed
2021/12/02 00:18:58 eventingSetup.go:46: No eventing-con process running
2021/12/02 00:18:58 eventingSetup.go:418: Function: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings not found"
 }
}
2021/12/02 00:19:00 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:19:05 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 00:19:05 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 00:19:07 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:19:12 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 00:19:12 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:19:17 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 00:19:17 eventingSetup.go:125: Deploying app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/02 00:19:18 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/02 00:19:23 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/02 00:19:28 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get deployed
2021/12/02 00:19:33 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get deployed
2021/12/02 00:19:38 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get deployed
2021/12/02 00:19:38 clusterSetup.go:354: App: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:19:38 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:19:38 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 00:19:38 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/02 00:19:38 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/02 00:19:44 Rebalance progress: 12.5
2021/12/02 00:19:49 Rebalance progress: 12.5
2021/12/02 00:19:54 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:19:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:19:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 00:19:54 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/02 00:19:54 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/02 00:19:59 Rebalance progress: 12.5
2021/12/02 00:20:04 Rebalance progress: 12.5
2021/12/02 00:20:09 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:20:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:20:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 00:20:09 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/02 00:20:09 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/02 00:20:14 Rebalance progress: 12.5
2021/12/02 00:20:19 Rebalance progress: 12.5
2021/12/02 00:20:24 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:20:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:20:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 00:20:24 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/02 00:20:29 Rebalance progress: 12.5
2021/12/02 00:20:34 Rebalance progress: 12.5
2021/12/02 00:20:39 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:20:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:20:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 00:20:39 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/02 00:20:44 Rebalance progress: 12.5
2021/12/02 00:20:49 Rebalance progress: 12.5
2021/12/02 00:20:54 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:20:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:20:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 00:20:54 clusterSetup.go:253: Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/02 00:20:59 Rebalance progress: 12.5
2021/12/02 00:21:05 Rebalance progress: 12.5
2021/12/02 00:21:09 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:21:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:21:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 00:21:10 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 00:21:15 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/02 00:21:20 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/02 00:21:25 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/02 00:21:30 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/02 00:21:30 clusterSetup.go:405: App: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings got un-deployed
2021/12/02 00:21:30 eventingSetup.go:46: No eventing-con process running
2021/12/02 00:21:30 eventingSetup.go:418: Function: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings deleting in the background"
}
2021/12/02 00:21:32 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:21:37 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 00:21:42 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 10628
2021/12/02 00:21:47 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 10628
2021/12/02 00:21:47 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 00:21:48 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:21:53 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 00:21:58 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 10625
2021/12/02 00:21:58 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings (189.90s)
=== RUN   TestEventingSwapRebOnUpdateBucketOp
2021/12/02 00:22:08 clusterSetup.go:397: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get un-deployed
2021/12/02 00:22:08 clusterSetup.go:405: App: TestEventingSwapRebOnUpdateBucketOp got un-deployed
2021/12/02 00:22:08 eventingSetup.go:46: No eventing-con process running
2021/12/02 00:22:08 eventingSetup.go:418: Function: TestEventingSwapRebOnUpdateBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingSwapRebOnUpdateBucketOp not found"
 }
}
2021/12/02 00:22:10 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:22:15 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 00:22:15 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 00:22:17 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:22:22 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 00:22:22 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:22:22 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 00:22:22 eventingSetup.go:125: Deploying app: TestEventingSwapRebOnUpdateBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/02 00:22:23 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingSwapRebOnUpdateBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/02 00:22:28 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/02 00:22:33 clusterSetup.go:346: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get deployed
2021/12/02 00:22:38 clusterSetup.go:346: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get deployed
2021/12/02 00:22:43 clusterSetup.go:346: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get deployed
2021/12/02 00:22:43 clusterSetup.go:354: App: TestEventingSwapRebOnUpdateBucketOp got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:22:43 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:22:43 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 00:22:43 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/02 00:22:43 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/02 00:22:48 Rebalance progress: 12.5
2021/12/02 00:22:53 Rebalance progress: 12.5
2021/12/02 00:22:58 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:22:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:22:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 00:22:58 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/02 00:22:59 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/02 00:23:04 Rebalance progress: 12.5
2021/12/02 00:23:09 Rebalance progress: 12.5
2021/12/02 00:23:14 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:23:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:23:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 00:23:14 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/02 00:23:19 Rebalance progress: 12.5
2021/12/02 00:23:24 Rebalance progress: 12.5
2021/12/02 00:23:29 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:23:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:23:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 00:23:29 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 00:23:34 clusterSetup.go:397: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get un-deployed
2021/12/02 00:23:39 clusterSetup.go:397: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get un-deployed
2021/12/02 00:23:44 clusterSetup.go:397: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get un-deployed
2021/12/02 00:23:49 clusterSetup.go:397: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get un-deployed
2021/12/02 00:23:49 clusterSetup.go:405: App: TestEventingSwapRebOnUpdateBucketOp got un-deployed
2021/12/02 00:23:49 eventingSetup.go:46: No eventing-con process running
2021/12/02 00:23:49 eventingSetup.go:418: Function: TestEventingSwapRebOnUpdateBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingSwapRebOnUpdateBucketOp deleting in the background"
}
2021/12/02 00:23:51 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:23:56 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 00:24:01 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6098
2021/12/02 00:24:06 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6098
2021/12/02 00:24:06 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 00:24:08 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:24:13 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 00:24:18 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6098
2021/12/02 00:24:18 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingSwapRebOnUpdateBucketOp (139.53s)
=== RUN   TestMetaRollbackWithEventingReb
2021/12/02 00:24:23 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/02 00:24:23 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/02 00:24:28 Rebalance progress: 100
2021/12/02 00:24:33 Waiting for app: TestMetaRollbackWithEventingReb to get un-deployed
2021/12/02 00:24:33 App: TestMetaRollbackWithEventingReb got un-deployed
2021/12/02 00:24:33 No eventing-con process running
2021/12/02 00:24:33 Function: TestMetaRollbackWithEventingReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMetaRollbackWithEventingReb not found"
 }
}
2021/12/02 00:24:35 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:24:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 00:24:40 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 00:24:42 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:24:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 00:24:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:24:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 00:24:47 Deploying app: TestMetaRollbackWithEventingReb. Item count src bucket: 0 dst bucket: 0
2021/12/02 00:24:48 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMetaRollbackWithEventingReb' in metakv",
  "warnings": null
 }
}
2021/12/02 00:24:53 Starting bucket ops to source bucket
2021/12/02 00:24:58 Waiting for app: TestMetaRollbackWithEventingReb to get deployed
2021/12/02 00:25:03 Waiting for app: TestMetaRollbackWithEventingReb to get deployed
2021/12/02 00:25:08 Waiting for app: TestMetaRollbackWithEventingReb to get deployed
2021/12/02 00:25:08 App: TestMetaRollbackWithEventingReb got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:25:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:25:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 00:25:08 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/02 00:25:08 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/02 00:25:13 Rebalance progress: 12.5
2021/12/02 00:25:18 Rebalance progress: 12.5
2021/12/02 00:25:23 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:25:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:25:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 00:25:23 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002] from the cluster
2021/12/02 00:25:23 bucketOps.go:134: Metakv lookup failed, err: Get "/_metakv/eventing/tempApps/TestMetaRollbackWithEventingReb/0": Unable to initialize cbauth's revrpc: cbauth environment variable CBAUTH_REVRPC_URL is not set
2021/12/02 00:25:29 Rebalance progress: 12.5
2021/12/02 00:25:34 Rebalance progress: 12.5
2021/12/02 00:25:39 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:25:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:25:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 00:25:39 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 00:25:44 clusterSetup.go:397: Waiting for app: TestMetaRollbackWithEventingReb to get un-deployed
2021/12/02 00:25:49 clusterSetup.go:397: Waiting for app: TestMetaRollbackWithEventingReb to get un-deployed
2021/12/02 00:25:54 clusterSetup.go:397: Waiting for app: TestMetaRollbackWithEventingReb to get un-deployed
2021/12/02 00:25:59 clusterSetup.go:397: Waiting for app: TestMetaRollbackWithEventingReb to get un-deployed
2021/12/02 00:25:59 clusterSetup.go:405: App: TestMetaRollbackWithEventingReb got un-deployed
2021/12/02 00:25:59 eventingSetup.go:46: No eventing-con process running
2021/12/02 00:25:59 eventingSetup.go:418: Function: TestMetaRollbackWithEventingReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMetaRollbackWithEventingReb deleting in the background"
}
2021/12/02 00:26:01 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:26:06 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 00:26:11 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2593
2021/12/02 00:26:16 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2593
2021/12/02 00:26:16 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 00:26:17 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:26:22 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 00:26:27 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4555
2021/12/02 00:26:27 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestMetaRollbackWithEventingReb (129.49s)
=== RUN   TestMetaPartialRollbackWithEventingReb
2021/12/02 00:26:32 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/02 00:26:33 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/02 00:26:38 Rebalance progress: 100
2021/12/02 00:26:43 Waiting for app: TestMetaPartialRollbackWithEventingReb to get un-deployed
2021/12/02 00:26:43 App: TestMetaPartialRollbackWithEventingReb got un-deployed
2021/12/02 00:26:43 No eventing-con process running
2021/12/02 00:26:43 Function: TestMetaPartialRollbackWithEventingReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMetaPartialRollbackWithEventingReb not found"
 }
}
2021/12/02 00:26:44 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:26:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 00:26:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 00:26:51 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:26:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 00:26:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:26:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 00:26:56 Deploying app: TestMetaPartialRollbackWithEventingReb. Item count src bucket: 0 dst bucket: 0
2021/12/02 00:26:57 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMetaPartialRollbackWithEventingReb' in metakv",
  "warnings": null
 }
}
2021/12/02 00:27:02 Starting bucket ops to source bucket
2021/12/02 00:27:07 Waiting for app: TestMetaPartialRollbackWithEventingReb to get deployed
2021/12/02 00:27:12 Waiting for app: TestMetaPartialRollbackWithEventingReb to get deployed
2021/12/02 00:27:17 Waiting for app: TestMetaPartialRollbackWithEventingReb to get deployed
2021/12/02 00:27:17 App: TestMetaPartialRollbackWithEventingReb got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:27:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:27:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 00:27:17 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/02 00:27:18 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/02 00:27:23 Rebalance progress: 12.5
2021/12/02 00:27:28 Rebalance progress: 12.5
2021/12/02 00:27:33 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:27:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:27:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 00:27:33 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002] from the cluster
2021/12/02 00:27:33 bucketOps.go:65: Metakv lookup failed, err: Get "/_metakv/eventing/tempApps/TestMetaPartialRollbackWithEventingReb/0": Unable to initialize cbauth's revrpc: cbauth environment variable CBAUTH_REVRPC_URL is not set
2021/12/02 00:27:38 Rebalance progress: 12.5
2021/12/02 00:27:43 Rebalance progress: 12.5
2021/12/02 00:27:48 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:27:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:27:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 00:27:48 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 00:27:48 bucketOps.go:65: Metakv lookup failed, err: Get "/_metakv/eventing/tempApps/TestMetaPartialRollbackWithEventingReb/0": Unable to initialize cbauth's revrpc: cbauth environment variable CBAUTH_REVRPC_URL is not set
2021/12/02 00:27:53 clusterSetup.go:397: Waiting for app: TestMetaPartialRollbackWithEventingReb to get un-deployed
2021/12/02 00:27:58 clusterSetup.go:397: Waiting for app: TestMetaPartialRollbackWithEventingReb to get un-deployed
2021/12/02 00:28:03 clusterSetup.go:397: Waiting for app: TestMetaPartialRollbackWithEventingReb to get un-deployed
2021/12/02 00:28:08 clusterSetup.go:397: Waiting for app: TestMetaPartialRollbackWithEventingReb to get un-deployed
2021/12/02 00:28:08 clusterSetup.go:405: App: TestMetaPartialRollbackWithEventingReb got un-deployed
2021/12/02 00:28:08 eventingSetup.go:46: No eventing-con process running
2021/12/02 00:28:08 eventingSetup.go:418: Function: TestMetaPartialRollbackWithEventingReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMetaPartialRollbackWithEventingReb deleting in the background"
}
2021/12/02 00:28:10 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:28:15 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 00:28:20 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4255
2021/12/02 00:28:25 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4255
2021/12/02 00:28:25 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 00:28:27 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:28:32 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 00:28:37 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 37236
2021/12/02 00:28:37 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestMetaPartialRollbackWithEventingReb (129.49s)
=== RUN   TestEventingRebMultipleHandlersOneByOne
2021/12/02 00:28:47 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get un-deployed
2021/12/02 00:28:47 clusterSetup.go:405: App: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op got un-deployed
2021/12/02 00:28:47 eventingSetup.go:46: No eventing-con process running
2021/12/02 00:28:47 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op not found"
 }
}
2021/12/02 00:28:49 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:28:54 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 00:28:54 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 00:28:55 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:29:00 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 00:29:00 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:29:05 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 00:29:05 eventingSetup.go:125: Deploying app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op. Item count src bucket: 0 dst bucket: 0
2021/12/02 00:29:07 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op' in metakv",
  "warnings": null
 }
}
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:29:07 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 00:29:07 eventingSetup.go:125: Deploying app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/02 00:29:08 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebMultipleHandlersOneByOnebucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/02 00:29:13 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/02 00:29:18 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get deployed
2021/12/02 00:29:23 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get deployed
2021/12/02 00:29:28 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get deployed
2021/12/02 00:29:28 clusterSetup.go:354: App: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op got deployed
2021/12/02 00:29:33 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get deployed
2021/12/02 00:29:38 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get deployed
2021/12/02 00:29:43 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get deployed
2021/12/02 00:29:43 clusterSetup.go:354: App: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:29:43 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:29:43 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 00:29:43 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/02 00:29:43 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/02 00:29:48 Rebalance progress: 12.5
2021/12/02 00:29:53 Rebalance progress: 12.5
2021/12/02 00:29:58 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:29:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:29:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 00:29:58 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/02 00:29:59 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/02 00:30:04 Rebalance progress: 12.5
2021/12/02 00:30:09 Rebalance progress: 12.5
2021/12/02 00:30:14 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:30:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:30:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 00:30:14 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/02 00:30:14 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/02 00:30:19 Rebalance progress: 12.5
2021/12/02 00:30:24 Rebalance progress: 12.5
2021/12/02 00:30:29 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:30:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:30:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 00:30:29 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/02 00:30:34 Rebalance progress: 12.5
2021/12/02 00:30:39 Rebalance progress: 12.5
2021/12/02 00:30:44 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:30:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:30:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 00:30:44 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/02 00:30:49 Rebalance progress: 12.5
2021/12/02 00:30:54 Rebalance progress: 12.5
2021/12/02 00:30:59 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:30:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:30:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 00:30:59 clusterSetup.go:253: Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/02 00:31:04 Rebalance progress: 12.5
2021/12/02 00:31:09 Rebalance progress: 12.5
2021/12/02 00:31:14 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:31:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:31:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 00:31:14 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 00:31:19 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get un-deployed
2021/12/02 00:31:24 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get un-deployed
2021/12/02 00:31:29 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get un-deployed
2021/12/02 00:31:35 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get un-deployed
2021/12/02 00:31:35 clusterSetup.go:405: App: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op got un-deployed
2021/12/02 00:31:35 eventingSetup.go:46: No eventing-con process running
2021/12/02 00:31:35 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op deleting in the background"
}
2021/12/02 00:31:38 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:31:43 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 00:31:48 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12145
2021/12/02 00:31:48 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 00:31:50 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:31:55 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 00:32:00 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2760
2021/12/02 00:32:05 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2760
2021/12/02 00:32:05 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 00:32:05 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 00:32:10 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get un-deployed
2021/12/02 00:32:15 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get un-deployed
2021/12/02 00:32:20 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get un-deployed
2021/12/02 00:32:25 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get un-deployed
2021/12/02 00:32:25 clusterSetup.go:405: App: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update got un-deployed
2021/12/02 00:32:25 eventingSetup.go:46: No eventing-con process running
2021/12/02 00:32:25 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update deleting in the background"
}
2021/12/02 00:32:27 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:32:32 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 00:32:32 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 00:32:34 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:32:39 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 00:32:39 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebMultipleHandlersOneByOne (242.12s)
=== RUN   TestEventingRebMultipleHandlersAllAtOnce
2021/12/02 00:32:49 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get un-deployed
2021/12/02 00:32:49 clusterSetup.go:405: App: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op got un-deployed
2021/12/02 00:32:49 eventingSetup.go:46: No eventing-con process running
2021/12/02 00:32:49 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op not found"
 }
}
2021/12/02 00:32:51 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:32:56 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 00:32:56 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 00:32:58 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:33:03 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 00:33:03 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:33:08 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 00:33:08 eventingSetup.go:125: Deploying app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op. Item count src bucket: 0 dst bucket: 0
2021/12/02 00:33:09 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op' in metakv",
  "warnings": null
 }
}
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:33:09 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 00:33:09 eventingSetup.go:125: Deploying app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/02 00:33:10 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/02 00:33:15 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/02 00:33:20 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get deployed
2021/12/02 00:33:25 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get deployed
2021/12/02 00:33:30 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get deployed
2021/12/02 00:33:30 clusterSetup.go:354: App: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op got deployed
2021/12/02 00:33:35 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get deployed
2021/12/02 00:33:40 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get deployed
2021/12/02 00:33:45 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get deployed
2021/12/02 00:33:45 clusterSetup.go:354: App: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:33:45 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:33:45 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 00:33:45 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/02 00:33:45 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/02 00:33:46 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/02 00:33:46 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/02 00:33:51 Rebalance progress: 12.5
2021/12/02 00:33:56 Rebalance progress: 12.5
2021/12/02 00:34:01 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:34:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:34:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 00:34:02 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002 127.0.0.1:19003] from the cluster
2021/12/02 00:34:07 Rebalance progress: 12.5
2021/12/02 00:34:12 Rebalance progress: 12.5
2021/12/02 00:34:17 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:34:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:34:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 00:34:17 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 00:34:22 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get un-deployed
2021/12/02 00:34:27 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get un-deployed
2021/12/02 00:34:32 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get un-deployed
2021/12/02 00:34:37 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get un-deployed
2021/12/02 00:34:37 clusterSetup.go:405: App: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op got un-deployed
2021/12/02 00:34:37 eventingSetup.go:46: No eventing-con process running
2021/12/02 00:34:37 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op deleting in the background"
}
2021/12/02 00:34:40 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:34:45 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 00:34:50 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 652
2021/12/02 00:34:55 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 652
2021/12/02 00:34:55 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 00:34:57 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:35:02 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 00:35:07 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6164
2021/12/02 00:35:07 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 00:35:07 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 00:35:12 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get un-deployed
2021/12/02 00:35:17 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get un-deployed
2021/12/02 00:35:22 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get un-deployed
2021/12/02 00:35:27 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get un-deployed
2021/12/02 00:35:27 clusterSetup.go:405: App: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update got un-deployed
2021/12/02 00:35:27 eventingSetup.go:46: No eventing-con process running
2021/12/02 00:35:28 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update deleting in the background"
}
2021/12/02 00:35:29 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:35:34 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 00:35:34 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 00:35:36 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:35:41 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 00:35:41 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebMultipleHandlersAllAtOnce (182.23s)
=== RUN   TestEventingSwapRebMultipleHandlers
2021/12/02 00:35:51 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get un-deployed
2021/12/02 00:35:51 clusterSetup.go:405: App: TestEventingSwapRebMultipleHandlersbucket_op_on_update got un-deployed
2021/12/02 00:35:51 eventingSetup.go:46: No eventing-con process running
2021/12/02 00:35:51 eventingSetup.go:418: Function: TestEventingSwapRebMultipleHandlersbucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingSwapRebMultipleHandlersbucket_op_on_update not found"
 }
}
2021/12/02 00:35:53 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:35:58 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 00:35:58 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 00:36:00 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:36:05 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 00:36:05 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 00:36:10 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get un-deployed
2021/12/02 00:36:10 clusterSetup.go:405: App: TestEventingSwapRebMultipleHandlerssys_test_bucket_op got un-deployed
2021/12/02 00:36:10 eventingSetup.go:46: No eventing-con process running
2021/12/02 00:36:10 eventingSetup.go:418: Function: TestEventingSwapRebMultipleHandlerssys_test_bucket_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingSwapRebMultipleHandlerssys_test_bucket_op not found"
 }
}
2021/12/02 00:36:12 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:36:17 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 00:36:17 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 00:36:18 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:36:23 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 00:36:23 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:36:23 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 00:36:23 eventingSetup.go:125: Deploying app: TestEventingSwapRebMultipleHandlersbucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/02 00:36:25 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingSwapRebMultipleHandlersbucket_op_on_update' in metakv",
  "warnings": null
 }
}
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:36:25 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 00:36:25 eventingSetup.go:125: Deploying app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op. Item count src bucket: 0 dst bucket: 0
2021/12/02 00:36:26 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingSwapRebMultipleHandlerssys_test_bucket_op' in metakv",
  "warnings": null
 }
}
2021/12/02 00:36:31 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/02 00:36:36 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get deployed
2021/12/02 00:36:41 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get deployed
2021/12/02 00:36:46 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get deployed
2021/12/02 00:36:46 clusterSetup.go:354: App: TestEventingSwapRebMultipleHandlersbucket_op_on_update got deployed
2021/12/02 00:36:51 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get deployed
2021/12/02 00:36:56 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get deployed
2021/12/02 00:37:01 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get deployed
2021/12/02 00:37:01 clusterSetup.go:354: App: TestEventingSwapRebMultipleHandlerssys_test_bucket_op got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:37:01 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:37:01 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 00:37:01 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/02 00:37:01 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/02 00:37:07 Rebalance progress: 12.5
2021/12/02 00:37:12 Rebalance progress: 12.5
2021/12/02 00:37:17 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:37:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:37:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 00:37:17 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/02 00:37:17 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/02 00:37:22 Rebalance progress: 12.5
2021/12/02 00:37:27 Rebalance progress: 12.5
2021/12/02 00:37:32 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:37:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:37:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 00:37:32 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/02 00:37:37 Rebalance progress: 12.5
2021/12/02 00:37:42 Rebalance progress: 12.5
2021/12/02 00:37:47 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:37:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:37:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 00:37:47 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 00:37:52 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get un-deployed
2021/12/02 00:37:57 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get un-deployed
2021/12/02 00:38:02 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get un-deployed
2021/12/02 00:38:07 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get un-deployed
2021/12/02 00:38:07 clusterSetup.go:405: App: TestEventingSwapRebMultipleHandlersbucket_op_on_update got un-deployed
2021/12/02 00:38:07 eventingSetup.go:46: No eventing-con process running
2021/12/02 00:38:07 eventingSetup.go:418: Function: TestEventingSwapRebMultipleHandlersbucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingSwapRebMultipleHandlersbucket_op_on_update deleting in the background"
}
2021/12/02 00:38:11 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:38:16 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 00:38:21 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1517
2021/12/02 00:38:26 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1517
2021/12/02 00:38:26 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 00:38:28 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:38:33 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 00:38:38 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7643
2021/12/02 00:38:38 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 00:38:38 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 00:38:43 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get un-deployed
2021/12/02 00:38:48 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get un-deployed
2021/12/02 00:38:53 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get un-deployed
2021/12/02 00:38:58 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get un-deployed
2021/12/02 00:38:58 clusterSetup.go:405: App: TestEventingSwapRebMultipleHandlerssys_test_bucket_op got un-deployed
2021/12/02 00:38:58 eventingSetup.go:46: No eventing-con process running
2021/12/02 00:38:58 eventingSetup.go:418: Function: TestEventingSwapRebMultipleHandlerssys_test_bucket_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingSwapRebMultipleHandlerssys_test_bucket_op deleting in the background"
}
2021/12/02 00:39:00 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:39:05 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 00:39:05 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 00:39:07 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:39:12 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 00:39:12 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingSwapRebMultipleHandlers (210.87s)
=== RUN   TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne
2021/12/02 00:39:22 clusterSetup.go:397: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/02 00:39:22 clusterSetup.go:405: App: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/02 00:39:22 eventingSetup.go:46: No eventing-con process running
2021/12/02 00:39:22 eventingSetup.go:418: Function: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne not found"
 }
}
2021/12/02 00:39:24 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:39:29 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 00:39:29 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 00:39:31 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:39:36 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 00:39:36 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:39:41 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 00:39:41 eventingSetup.go:125: Deploying app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne. Item count src bucket: 0 dst bucket: 0
2021/12/02 00:39:42 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne' in metakv",
  "warnings": null
 }
}
2021/12/02 00:39:47 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/02 00:39:52 clusterSetup.go:346: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/02 00:39:57 clusterSetup.go:346: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/02 00:40:02 clusterSetup.go:346: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/02 00:40:02 clusterSetup.go:354: App: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:40:02 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:40:02 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 00:40:02 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/02 00:40:02 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/02 00:40:22 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:40:23 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:40:23 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:40:53 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:40:53 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 00:40:53 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/02 00:40:58 Rebalance progress: 12.5
2021/12/02 00:41:03 Rebalance progress: 12.5
2021/12/02 00:41:08 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:41:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:41:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 00:41:08 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/02 00:41:08 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/02 00:41:13 Rebalance progress: 12.5
2021/12/02 00:41:18 Rebalance progress: 12.5
2021/12/02 00:41:23 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:41:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:41:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 00:41:23 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/02 00:41:29 Rebalance progress: 12.5
2021/12/02 00:41:34 Rebalance progress: 12.5
2021/12/02 00:41:39 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:41:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:41:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 00:41:39 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 00:41:44 clusterSetup.go:397: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/02 00:41:49 clusterSetup.go:397: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/02 00:41:54 clusterSetup.go:397: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/02 00:41:59 clusterSetup.go:397: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/02 00:41:59 clusterSetup.go:405: App: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/02 00:41:59 eventingSetup.go:46: No eventing-con process running
2021/12/02 00:41:59 eventingSetup.go:418: Function: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne deleting in the background"
}
2021/12/02 00:42:01 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:42:06 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 00:42:11 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6961
2021/12/02 00:42:16 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6961
2021/12/02 00:42:16 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 00:42:17 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:42:22 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 00:42:27 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 11164
2021/12/02 00:42:27 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne (195.31s)
=== RUN   TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne
2021/12/02 00:42:37 clusterSetup.go:397: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/02 00:42:37 clusterSetup.go:405: App: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/02 00:42:37 eventingSetup.go:46: No eventing-con process running
2021/12/02 00:42:37 eventingSetup.go:418: Function: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne not found"
 }
}
2021/12/02 00:42:39 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:42:44 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 00:42:44 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 00:42:46 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:42:51 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 00:42:51 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:42:56 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 00:42:56 eventingSetup.go:125: Deploying app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne. Item count src bucket: 0 dst bucket: 0
2021/12/02 00:42:57 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne' in metakv",
  "warnings": null
 }
}
2021/12/02 00:43:02 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/02 00:43:07 clusterSetup.go:346: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/02 00:43:12 clusterSetup.go:346: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/02 00:43:17 clusterSetup.go:346: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/02 00:43:17 clusterSetup.go:354: App: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:43:17 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:43:17 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 00:43:17 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/02 00:43:18 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/02 00:43:38 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:43:38 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:43:38 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:43:48 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:43:48 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 00:43:48 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/02 00:43:49 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/02 00:44:09 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:44:09 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:44:09 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:44:19 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:44:19 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 00:44:19 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/02 00:44:20 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/02 00:44:40 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:44:40 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:44:40 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:44:50 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:44:50 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 00:44:50 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/02 00:44:51 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/02 00:45:11 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:45:11 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:45:11 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:45:21 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:45:21 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 00:45:21 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/02 00:45:21 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/02 00:45:41 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:45:42 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:45:42 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:45:52 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:45:52 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 00:45:52 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/02 00:45:57 Rebalance progress: 12.5
2021/12/02 00:46:02 Rebalance progress: 12.5
2021/12/02 00:46:07 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:46:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:46:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 00:46:07 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/02 00:46:07 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/02 00:46:12 Rebalance progress: 12.5
2021/12/02 00:46:17 Rebalance progress: 12.5
2021/12/02 00:46:22 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:46:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:46:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 00:46:22 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/02 00:46:27 Rebalance progress: 12.5
2021/12/02 00:46:32 Rebalance progress: 12.5
2021/12/02 00:46:37 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:46:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:46:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 00:46:37 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 00:46:42 clusterSetup.go:397: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/02 00:46:47 clusterSetup.go:397: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/02 00:46:52 clusterSetup.go:397: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/02 00:46:57 clusterSetup.go:397: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/02 00:46:57 clusterSetup.go:405: App: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/02 00:46:58 eventingSetup.go:46: No eventing-con process running
2021/12/02 00:46:58 eventingSetup.go:418: Function: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne deleting in the background"
}
2021/12/02 00:46:59 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:47:04 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 00:47:09 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 21501
2021/12/02 00:47:09 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 00:47:11 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:47:16 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 00:47:21 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 21501
2021/12/02 00:47:26 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 21501
2021/12/02 00:47:26 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne (298.74s)
=== RUN   TestEventingFailoverOnUpdateBucketOp
2021/12/02 00:47:36 clusterSetup.go:397: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get un-deployed
2021/12/02 00:47:36 clusterSetup.go:405: App: TestEventingFailoverOnUpdateBucketOp got un-deployed
2021/12/02 00:47:36 eventingSetup.go:46: No eventing-con process running
2021/12/02 00:47:36 eventingSetup.go:418: Function: TestEventingFailoverOnUpdateBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingFailoverOnUpdateBucketOp not found"
 }
}
2021/12/02 00:47:38 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:47:43 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 00:47:43 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 00:47:45 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:47:50 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 00:47:50 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:47:55 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 00:47:55 eventingSetup.go:125: Deploying app: TestEventingFailoverOnUpdateBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/02 00:47:56 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingFailoverOnUpdateBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/02 00:48:01 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/02 00:48:06 clusterSetup.go:346: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get deployed
2021/12/02 00:48:11 clusterSetup.go:346: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get deployed
2021/12/02 00:48:16 clusterSetup.go:346: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get deployed
2021/12/02 00:48:16 clusterSetup.go:354: App: TestEventingFailoverOnUpdateBucketOp got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:48:16 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:48:16 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 00:48:16 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/02 00:48:16 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/02 00:48:21 Rebalance progress: 12.5
2021/12/02 00:48:26 Rebalance progress: 12.5
2021/12/02 00:48:31 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:48:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:48:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 00:48:31 clusterSetup.go:229: Failing over: [127.0.0.1:19001]
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:48:41 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:48:41 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 00:48:41 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/02 00:48:46 Rebalance progress: 12.5
2021/12/02 00:48:51 Rebalance progress: 12.5
2021/12/02 00:48:56 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:48:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:48:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 00:48:57 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/02 00:48:57 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/02 00:49:02 Rebalance progress: 12.5
2021/12/02 00:49:07 Rebalance progress: 12.5
2021/12/02 00:49:12 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:49:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:49:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 00:49:12 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/02 00:49:17 Rebalance progress: 12.5
2021/12/02 00:49:22 Rebalance progress: 12.5
2021/12/02 00:49:27 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:49:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:49:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 00:49:27 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 00:49:32 clusterSetup.go:397: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get un-deployed
2021/12/02 00:49:37 clusterSetup.go:397: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get un-deployed
2021/12/02 00:49:42 clusterSetup.go:397: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get un-deployed
2021/12/02 00:49:47 clusterSetup.go:397: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get un-deployed
2021/12/02 00:49:47 clusterSetup.go:405: App: TestEventingFailoverOnUpdateBucketOp got un-deployed
2021/12/02 00:49:47 eventingSetup.go:46: No eventing-con process running
2021/12/02 00:49:47 eventingSetup.go:418: Function: TestEventingFailoverOnUpdateBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingFailoverOnUpdateBucketOp deleting in the background"
}
2021/12/02 00:49:49 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:49:54 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 00:49:59 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 8599
2021/12/02 00:49:59 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 00:50:01 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:50:06 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 00:50:11 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6243
2021/12/02 00:50:16 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6243
2021/12/02 00:50:16 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingFailoverOnUpdateBucketOp (169.57s)
=== RUN   TestEventingKVRebalanceOnUpdateBucketOp
2021/12/02 00:50:26 clusterSetup.go:397: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get un-deployed
2021/12/02 00:50:26 clusterSetup.go:405: App: TestEventingKVRebalanceOnUpdateBucketOp got un-deployed
2021/12/02 00:50:26 eventingSetup.go:46: No eventing-con process running
2021/12/02 00:50:26 eventingSetup.go:418: Function: TestEventingKVRebalanceOnUpdateBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingKVRebalanceOnUpdateBucketOp not found"
 }
}
2021/12/02 00:50:28 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:50:33 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 00:50:33 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 00:50:34 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:50:39 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 00:50:39 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:50:44 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 00:50:44 eventingSetup.go:125: Deploying app: TestEventingKVRebalanceOnUpdateBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/02 00:50:45 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingKVRebalanceOnUpdateBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/02 00:50:50 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/02 00:50:55 clusterSetup.go:346: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get deployed
2021/12/02 00:51:00 clusterSetup.go:346: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get deployed
2021/12/02 00:51:05 clusterSetup.go:346: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get deployed
2021/12/02 00:51:05 clusterSetup.go:354: App: TestEventingKVRebalanceOnUpdateBucketOp got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:51:05 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:51:05 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 00:51:06 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing,kv to the cluster
2021/12/02 00:51:06 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/02 00:51:11 Rebalance progress: 12.5
2021/12/02 00:51:16 Rebalance progress: 12.5
2021/12/02 00:51:21 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:51:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:51:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 00:51:21 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing,kv to the cluster
2021/12/02 00:51:21 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/02 00:51:26 Rebalance progress: 12.5
2021/12/02 00:51:31 Rebalance progress: 12.5
2021/12/02 00:51:36 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:51:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:51:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 00:51:36 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/02 00:51:41 Rebalance progress: 12.5
2021/12/02 00:51:46 Rebalance progress: 12.5
2021/12/02 00:51:51 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:51:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:51:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 00:51:51 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 00:51:56 clusterSetup.go:397: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get un-deployed
2021/12/02 00:52:01 clusterSetup.go:397: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get un-deployed
2021/12/02 00:52:06 clusterSetup.go:397: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get un-deployed
2021/12/02 00:52:11 clusterSetup.go:397: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get un-deployed
2021/12/02 00:52:12 clusterSetup.go:405: App: TestEventingKVRebalanceOnUpdateBucketOp got un-deployed
2021/12/02 00:52:12 eventingSetup.go:46: No eventing-con process running
2021/12/02 00:52:12 eventingSetup.go:418: Function: TestEventingKVRebalanceOnUpdateBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingKVRebalanceOnUpdateBucketOp deleting in the background"
}
2021/12/02 00:52:13 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:52:18 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 00:52:23 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6071
2021/12/02 00:52:28 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6071
2021/12/02 00:52:28 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 00:52:30 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:52:35 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 00:52:40 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1373
2021/12/02 00:52:45 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1373
2021/12/02 00:52:45 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingKVRebalanceOnUpdateBucketOp (149.46s)
=== RUN   TestEventingRebInWithOps
2021/12/02 00:52:55 clusterSetup.go:397: Waiting for app: TestEventingRebInWithOps to get un-deployed
2021/12/02 00:52:55 clusterSetup.go:405: App: TestEventingRebInWithOps got un-deployed
2021/12/02 00:52:55 eventingSetup.go:46: No eventing-con process running
2021/12/02 00:52:55 eventingSetup.go:418: Function: TestEventingRebInWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebInWithOps not found"
 }
}
2021/12/02 00:52:57 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:53:02 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 00:53:02 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 00:53:04 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:53:09 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 00:53:09 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:53:14 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 00:53:14 eventingSetup.go:125: Deploying app: TestEventingRebInWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/02 00:53:15 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebInWithOps' in metakv",
  "warnings": null
 }
}
2021/12/02 00:53:20 clusterSetup.go:346: Waiting for app: TestEventingRebInWithOps to get deployed
2021/12/02 00:53:25 clusterSetup.go:346: Waiting for app: TestEventingRebInWithOps to get deployed
2021/12/02 00:53:30 clusterSetup.go:346: Waiting for app: TestEventingRebInWithOps to get deployed
2021/12/02 00:53:35 clusterSetup.go:346: Waiting for app: TestEventingRebInWithOps to get deployed
2021/12/02 00:53:35 clusterSetup.go:354: App: TestEventingRebInWithOps got deployed
2021/12/02 00:53:35 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/02 00:53:35 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/02 00:53:36 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/02 00:53:41 Rebalance progress: 12.5
2021/12/02 00:53:46 Rebalance progress: 12.5
2021/12/02 00:53:51 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:53:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:53:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 00:53:51 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/02 00:53:56 Rebalance progress: 12.5
2021/12/02 00:54:01 Rebalance progress: 12.5
2021/12/02 00:54:06 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:54:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 00:54:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 00:54:06 Waiting for src bucket item count to get to: 200000 curr count: 36701
2021/12/02 00:54:11 Waiting for src bucket item count to get to: 200000 curr count: 52414
2021/12/02 00:54:16 Waiting for src bucket item count to get to: 200000 curr count: 52414
2021/12/02 00:54:21 Waiting for src bucket item count to get to: 200000 curr count: 68378
2021/12/02 00:54:26 Waiting for src bucket item count to get to: 200000 curr count: 68378
2021/12/02 00:54:31 Waiting for src bucket item count to get to: 200000 curr count: 83026
2021/12/02 00:54:36 Waiting for src bucket item count to get to: 200000 curr count: 83026
2021/12/02 00:54:41 Waiting for src bucket item count to get to: 200000 curr count: 98407
2021/12/02 00:54:46 Waiting for src bucket item count to get to: 200000 curr count: 98407
2021/12/02 00:54:51 Waiting for src bucket item count to get to: 200000 curr count: 114163
2021/12/02 00:54:56 Waiting for src bucket item count to get to: 200000 curr count: 114163
2021/12/02 00:55:01 Waiting for src bucket item count to get to: 200000 curr count: 130282
2021/12/02 00:55:06 Waiting for src bucket item count to get to: 200000 curr count: 130282
2021/12/02 00:55:11 Waiting for src bucket item count to get to: 200000 curr count: 130282
2021/12/02 00:55:16 Waiting for src bucket item count to get to: 200000 curr count: 146423
2021/12/02 00:55:21 Waiting for src bucket item count to get to: 200000 curr count: 146423
2021/12/02 00:55:26 Waiting for src bucket item count to get to: 200000 curr count: 162520
2021/12/02 00:55:31 Waiting for src bucket item count to get to: 200000 curr count: 162520
2021/12/02 00:55:36 Waiting for src bucket item count to get to: 200000 curr count: 177445
2021/12/02 00:55:41 Waiting for src bucket item count to get to: 200000 curr count: 177445
2021/12/02 00:55:46 Waiting for src bucket item count to get to: 200000 curr count: 193359
2021/12/02 00:55:51 Waiting for src bucket item count to get to: 200000 curr count: 193359
2021/12/02 00:55:56 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/02 00:55:56 Post SET ops. Expected item count: 200000 got 200000
2021/12/02 00:55:56 Starting bucket ops to source bucket
2021/12/02 00:59:47 Waiting for dst bucket item count to get to: 0 curr count: 5290
2021/12/02 00:59:52 Waiting for dst bucket item count to get to: 0 curr count: 5290
2021/12/02 00:59:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 00:59:52 Post DELETE ops. Expected item count: 0 got 0
2021/12/02 00:59:52 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 00:59:57 Waiting for app: TestEventingRebInWithOps to get un-deployed
2021/12/02 01:00:02 Waiting for app: TestEventingRebInWithOps to get un-deployed
2021/12/02 01:00:07 Waiting for app: TestEventingRebInWithOps to get un-deployed
2021/12/02 01:00:12 Waiting for app: TestEventingRebInWithOps to get un-deployed
2021/12/02 01:00:12 App: TestEventingRebInWithOps got un-deployed
2021/12/02 01:00:12 No eventing-con process running
2021/12/02 01:00:12 Function: TestEventingRebInWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebInWithOps deleting in the background"
}
2021/12/02 01:00:14 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 01:00:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 01:00:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 01:00:21 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 01:00:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 01:00:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebInWithOps (460.90s)
=== RUN   TestEventingRebOutWithOps
2021/12/02 01:00:31 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/02 01:00:32 Removing node(s): [] from the cluster
2021/12/02 01:00:37 Rebalance progress: 100
2021/12/02 01:00:42 Waiting for app: TestEventingRebOutWithOps to get un-deployed
2021/12/02 01:00:42 App: TestEventingRebOutWithOps got un-deployed
2021/12/02 01:00:42 No eventing-con process running
2021/12/02 01:00:42 Function: TestEventingRebOutWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebOutWithOps not found"
 }
}
2021/12/02 01:00:43 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 01:00:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 01:00:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 01:00:50 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 01:00:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 01:00:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 01:01:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 01:01:00 Deploying app: TestEventingRebOutWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/02 01:01:01 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebOutWithOps' in metakv",
  "warnings": null
 }
}
2021/12/02 01:01:06 Waiting for app: TestEventingRebOutWithOps to get deployed
2021/12/02 01:01:11 Waiting for app: TestEventingRebOutWithOps to get deployed
2021/12/02 01:01:16 Waiting for app: TestEventingRebOutWithOps to get deployed
2021/12/02 01:01:21 Waiting for app: TestEventingRebOutWithOps to get deployed
2021/12/02 01:01:21 App: TestEventingRebOutWithOps got deployed
2021/12/02 01:01:21 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/02 01:01:21 Starting bucket ops to source bucket
2021/12/02 01:01:27 Rebalance progress: 12.5
2021/12/02 01:01:32 Rebalance progress: 12.5
2021/12/02 01:01:37 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 01:01:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 01:01:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 01:01:37 Waiting for src bucket item count to get to: 200000 curr count: 11149
2021/12/02 01:01:42 Waiting for src bucket item count to get to: 200000 curr count: 26815
2021/12/02 01:01:47 Waiting for src bucket item count to get to: 200000 curr count: 26815
2021/12/02 01:01:52 Waiting for src bucket item count to get to: 200000 curr count: 42747
2021/12/02 01:01:57 Waiting for src bucket item count to get to: 200000 curr count: 42747
2021/12/02 01:02:02 Waiting for src bucket item count to get to: 200000 curr count: 58994
2021/12/02 01:02:07 Waiting for src bucket item count to get to: 200000 curr count: 58994
2021/12/02 01:02:12 Waiting for src bucket item count to get to: 200000 curr count: 58994
2021/12/02 01:02:17 Waiting for src bucket item count to get to: 200000 curr count: 74005
2021/12/02 01:02:22 Waiting for src bucket item count to get to: 200000 curr count: 74005
2021/12/02 01:02:27 Waiting for src bucket item count to get to: 200000 curr count: 89970
2021/12/02 01:02:32 Waiting for src bucket item count to get to: 200000 curr count: 89970
2021/12/02 01:02:37 Waiting for src bucket item count to get to: 200000 curr count: 105818
2021/12/02 01:02:42 Waiting for src bucket item count to get to: 200000 curr count: 105818
2021/12/02 01:02:47 Waiting for src bucket item count to get to: 200000 curr count: 122089
2021/12/02 01:02:52 Waiting for src bucket item count to get to: 200000 curr count: 122089
2021/12/02 01:02:57 Waiting for src bucket item count to get to: 200000 curr count: 138175
2021/12/02 01:03:02 Waiting for src bucket item count to get to: 200000 curr count: 138175
2021/12/02 01:03:07 Waiting for src bucket item count to get to: 200000 curr count: 154254
2021/12/02 01:03:12 Waiting for src bucket item count to get to: 200000 curr count: 154254
2021/12/02 01:03:17 Waiting for src bucket item count to get to: 200000 curr count: 169359
2021/12/02 01:03:22 Waiting for src bucket item count to get to: 200000 curr count: 169359
2021/12/02 01:03:27 Waiting for src bucket item count to get to: 200000 curr count: 185076
2021/12/02 01:03:32 Waiting for src bucket item count to get to: 200000 curr count: 185076
2021/12/02 01:03:37 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/02 01:03:37 Post SET ops. Expected item count: 200000 got 200000
2021/12/02 01:03:37 Starting bucket ops to source bucket
2021/12/02 01:07:29 Waiting for dst bucket item count to get to: 0 curr count: 9047
2021/12/02 01:07:34 Waiting for dst bucket item count to get to: 0 curr count: 9047
2021/12/02 01:07:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 01:07:34 Post DELETE ops. Expected item count: 0 got 0
2021/12/02 01:07:34 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 01:07:39 Waiting for app: TestEventingRebOutWithOps to get un-deployed
2021/12/02 01:07:44 Waiting for app: TestEventingRebOutWithOps to get un-deployed
2021/12/02 01:07:49 Waiting for app: TestEventingRebOutWithOps to get un-deployed
2021/12/02 01:07:54 Waiting for app: TestEventingRebOutWithOps to get un-deployed
2021/12/02 01:07:54 App: TestEventingRebOutWithOps got un-deployed
2021/12/02 01:07:55 No eventing-con process running
2021/12/02 01:07:55 Function: TestEventingRebOutWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebOutWithOps deleting in the background"
}
2021/12/02 01:07:56 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 01:08:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 01:08:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 01:08:03 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 01:08:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 01:08:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebOutWithOps (462.04s)
=== RUN   TestEventingSwapRebWithOps
2021/12/02 01:08:18 Waiting for app: TestEventingSwapRebWithOps to get un-deployed
2021/12/02 01:08:18 App: TestEventingSwapRebWithOps got un-deployed
2021/12/02 01:08:18 No eventing-con process running
2021/12/02 01:08:18 Function: TestEventingSwapRebWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingSwapRebWithOps not found"
 }
}
2021/12/02 01:08:20 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 01:08:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 01:08:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 01:08:27 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 01:08:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 01:08:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 01:08:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 01:08:37 Deploying app: TestEventingSwapRebWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/02 01:08:38 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingSwapRebWithOps' in metakv",
  "warnings": null
 }
}
2021/12/02 01:08:43 Waiting for app: TestEventingSwapRebWithOps to get deployed
2021/12/02 01:08:48 Waiting for app: TestEventingSwapRebWithOps to get deployed
2021/12/02 01:08:53 Waiting for app: TestEventingSwapRebWithOps to get deployed
2021/12/02 01:08:58 Waiting for app: TestEventingSwapRebWithOps to get deployed
2021/12/02 01:08:58 App: TestEventingSwapRebWithOps got deployed
2021/12/02 01:08:58 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/02 01:08:58 Starting bucket ops to source bucket
2021/12/02 01:08:59 Removing node(s): [] from the cluster
2021/12/02 01:09:04 Rebalance progress: 12.5
2021/12/02 01:09:09 Rebalance progress: 12.5
2021/12/02 01:09:14 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 01:09:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 01:09:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 01:09:14 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/02 01:09:14 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/02 01:09:19 Rebalance progress: 12.5
2021/12/02 01:09:24 Rebalance progress: 12.5
2021/12/02 01:09:29 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 01:09:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 01:09:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 01:09:29 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/02 01:09:34 Rebalance progress: 12.5
2021/12/02 01:09:39 Rebalance progress: 12.5
2021/12/02 01:09:44 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 01:09:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 01:09:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 01:09:44 Waiting for src bucket item count to get to: 200000 curr count: 62152
2021/12/02 01:09:49 Waiting for src bucket item count to get to: 200000 curr count: 62152
2021/12/02 01:09:54 Waiting for src bucket item count to get to: 200000 curr count: 76732
2021/12/02 01:09:59 Waiting for src bucket item count to get to: 200000 curr count: 76732
2021/12/02 01:10:04 Waiting for src bucket item count to get to: 200000 curr count: 92588
2021/12/02 01:10:09 Waiting for src bucket item count to get to: 200000 curr count: 92588
2021/12/02 01:10:14 Waiting for src bucket item count to get to: 200000 curr count: 108664
2021/12/02 01:10:19 Waiting for src bucket item count to get to: 200000 curr count: 108664
2021/12/02 01:10:24 Waiting for src bucket item count to get to: 200000 curr count: 124870
2021/12/02 01:10:29 Waiting for src bucket item count to get to: 200000 curr count: 124870
2021/12/02 01:10:34 Waiting for src bucket item count to get to: 200000 curr count: 140877
2021/12/02 01:10:39 Waiting for src bucket item count to get to: 200000 curr count: 140877
2021/12/02 01:10:44 Waiting for src bucket item count to get to: 200000 curr count: 157079
2021/12/02 01:10:49 Waiting for src bucket item count to get to: 200000 curr count: 157079
2021/12/02 01:10:54 Waiting for src bucket item count to get to: 200000 curr count: 172177
2021/12/02 01:10:59 Waiting for src bucket item count to get to: 200000 curr count: 172177
2021/12/02 01:11:04 Waiting for src bucket item count to get to: 200000 curr count: 188027
2021/12/02 01:11:09 Waiting for src bucket item count to get to: 200000 curr count: 188027
2021/12/02 01:11:14 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/02 01:11:14 Post SET ops. Expected item count: 200000 got 200000
2021/12/02 01:11:14 Starting bucket ops to source bucket
2021/12/02 01:15:06 Waiting for dst bucket item count to get to: 0 curr count: 20293
2021/12/02 01:15:11 Waiting for dst bucket item count to get to: 0 curr count: 2212
2021/12/02 01:15:16 Waiting for dst bucket item count to get to: 0 curr count: 2212
2021/12/02 01:15:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 01:15:16 Post DELETE ops. Expected item count: 0 got 0
2021/12/02 01:15:16 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 01:15:21 Waiting for app: TestEventingSwapRebWithOps to get un-deployed
2021/12/02 01:15:26 Waiting for app: TestEventingSwapRebWithOps to get un-deployed
2021/12/02 01:15:31 Waiting for app: TestEventingSwapRebWithOps to get un-deployed
2021/12/02 01:15:36 Waiting for app: TestEventingSwapRebWithOps to get un-deployed
2021/12/02 01:15:36 App: TestEventingSwapRebWithOps got un-deployed
2021/12/02 01:15:36 No eventing-con process running
2021/12/02 01:15:36 Function: TestEventingSwapRebWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingSwapRebWithOps deleting in the background"
}
2021/12/02 01:15:38 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 01:15:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 01:15:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 01:15:44 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 01:15:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 01:15:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingSwapRebWithOps (461.21s)
=== RUN   TestKVRebInWithOps
2021/12/02 01:15:59 Waiting for app: TestKVRebInWithOps to get un-deployed
2021/12/02 01:15:59 App: TestKVRebInWithOps got un-deployed
2021/12/02 01:15:59 No eventing-con process running
2021/12/02 01:15:59 Function: TestKVRebInWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVRebInWithOps not found"
 }
}
2021/12/02 01:16:01 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 01:16:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 01:16:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 01:16:08 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 01:16:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 01:16:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 01:16:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 01:16:18 Deploying app: TestKVRebInWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/02 01:16:19 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVRebInWithOps' in metakv",
  "warnings": null
 }
}
2021/12/02 01:16:24 Waiting for app: TestKVRebInWithOps to get deployed
2021/12/02 01:16:29 Waiting for app: TestKVRebInWithOps to get deployed
2021/12/02 01:16:34 Waiting for app: TestKVRebInWithOps to get deployed
2021/12/02 01:16:39 Waiting for app: TestKVRebInWithOps to get deployed
2021/12/02 01:16:39 App: TestKVRebInWithOps got deployed
2021/12/02 01:16:39 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/02 01:16:39 Starting bucket ops to source bucket
2021/12/02 01:16:40 Removing node(s): [] from the cluster
2021/12/02 01:16:45 Rebalance progress: 12.5
2021/12/02 01:16:50 Rebalance progress: 12.5
2021/12/02 01:16:55 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 01:16:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 01:16:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 01:16:55 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/02 01:17:00 Rebalance progress: 12.5
2021/12/02 01:17:05 Rebalance progress: 12.5
2021/12/02 01:17:10 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 01:17:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 01:17:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 01:17:10 Waiting for src bucket item count to get to: 200000 curr count: 30110
2021/12/02 01:17:15 Waiting for src bucket item count to get to: 200000 curr count: 46209
2021/12/02 01:17:20 Waiting for src bucket item count to get to: 200000 curr count: 46209
2021/12/02 01:17:25 Waiting for src bucket item count to get to: 200000 curr count: 61701
2021/12/02 01:17:30 Waiting for src bucket item count to get to: 200000 curr count: 61701
2021/12/02 01:17:35 Waiting for src bucket item count to get to: 200000 curr count: 76559
2021/12/02 01:17:40 Waiting for src bucket item count to get to: 200000 curr count: 76559
2021/12/02 01:17:45 Waiting for src bucket item count to get to: 200000 curr count: 92084
2021/12/02 01:17:50 Waiting for src bucket item count to get to: 200000 curr count: 108268
2021/12/02 01:17:55 Waiting for src bucket item count to get to: 200000 curr count: 108268
2021/12/02 01:18:00 Waiting for src bucket item count to get to: 200000 curr count: 124281
2021/12/02 01:18:05 Waiting for src bucket item count to get to: 200000 curr count: 124281
2021/12/02 01:18:10 Waiting for src bucket item count to get to: 200000 curr count: 124281
2021/12/02 01:18:15 Waiting for src bucket item count to get to: 200000 curr count: 140407
2021/12/02 01:18:20 Waiting for src bucket item count to get to: 200000 curr count: 140407
2021/12/02 01:18:25 Waiting for src bucket item count to get to: 200000 curr count: 156603
2021/12/02 01:18:30 Waiting for src bucket item count to get to: 200000 curr count: 156603
2021/12/02 01:18:35 Waiting for src bucket item count to get to: 200000 curr count: 171728
2021/12/02 01:18:40 Waiting for src bucket item count to get to: 200000 curr count: 171728
2021/12/02 01:18:45 Waiting for src bucket item count to get to: 200000 curr count: 187617
2021/12/02 01:18:50 Waiting for src bucket item count to get to: 200000 curr count: 187617
2021/12/02 01:18:55 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/02 01:18:55 Post SET ops. Expected item count: 200000 got 200000
2021/12/02 01:18:55 Starting bucket ops to source bucket
2021/12/02 01:22:45 Waiting for dst bucket item count to get to: 0 curr count: 19803
2021/12/02 01:22:50 Waiting for dst bucket item count to get to: 0 curr count: 1637
2021/12/02 01:22:55 Waiting for dst bucket item count to get to: 0 curr count: 1637
2021/12/02 01:22:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 01:22:55 Post DELETE ops. Expected item count: 0 got 0
2021/12/02 01:22:55 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 01:23:00 Waiting for app: TestKVRebInWithOps to get un-deployed
2021/12/02 01:23:05 Waiting for app: TestKVRebInWithOps to get un-deployed
2021/12/02 01:23:10 Waiting for app: TestKVRebInWithOps to get un-deployed
2021/12/02 01:23:15 Waiting for app: TestKVRebInWithOps to get un-deployed
2021/12/02 01:23:15 App: TestKVRebInWithOps got un-deployed
2021/12/02 01:23:15 No eventing-con process running
2021/12/02 01:23:15 Function: TestKVRebInWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVRebInWithOps deleting in the background"
}
2021/12/02 01:23:17 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 01:23:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 01:23:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 01:23:24 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 01:23:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 01:23:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebInWithOps (459.69s)
=== RUN   TestKVRebOutWithOps
2021/12/02 01:23:34 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/02 01:23:34 Removing node(s): [] from the cluster
2021/12/02 01:23:39 Rebalance progress: 100
2021/12/02 01:23:44 Waiting for app: TestKVRebOutWithOps to get un-deployed
2021/12/02 01:23:44 App: TestKVRebOutWithOps got un-deployed
2021/12/02 01:23:44 No eventing-con process running
2021/12/02 01:23:44 Function: TestKVRebOutWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVRebOutWithOps not found"
 }
}
2021/12/02 01:23:46 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 01:23:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 01:23:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 01:23:53 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 01:23:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 01:23:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 01:24:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 01:24:03 Deploying app: TestKVRebOutWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/02 01:24:04 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVRebOutWithOps' in metakv",
  "warnings": null
 }
}
2021/12/02 01:24:09 Waiting for app: TestKVRebOutWithOps to get deployed
2021/12/02 01:24:14 Waiting for app: TestKVRebOutWithOps to get deployed
2021/12/02 01:24:19 Waiting for app: TestKVRebOutWithOps to get deployed
2021/12/02 01:24:24 Waiting for app: TestKVRebOutWithOps to get deployed
2021/12/02 01:24:24 App: TestKVRebOutWithOps got deployed
2021/12/02 01:24:24 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/02 01:24:24 Starting bucket ops to source bucket
2021/12/02 01:24:29 Rebalance progress: 12.5
2021/12/02 01:24:34 Rebalance progress: 12.5
2021/12/02 01:24:39 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 01:24:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 01:24:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 01:24:39 Waiting for src bucket item count to get to: 200000 curr count: 7303
2021/12/02 01:24:44 Waiting for src bucket item count to get to: 200000 curr count: 22967
2021/12/02 01:24:49 Waiting for src bucket item count to get to: 200000 curr count: 22967
2021/12/02 01:24:54 Waiting for src bucket item count to get to: 200000 curr count: 39033
2021/12/02 01:24:59 Waiting for src bucket item count to get to: 200000 curr count: 39033
2021/12/02 01:25:04 Waiting for src bucket item count to get to: 200000 curr count: 55206
2021/12/02 01:25:09 Waiting for src bucket item count to get to: 200000 curr count: 55206
2021/12/02 01:25:14 Waiting for src bucket item count to get to: 200000 curr count: 55206
2021/12/02 01:25:19 Waiting for src bucket item count to get to: 200000 curr count: 71221
2021/12/02 01:25:24 Waiting for src bucket item count to get to: 200000 curr count: 71221
2021/12/02 01:25:29 Waiting for src bucket item count to get to: 200000 curr count: 86320
2021/12/02 01:25:34 Waiting for src bucket item count to get to: 200000 curr count: 86320
2021/12/02 01:25:39 Waiting for src bucket item count to get to: 200000 curr count: 101907
2021/12/02 01:25:44 Waiting for src bucket item count to get to: 200000 curr count: 117759
2021/12/02 01:25:49 Waiting for src bucket item count to get to: 200000 curr count: 117759
2021/12/02 01:25:54 Waiting for src bucket item count to get to: 200000 curr count: 133627
2021/12/02 01:25:59 Waiting for src bucket item count to get to: 200000 curr count: 133627
2021/12/02 01:26:04 Waiting for src bucket item count to get to: 200000 curr count: 149605
2021/12/02 01:26:09 Waiting for src bucket item count to get to: 200000 curr count: 149605
2021/12/02 01:26:14 Waiting for src bucket item count to get to: 200000 curr count: 165607
2021/12/02 01:26:19 Waiting for src bucket item count to get to: 200000 curr count: 165607
2021/12/02 01:26:24 Waiting for src bucket item count to get to: 200000 curr count: 180578
2021/12/02 01:26:29 Waiting for src bucket item count to get to: 200000 curr count: 180578
2021/12/02 01:26:34 Waiting for src bucket item count to get to: 200000 curr count: 196333
2021/12/02 01:26:39 Waiting for src bucket item count to get to: 200000 curr count: 196333
2021/12/02 01:26:44 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/02 01:26:44 Post SET ops. Expected item count: 200000 got 200000
2021/12/02 01:26:44 Starting bucket ops to source bucket
2021/12/02 01:30:36 Waiting for dst bucket item count to get to: 0 curr count: 20858
2021/12/02 01:30:41 Waiting for dst bucket item count to get to: 0 curr count: 2776
2021/12/02 01:30:46 Waiting for dst bucket item count to get to: 0 curr count: 2776
2021/12/02 01:30:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 01:30:46 Post DELETE ops. Expected item count: 0 got 0
2021/12/02 01:30:46 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 01:30:51 Waiting for app: TestKVRebOutWithOps to get un-deployed
2021/12/02 01:30:56 Waiting for app: TestKVRebOutWithOps to get un-deployed
2021/12/02 01:31:01 Waiting for app: TestKVRebOutWithOps to get un-deployed
2021/12/02 01:31:06 Waiting for app: TestKVRebOutWithOps to get un-deployed
2021/12/02 01:31:06 App: TestKVRebOutWithOps got un-deployed
2021/12/02 01:31:06 No eventing-con process running
2021/12/02 01:31:06 Function: TestKVRebOutWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVRebOutWithOps deleting in the background"
}
2021/12/02 01:31:08 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 01:31:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 01:31:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 01:31:15 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 01:31:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 01:31:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebOutWithOps (470.61s)
=== RUN   TestKVSwapRebWithOps
2021/12/02 01:31:30 Waiting for app: TestKVSwapRebWithOps to get un-deployed
2021/12/02 01:31:30 App: TestKVSwapRebWithOps got un-deployed
2021/12/02 01:31:30 No eventing-con process running
2021/12/02 01:31:30 Function: TestKVSwapRebWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVSwapRebWithOps not found"
 }
}
2021/12/02 01:31:31 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 01:31:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 01:31:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 01:31:38 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 01:31:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 01:31:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 01:31:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 01:31:48 Deploying app: TestKVSwapRebWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/02 01:31:49 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVSwapRebWithOps' in metakv",
  "warnings": null
 }
}
2021/12/02 01:31:54 Waiting for app: TestKVSwapRebWithOps to get deployed
2021/12/02 01:31:59 Waiting for app: TestKVSwapRebWithOps to get deployed
2021/12/02 01:32:04 Waiting for app: TestKVSwapRebWithOps to get deployed
2021/12/02 01:32:09 Waiting for app: TestKVSwapRebWithOps to get deployed
2021/12/02 01:32:09 App: TestKVSwapRebWithOps got deployed
2021/12/02 01:32:09 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/02 01:32:09 Starting bucket ops to source bucket
2021/12/02 01:32:10 Removing node(s): [] from the cluster
2021/12/02 01:32:15 Rebalance progress: 12.5
2021/12/02 01:32:20 Rebalance progress: 12.5
2021/12/02 01:32:25 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 01:32:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 01:32:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 01:32:25 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/02 01:32:25 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/02 01:32:30 Rebalance progress: 12.5
2021/12/02 01:32:35 Rebalance progress: 12.5
2021/12/02 01:32:40 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 01:32:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 01:32:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 01:32:40 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/02 01:32:45 Rebalance progress: 12.5
2021/12/02 01:32:50 Rebalance progress: 12.5
2021/12/02 01:32:55 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 01:32:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 01:32:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 01:32:56 Waiting for src bucket item count to get to: 200000 curr count: 59727
2021/12/02 01:33:01 Waiting for src bucket item count to get to: 200000 curr count: 74256
2021/12/02 01:33:06 Waiting for src bucket item count to get to: 200000 curr count: 74256
2021/12/02 01:33:11 Waiting for src bucket item count to get to: 200000 curr count: 89829
2021/12/02 01:33:16 Waiting for src bucket item count to get to: 200000 curr count: 89829
2021/12/02 01:33:21 Waiting for src bucket item count to get to: 200000 curr count: 105685
2021/12/02 01:33:26 Waiting for src bucket item count to get to: 200000 curr count: 105685
2021/12/02 01:33:31 Waiting for src bucket item count to get to: 200000 curr count: 121445
2021/12/02 01:33:36 Waiting for src bucket item count to get to: 200000 curr count: 121445
2021/12/02 01:33:41 Waiting for src bucket item count to get to: 200000 curr count: 137338
2021/12/02 01:33:46 Waiting for src bucket item count to get to: 200000 curr count: 137338
2021/12/02 01:33:51 Waiting for src bucket item count to get to: 200000 curr count: 152977
2021/12/02 01:33:56 Waiting for src bucket item count to get to: 200000 curr count: 152977
2021/12/02 01:34:01 Waiting for src bucket item count to get to: 200000 curr count: 152977
2021/12/02 01:34:06 Waiting for src bucket item count to get to: 200000 curr count: 167813
2021/12/02 01:34:11 Waiting for src bucket item count to get to: 200000 curr count: 167813
2021/12/02 01:34:16 Waiting for src bucket item count to get to: 200000 curr count: 183549
2021/12/02 01:34:21 Waiting for src bucket item count to get to: 200000 curr count: 183549
2021/12/02 01:34:26 Waiting for src bucket item count to get to: 200000 curr count: 199689
2021/12/02 01:34:31 Waiting for src bucket item count to get to: 200000 curr count: 199689
2021/12/02 01:34:36 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/02 01:34:36 Post SET ops. Expected item count: 200000 got 200000
2021/12/02 01:34:36 Starting bucket ops to source bucket
2021/12/02 01:38:26 Waiting for dst bucket item count to get to: 0 curr count: 3113
2021/12/02 01:38:31 Waiting for dst bucket item count to get to: 0 curr count: 3113
2021/12/02 01:38:31 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 01:38:31 Post DELETE ops. Expected item count: 0 got 0
2021/12/02 01:38:31 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 01:38:36 Waiting for app: TestKVSwapRebWithOps to get un-deployed
2021/12/02 01:38:41 Waiting for app: TestKVSwapRebWithOps to get un-deployed
2021/12/02 01:38:46 Waiting for app: TestKVSwapRebWithOps to get un-deployed
2021/12/02 01:38:51 Waiting for app: TestKVSwapRebWithOps to get un-deployed
2021/12/02 01:38:51 App: TestKVSwapRebWithOps got un-deployed
2021/12/02 01:38:51 No eventing-con process running
2021/12/02 01:38:51 Function: TestKVSwapRebWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVSwapRebWithOps deleting in the background"
}
2021/12/02 01:38:53 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 01:38:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 01:38:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 01:39:00 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 01:39:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 01:39:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVSwapRebWithOps (465.17s)
=== RUN   TestEventingRebInWithTimers
2021/12/02 01:39:15 Waiting for app: TestEventingRebInWithTimers to get un-deployed
2021/12/02 01:39:15 App: TestEventingRebInWithTimers got un-deployed
2021/12/02 01:39:15 No eventing-con process running
2021/12/02 01:39:15 Function: TestEventingRebInWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebInWithTimers not found"
 }
}
2021/12/02 01:39:17 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 01:39:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 01:39:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 01:39:23 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 01:39:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 01:39:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 01:39:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 01:39:33 Deploying app: TestEventingRebInWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/02 01:39:34 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebInWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/02 01:39:39 Waiting for app: TestEventingRebInWithTimers to get deployed
2021/12/02 01:39:45 Waiting for app: TestEventingRebInWithTimers to get deployed
2021/12/02 01:39:50 Waiting for app: TestEventingRebInWithTimers to get deployed
2021/12/02 01:39:55 Waiting for app: TestEventingRebInWithTimers to get deployed
2021/12/02 01:39:55 App: TestEventingRebInWithTimers got deployed
2021/12/02 01:39:55 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/02 01:39:55 Starting bucket ops to source bucket
2021/12/02 01:39:55 Removing node(s): [] from the cluster
2021/12/02 01:40:00 Rebalance progress: 12.5
2021/12/02 01:40:05 Rebalance progress: 12.5
2021/12/02 01:40:10 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 01:40:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 01:40:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 01:40:10 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/02 01:40:15 Rebalance progress: 12.5
2021/12/02 01:40:20 Rebalance progress: 12.5
2021/12/02 01:40:25 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 01:40:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 01:40:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 01:40:25 Waiting for src bucket item count to get to: 50000 curr count: 30575
2021/12/02 01:40:30 Waiting for src bucket item count to get to: 50000 curr count: 30575
2021/12/02 01:40:35 Waiting for src bucket item count to get to: 50000 curr count: 41839
2021/12/02 01:40:45 Waiting for dst bucket item count to get to: 50000 curr count: 34352
2021/12/02 01:40:50 Waiting for dst bucket item count to get to: 50000 curr count: 34352
2021/12/02 01:40:55 Waiting for dst bucket item count to get to: 50000 curr count: 43533
2021/12/02 01:41:00 Waiting for dst bucket item count to get to: 50000 curr count: 43533
2021/12/02 01:41:00 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/02 01:41:00 Post SET ops. Expected item count: 50000 got 50000
2021/12/02 01:41:00 Starting bucket ops to source bucket
2021/12/02 01:42:11 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/02 01:42:16 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/02 01:42:21 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/02 01:42:26 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/02 01:42:31 Waiting for dst bucket item count to get to: 0 curr count: 41346
2021/12/02 01:42:36 Waiting for dst bucket item count to get to: 0 curr count: 41346
2021/12/02 01:42:41 Waiting for dst bucket item count to get to: 0 curr count: 25977
2021/12/02 01:42:46 Waiting for dst bucket item count to get to: 0 curr count: 25977
2021/12/02 01:42:51 Waiting for dst bucket item count to get to: 0 curr count: 9267
2021/12/02 01:42:56 Waiting for dst bucket item count to get to: 0 curr count: 9267
2021/12/02 01:42:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 01:42:56 Post DELETE ops. Expected item count: 0 got 0
2021/12/02 01:42:56 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 01:43:01 Waiting for app: TestEventingRebInWithTimers to get un-deployed
2021/12/02 01:43:06 Waiting for app: TestEventingRebInWithTimers to get un-deployed
2021/12/02 01:43:11 Waiting for app: TestEventingRebInWithTimers to get un-deployed
2021/12/02 01:43:16 Waiting for app: TestEventingRebInWithTimers to get un-deployed
2021/12/02 01:43:16 App: TestEventingRebInWithTimers got un-deployed
2021/12/02 01:43:16 No eventing-con process running
2021/12/02 01:43:16 Function: TestEventingRebInWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebInWithTimers deleting in the background"
}
2021/12/02 01:43:18 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 01:43:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 01:43:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 01:43:25 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 01:43:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 01:43:30 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebInWithTimers (265.20s)
=== RUN   TestEventingRebOutWithTimers
2021/12/02 01:43:35 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/02 01:43:36 Removing node(s): [] from the cluster
2021/12/02 01:43:41 Rebalance progress: 100
2021/12/02 01:43:46 Waiting for app: TestEventingRebOutWithTimers to get un-deployed
2021/12/02 01:43:46 App: TestEventingRebOutWithTimers got un-deployed
2021/12/02 01:43:46 No eventing-con process running
2021/12/02 01:43:46 Function: TestEventingRebOutWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebOutWithTimers not found"
 }
}
2021/12/02 01:43:48 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 01:43:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 01:43:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 01:43:54 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 01:43:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 01:43:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 01:44:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 01:44:04 Deploying app: TestEventingRebOutWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/02 01:44:05 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebOutWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/02 01:44:10 Waiting for app: TestEventingRebOutWithTimers to get deployed
2021/12/02 01:44:15 Waiting for app: TestEventingRebOutWithTimers to get deployed
2021/12/02 01:44:20 Waiting for app: TestEventingRebOutWithTimers to get deployed
2021/12/02 01:44:25 Waiting for app: TestEventingRebOutWithTimers to get deployed
2021/12/02 01:44:25 App: TestEventingRebOutWithTimers got deployed
2021/12/02 01:44:25 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/02 01:44:25 Starting bucket ops to source bucket
2021/12/02 01:44:31 Rebalance progress: 12.5
2021/12/02 01:44:36 Rebalance progress: 12.5
2021/12/02 01:44:41 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 01:44:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 01:44:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 01:44:41 Waiting for src bucket item count to get to: 50000 curr count: 18713
2021/12/02 01:44:46 Waiting for src bucket item count to get to: 50000 curr count: 18713
2021/12/02 01:44:51 Waiting for src bucket item count to get to: 50000 curr count: 29476
2021/12/02 01:44:56 Waiting for src bucket item count to get to: 50000 curr count: 29476
2021/12/02 01:45:01 Waiting for src bucket item count to get to: 50000 curr count: 41193
2021/12/02 01:45:06 Waiting for src bucket item count to get to: 50000 curr count: 41193
2021/12/02 01:45:11 Waiting for src bucket item count to get to: 50000 curr count: 41193
2021/12/02 01:45:21 Waiting for dst bucket item count to get to: 50000 curr count: 38532
2021/12/02 01:45:26 Waiting for dst bucket item count to get to: 50000 curr count: 38532
2021/12/02 01:45:31 Waiting for dst bucket item count to get to: 50000 curr count: 46600
2021/12/02 01:45:36 Waiting for dst bucket item count to get to: 50000 curr count: 46600
2021/12/02 01:45:36 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/02 01:45:36 Post SET ops. Expected item count: 50000 got 50000
2021/12/02 01:45:36 Starting bucket ops to source bucket
2021/12/02 01:46:46 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/02 01:46:51 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/02 01:46:56 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/02 01:47:01 Waiting for dst bucket item count to get to: 0 curr count: 48152
2021/12/02 01:47:06 Waiting for dst bucket item count to get to: 0 curr count: 48152
2021/12/02 01:47:11 Waiting for dst bucket item count to get to: 0 curr count: 36384
2021/12/02 01:47:16 Waiting for dst bucket item count to get to: 0 curr count: 36384
2021/12/02 01:47:21 Waiting for dst bucket item count to get to: 0 curr count: 12762
2021/12/02 01:47:26 Waiting for dst bucket item count to get to: 0 curr count: 12762
2021/12/02 01:47:31 Waiting for dst bucket item count to get to: 0 curr count: 865
2021/12/02 01:47:36 Waiting for dst bucket item count to get to: 0 curr count: 865
2021/12/02 01:47:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 01:47:36 Post DELETE ops. Expected item count: 0 got 0
2021/12/02 01:47:36 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 01:47:41 Waiting for app: TestEventingRebOutWithTimers to get un-deployed
2021/12/02 01:47:46 Waiting for app: TestEventingRebOutWithTimers to get un-deployed
2021/12/02 01:47:51 Waiting for app: TestEventingRebOutWithTimers to get un-deployed
2021/12/02 01:47:56 Waiting for app: TestEventingRebOutWithTimers to get un-deployed
2021/12/02 01:47:56 App: TestEventingRebOutWithTimers got un-deployed
2021/12/02 01:47:56 No eventing-con process running
2021/12/02 01:47:56 Function: TestEventingRebOutWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebOutWithTimers deleting in the background"
}
2021/12/02 01:47:58 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 01:48:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 01:48:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 01:48:05 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 01:48:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 01:48:10 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebOutWithTimers (279.93s)
=== RUN   TestEventingSwapRebWithTimers
2021/12/02 01:48:20 Waiting for app: TestEventingSwapRebWithTimers to get un-deployed
2021/12/02 01:48:20 App: TestEventingSwapRebWithTimers got un-deployed
2021/12/02 01:48:20 No eventing-con process running
2021/12/02 01:48:20 Function: TestEventingSwapRebWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingSwapRebWithTimers not found"
 }
}
2021/12/02 01:48:22 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 01:48:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 01:48:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 01:48:28 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 01:48:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 01:48:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 01:48:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 01:48:38 Deploying app: TestEventingSwapRebWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/02 01:48:40 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingSwapRebWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/02 01:48:45 Waiting for app: TestEventingSwapRebWithTimers to get deployed
2021/12/02 01:48:50 Waiting for app: TestEventingSwapRebWithTimers to get deployed
2021/12/02 01:48:55 Waiting for app: TestEventingSwapRebWithTimers to get deployed
2021/12/02 01:49:00 Waiting for app: TestEventingSwapRebWithTimers to get deployed
2021/12/02 01:49:00 App: TestEventingSwapRebWithTimers got deployed
2021/12/02 01:49:00 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/02 01:49:00 Starting bucket ops to source bucket
2021/12/02 01:49:00 Removing node(s): [] from the cluster
2021/12/02 01:49:05 Rebalance progress: 12.5
2021/12/02 01:49:10 Rebalance progress: 12.5
2021/12/02 01:49:15 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 01:49:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 01:49:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 01:49:15 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/02 01:49:15 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/02 01:49:20 Rebalance progress: 12.5
2021/12/02 01:49:25 Rebalance progress: 12.5
2021/12/02 01:49:30 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 01:49:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 01:49:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 01:49:30 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/02 01:49:35 Rebalance progress: 12.5
2021/12/02 01:49:40 Rebalance progress: 12.5
2021/12/02 01:49:45 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 01:49:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 01:49:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 01:49:45 Waiting for src bucket item count to get to: 50000 curr count: 44297
2021/12/02 01:49:55 Waiting for dst bucket item count to get to: 50000 curr count: 39444
2021/12/02 01:50:01 Waiting for dst bucket item count to get to: 50000 curr count: 39444
2021/12/02 01:50:06 Waiting for dst bucket item count to get to: 50000 curr count: 49321
2021/12/02 01:50:11 Waiting for dst bucket item count to get to: 50000 curr count: 49321
2021/12/02 01:50:11 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/02 01:50:11 Post SET ops. Expected item count: 50000 got 50000
2021/12/02 01:50:11 Starting bucket ops to source bucket
2021/12/02 01:51:21 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/02 01:51:26 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/02 01:51:31 Waiting for dst bucket item count to get to: 0 curr count: 48937
2021/12/02 01:51:36 Waiting for dst bucket item count to get to: 0 curr count: 48937
2021/12/02 01:51:41 Waiting for dst bucket item count to get to: 0 curr count: 38690
2021/12/02 01:51:46 Waiting for dst bucket item count to get to: 0 curr count: 38690
2021/12/02 01:51:51 Waiting for dst bucket item count to get to: 0 curr count: 28697
2021/12/02 01:51:56 Waiting for dst bucket item count to get to: 0 curr count: 28697
2021/12/02 01:52:01 Waiting for dst bucket item count to get to: 0 curr count: 6641
2021/12/02 01:52:06 Waiting for dst bucket item count to get to: 0 curr count: 6641
2021/12/02 01:52:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 01:52:06 Post DELETE ops. Expected item count: 0 got 0
2021/12/02 01:52:06 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 01:52:11 Waiting for app: TestEventingSwapRebWithTimers to get un-deployed
2021/12/02 01:52:16 Waiting for app: TestEventingSwapRebWithTimers to get un-deployed
2021/12/02 01:52:21 Waiting for app: TestEventingSwapRebWithTimers to get un-deployed
2021/12/02 01:52:26 Waiting for app: TestEventingSwapRebWithTimers to get un-deployed
2021/12/02 01:52:26 App: TestEventingSwapRebWithTimers got un-deployed
2021/12/02 01:52:26 No eventing-con process running
2021/12/02 01:52:26 Function: TestEventingSwapRebWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingSwapRebWithTimers deleting in the background"
}
2021/12/02 01:52:28 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 01:52:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 01:52:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 01:52:35 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 01:52:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 01:52:40 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingSwapRebWithTimers (269.92s)
=== RUN   TestKVRebInWithTimers
2021/12/02 01:52:50 Waiting for app: TestKVRebInWithTimers to get un-deployed
2021/12/02 01:52:50 App: TestKVRebInWithTimers got un-deployed
2021/12/02 01:52:50 No eventing-con process running
2021/12/02 01:52:50 Function: TestKVRebInWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVRebInWithTimers not found"
 }
}
2021/12/02 01:52:52 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 01:52:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 01:52:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 01:52:58 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 01:53:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 01:53:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 01:53:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 01:53:08 Deploying app: TestKVRebInWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/02 01:53:10 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVRebInWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/02 01:53:15 Waiting for app: TestKVRebInWithTimers to get deployed
2021/12/02 01:53:20 Waiting for app: TestKVRebInWithTimers to get deployed
2021/12/02 01:53:25 Waiting for app: TestKVRebInWithTimers to get deployed
2021/12/02 01:53:30 Waiting for app: TestKVRebInWithTimers to get deployed
2021/12/02 01:53:30 App: TestKVRebInWithTimers got deployed
2021/12/02 01:53:30 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/02 01:53:30 Starting bucket ops to source bucket
2021/12/02 01:53:30 Removing node(s): [] from the cluster
2021/12/02 01:53:35 Rebalance progress: 12.5
2021/12/02 01:53:40 Rebalance progress: 12.5
2021/12/02 01:53:45 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 01:53:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 01:53:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 01:53:45 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/02 01:53:50 Rebalance progress: 12.5
2021/12/02 01:53:55 Rebalance progress: 12.5
2021/12/02 01:54:00 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 01:54:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 01:54:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 01:54:00 Waiting for src bucket item count to get to: 50000 curr count: 23956
2021/12/02 01:54:05 Waiting for src bucket item count to get to: 50000 curr count: 33398
2021/12/02 01:54:10 Waiting for src bucket item count to get to: 50000 curr count: 45147
2021/12/02 01:54:15 Waiting for src bucket item count to get to: 50000 curr count: 45147
2021/12/02 01:54:25 Waiting for dst bucket item count to get to: 50000 curr count: 44369
2021/12/02 01:54:30 Waiting for dst bucket item count to get to: 50000 curr count: 44369
2021/12/02 01:54:30 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/02 01:54:30 Post SET ops. Expected item count: 50000 got 50000
2021/12/02 01:54:30 Starting bucket ops to source bucket
2021/12/02 01:55:41 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/02 01:55:46 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/02 01:55:51 Waiting for dst bucket item count to get to: 0 curr count: 49900
2021/12/02 01:55:56 Waiting for dst bucket item count to get to: 0 curr count: 49900
2021/12/02 01:56:01 Waiting for dst bucket item count to get to: 0 curr count: 39275
2021/12/02 01:56:06 Waiting for dst bucket item count to get to: 0 curr count: 39275
2021/12/02 01:56:11 Waiting for dst bucket item count to get to: 0 curr count: 17630
2021/12/02 01:56:16 Waiting for dst bucket item count to get to: 0 curr count: 17630
2021/12/02 01:56:21 Waiting for dst bucket item count to get to: 0 curr count: 5989
2021/12/02 01:56:26 Waiting for dst bucket item count to get to: 0 curr count: 5989
2021/12/02 01:56:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 01:56:26 Post DELETE ops. Expected item count: 0 got 0
2021/12/02 01:56:26 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 01:56:31 Waiting for app: TestKVRebInWithTimers to get un-deployed
2021/12/02 01:56:36 Waiting for app: TestKVRebInWithTimers to get un-deployed
2021/12/02 01:56:41 Waiting for app: TestKVRebInWithTimers to get un-deployed
2021/12/02 01:56:46 Waiting for app: TestKVRebInWithTimers to get un-deployed
2021/12/02 01:56:46 App: TestKVRebInWithTimers got un-deployed
2021/12/02 01:56:46 No eventing-con process running
2021/12/02 01:56:46 Function: TestKVRebInWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVRebInWithTimers deleting in the background"
}
2021/12/02 01:56:48 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 01:56:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 01:56:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 01:56:55 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 01:57:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 01:57:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebInWithTimers (260.11s)
=== RUN   TestKVRebOutWithTimers
2021/12/02 01:57:05 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/02 01:57:05 Removing node(s): [] from the cluster
2021/12/02 01:57:10 Rebalance progress: 100
2021/12/02 01:57:15 Waiting for app: TestKVRebOutWithTimers to get un-deployed
2021/12/02 01:57:15 App: TestKVRebOutWithTimers got un-deployed
2021/12/02 01:57:15 No eventing-con process running
2021/12/02 01:57:15 Function: TestKVRebOutWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVRebOutWithTimers not found"
 }
}
2021/12/02 01:57:17 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 01:57:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 01:57:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 01:57:24 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 01:57:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 01:57:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 01:57:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 01:57:34 Deploying app: TestKVRebOutWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/02 01:57:35 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVRebOutWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/02 01:57:40 Waiting for app: TestKVRebOutWithTimers to get deployed
2021/12/02 01:57:45 Waiting for app: TestKVRebOutWithTimers to get deployed
2021/12/02 01:57:50 Waiting for app: TestKVRebOutWithTimers to get deployed
2021/12/02 01:57:55 Waiting for app: TestKVRebOutWithTimers to get deployed
2021/12/02 01:57:55 App: TestKVRebOutWithTimers got deployed
2021/12/02 01:57:55 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/02 01:57:55 Starting bucket ops to source bucket
2021/12/02 01:58:00 Rebalance progress: 12.5
2021/12/02 01:58:05 Rebalance progress: 12.5
2021/12/02 01:58:10 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 01:58:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 01:58:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 01:58:10 Waiting for src bucket item count to get to: 50000 curr count: 18934
2021/12/02 01:58:15 Waiting for src bucket item count to get to: 50000 curr count: 18934
2021/12/02 01:58:20 Waiting for src bucket item count to get to: 50000 curr count: 28449
2021/12/02 01:58:25 Waiting for src bucket item count to get to: 50000 curr count: 28449
2021/12/02 01:58:30 Waiting for src bucket item count to get to: 50000 curr count: 40790
2021/12/02 01:58:35 Waiting for src bucket item count to get to: 50000 curr count: 40790
2021/12/02 01:58:40 Waiting for src bucket item count to get to: 50000 curr count: 40790
2021/12/02 01:58:50 Waiting for dst bucket item count to get to: 50000 curr count: 40820
2021/12/02 01:58:55 Waiting for dst bucket item count to get to: 50000 curr count: 40820
2021/12/02 01:59:00 Waiting for dst bucket item count to get to: 50000 curr count: 48884
2021/12/02 01:59:05 Waiting for dst bucket item count to get to: 50000 curr count: 48884
2021/12/02 01:59:05 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/02 01:59:05 Post SET ops. Expected item count: 50000 got 50000
2021/12/02 01:59:05 Starting bucket ops to source bucket
2021/12/02 02:00:16 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/02 02:00:21 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/02 02:00:26 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/02 02:00:31 Waiting for dst bucket item count to get to: 0 curr count: 44385
2021/12/02 02:00:36 Waiting for dst bucket item count to get to: 0 curr count: 44385
2021/12/02 02:00:41 Waiting for dst bucket item count to get to: 0 curr count: 28307
2021/12/02 02:00:46 Waiting for dst bucket item count to get to: 0 curr count: 28307
2021/12/02 02:00:51 Waiting for dst bucket item count to get to: 0 curr count: 8770
2021/12/02 02:00:56 Waiting for dst bucket item count to get to: 0 curr count: 8770
2021/12/02 02:00:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 02:00:56 Post DELETE ops. Expected item count: 0 got 0
2021/12/02 02:00:56 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 02:01:01 Waiting for app: TestKVRebOutWithTimers to get un-deployed
2021/12/02 02:01:06 Waiting for app: TestKVRebOutWithTimers to get un-deployed
2021/12/02 02:01:11 Waiting for app: TestKVRebOutWithTimers to get un-deployed
2021/12/02 02:01:16 Waiting for app: TestKVRebOutWithTimers to get un-deployed
2021/12/02 02:01:16 App: TestKVRebOutWithTimers got un-deployed
2021/12/02 02:01:16 No eventing-con process running
2021/12/02 02:01:16 Function: TestKVRebOutWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVRebOutWithTimers deleting in the background"
}
2021/12/02 02:01:18 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:01:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 02:01:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 02:01:25 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:01:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 02:01:30 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebOutWithTimers (269.74s)
=== RUN   TestKVSwapRebWithTimers
2021/12/02 02:01:40 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/02 02:01:40 App: TestKVSwapRebWithTimers got un-deployed
2021/12/02 02:01:40 No eventing-con process running
2021/12/02 02:01:40 Function: TestKVSwapRebWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVSwapRebWithTimers not found"
 }
}
2021/12/02 02:01:41 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:01:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 02:01:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 02:01:48 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:01:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 02:01:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:01:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 02:01:58 Deploying app: TestKVSwapRebWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/02 02:01:59 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVSwapRebWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/02 02:02:04 Waiting for app: TestKVSwapRebWithTimers to get deployed
2021/12/02 02:02:09 Waiting for app: TestKVSwapRebWithTimers to get deployed
2021/12/02 02:02:14 Waiting for app: TestKVSwapRebWithTimers to get deployed
2021/12/02 02:02:19 Waiting for app: TestKVSwapRebWithTimers to get deployed
2021/12/02 02:02:19 App: TestKVSwapRebWithTimers got deployed
2021/12/02 02:02:19 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/02 02:02:19 Starting bucket ops to source bucket
2021/12/02 02:02:20 Removing node(s): [] from the cluster
2021/12/02 02:02:25 Rebalance progress: 12.5
2021/12/02 02:02:30 Rebalance progress: 12.5
2021/12/02 02:02:35 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:02:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:02:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 02:02:35 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/02 02:02:35 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/02 02:02:41 Rebalance progress: 12.5
2021/12/02 02:02:46 Rebalance progress: 12.5
2021/12/02 02:02:51 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:02:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:02:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 02:02:51 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/02 02:02:56 Rebalance progress: 12.5
2021/12/02 02:03:01 Rebalance progress: 12.5
2021/12/02 02:03:06 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:03:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:03:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 02:03:06 Waiting for src bucket item count to get to: 50000 curr count: 46754
2021/12/02 02:03:16 Waiting for dst bucket item count to get to: 50000 curr count: 40574
2021/12/02 02:03:21 Waiting for dst bucket item count to get to: 50000 curr count: 40574
2021/12/02 02:03:26 Waiting for dst bucket item count to get to: 50000 curr count: 48883
2021/12/02 02:03:31 Waiting for dst bucket item count to get to: 50000 curr count: 48883
2021/12/02 02:03:31 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/02 02:03:31 Post SET ops. Expected item count: 50000 got 50000
2021/12/02 02:03:31 Starting bucket ops to source bucket
2021/12/02 02:04:41 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/02 02:04:46 Waiting for dst bucket item count to get to: 0 curr count: 49583
2021/12/02 02:04:51 Waiting for dst bucket item count to get to: 0 curr count: 49583
2021/12/02 02:04:56 Waiting for dst bucket item count to get to: 0 curr count: 40284
2021/12/02 02:05:01 Waiting for dst bucket item count to get to: 0 curr count: 40284
2021/12/02 02:05:06 Waiting for dst bucket item count to get to: 0 curr count: 40284
2021/12/02 02:05:11 Waiting for dst bucket item count to get to: 0 curr count: 30271
2021/12/02 02:05:16 Waiting for dst bucket item count to get to: 0 curr count: 30271
2021/12/02 02:05:21 Waiting for dst bucket item count to get to: 0 curr count: 9852
2021/12/02 02:05:26 Waiting for dst bucket item count to get to: 0 curr count: 9852
2021/12/02 02:05:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 02:05:26 Post DELETE ops. Expected item count: 0 got 0
2021/12/02 02:05:26 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 02:05:31 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/02 02:05:36 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/02 02:05:41 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/02 02:05:47 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/02 02:05:52 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/02 02:05:57 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/02 02:05:57 App: TestKVSwapRebWithTimers got un-deployed
2021/12/02 02:05:57 No eventing-con process running
2021/12/02 02:05:57 Function: TestKVSwapRebWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVSwapRebWithTimers deleting in the background"
}
2021/12/02 02:05:58 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:06:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 02:06:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 02:06:05 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:06:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 02:06:10 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVSwapRebWithTimers (280.59s)
=== RUN   TestDuplicateTimerCreationEventingRebIn
2021/12/02 02:06:20 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/02 02:06:20 App: TestDuplicateTimerCreationEventingRebIn got un-deployed
2021/12/02 02:06:20 No eventing-con process running
2021/12/02 02:06:20 Function: TestDuplicateTimerCreationEventingRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationEventingRebIn not found"
 }
}
2021/12/02 02:06:22 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:06:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 02:06:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 02:06:29 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:06:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 02:06:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:06:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 02:06:39 Deploying app: TestDuplicateTimerCreationEventingRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/02 02:06:40 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationEventingRebIn' in metakv",
  "warnings": null
 }
}
2021/12/02 02:06:45 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get deployed
2021/12/02 02:06:50 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get deployed
2021/12/02 02:06:55 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get deployed
2021/12/02 02:07:00 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get deployed
2021/12/02 02:07:00 App: TestDuplicateTimerCreationEventingRebIn got deployed
2021/12/02 02:07:00 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/02 02:07:00 Starting bucket ops to source bucket
2021/12/02 02:07:00 Removing node(s): [] from the cluster
2021/12/02 02:07:05 Rebalance progress: 12.5
2021/12/02 02:07:10 Rebalance progress: 12.5
2021/12/02 02:07:15 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:07:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:07:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 02:07:15 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/02 02:07:21 Rebalance progress: 12.5
2021/12/02 02:07:26 Rebalance progress: 12.5
2021/12/02 02:07:31 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:07:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:07:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 02:07:31 Waiting for src bucket item count to get to: 50000 curr count: 36453
2021/12/02 02:07:36 Waiting for src bucket item count to get to: 50000 curr count: 36453
2021/12/02 02:07:41 Waiting for src bucket item count to get to: 50000 curr count: 46373
2021/12/02 02:07:46 Waiting for src bucket item count to get to: 50000 curr count: 46373
2021/12/02 02:07:56 Waiting for dst bucket item count to get to: 50000 curr count: 34069
2021/12/02 02:08:01 Waiting for dst bucket item count to get to: 50000 curr count: 34069
2021/12/02 02:08:06 Waiting for dst bucket item count to get to: 50000 curr count: 45623
2021/12/02 02:08:11 Waiting for dst bucket item count to get to: 50000 curr count: 45623
2021/12/02 02:08:16 Waiting for dst bucket item count to get to: 50000 curr count: 49687
2021/12/02 02:08:21 Waiting for dst bucket item count to get to: 50000 curr count: 49687
2021/12/02 02:08:26 Waiting for dst bucket item count to get to: 50000 curr count: 49687
2021/12/02 02:08:26 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/02 02:08:26 Post SET ops. Expected item count: 50000 got 50000
2021/12/02 02:08:26 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 02:08:31 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/02 02:08:36 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/02 02:08:41 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/02 02:08:46 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/02 02:08:51 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/02 02:08:56 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/02 02:08:56 App: TestDuplicateTimerCreationEventingRebIn got un-deployed
2021/12/02 02:08:56 No eventing-con process running
2021/12/02 02:08:56 Function: TestDuplicateTimerCreationEventingRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationEventingRebIn deleting in the background"
}
2021/12/02 02:08:58 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:09:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 02:09:08 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/02 02:09:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 02:09:10 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:09:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 02:09:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationEventingRebIn (184.41s)
=== RUN   TestDuplicateTimerCreationEventingRebOut
2021/12/02 02:09:20 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/02 02:09:20 Removing node(s): [] from the cluster
2021/12/02 02:09:25 Rebalance progress: 100
2021/12/02 02:09:30 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/02 02:09:30 App: TestDuplicateTimerCreationEventingRebOut got un-deployed
2021/12/02 02:09:30 No eventing-con process running
2021/12/02 02:09:30 Function: TestDuplicateTimerCreationEventingRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationEventingRebOut not found"
 }
}
2021/12/02 02:09:32 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:09:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 02:09:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 02:09:39 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:09:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 02:09:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:09:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 02:09:49 Deploying app: TestDuplicateTimerCreationEventingRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/02 02:09:50 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationEventingRebOut' in metakv",
  "warnings": null
 }
}
2021/12/02 02:09:55 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get deployed
2021/12/02 02:10:00 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get deployed
2021/12/02 02:10:05 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get deployed
2021/12/02 02:10:10 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get deployed
2021/12/02 02:10:10 App: TestDuplicateTimerCreationEventingRebOut got deployed
2021/12/02 02:10:10 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/02 02:10:10 Starting bucket ops to source bucket
2021/12/02 02:10:15 Rebalance progress: 12.5
2021/12/02 02:10:20 Rebalance progress: 12.5
2021/12/02 02:10:25 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:10:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:10:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 02:10:25 Waiting for src bucket item count to get to: 50000 curr count: 12151
2021/12/02 02:10:30 Waiting for src bucket item count to get to: 50000 curr count: 26022
2021/12/02 02:10:35 Waiting for src bucket item count to get to: 50000 curr count: 26022
2021/12/02 02:10:40 Waiting for src bucket item count to get to: 50000 curr count: 36382
2021/12/02 02:10:45 Waiting for src bucket item count to get to: 50000 curr count: 36382
2021/12/02 02:10:50 Waiting for src bucket item count to get to: 50000 curr count: 46372
2021/12/02 02:10:55 Waiting for src bucket item count to get to: 50000 curr count: 46372
2021/12/02 02:11:00 Waiting for src bucket item count to get to: 50000 curr count: 46372
2021/12/02 02:11:10 Waiting for dst bucket item count to get to: 50000 curr count: 37876
2021/12/02 02:11:15 Waiting for dst bucket item count to get to: 50000 curr count: 37876
2021/12/02 02:11:20 Waiting for dst bucket item count to get to: 50000 curr count: 47979
2021/12/02 02:11:25 Waiting for dst bucket item count to get to: 50000 curr count: 47979
2021/12/02 02:11:25 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/02 02:11:25 Post SET ops. Expected item count: 50000 got 50000
2021/12/02 02:11:25 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 02:11:30 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/02 02:11:35 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/02 02:11:40 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/02 02:11:45 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/02 02:11:50 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/02 02:11:55 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/02 02:11:55 App: TestDuplicateTimerCreationEventingRebOut got un-deployed
2021/12/02 02:11:55 No eventing-con process running
2021/12/02 02:11:55 Function: TestDuplicateTimerCreationEventingRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationEventingRebOut deleting in the background"
}
2021/12/02 02:11:57 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:12:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 02:12:07 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/02 02:12:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 02:12:09 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:12:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 02:12:19 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/02 02:12:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationEventingRebOut (184.44s)
=== RUN   TestDuplicateTimerCreationEventingSwapReb
2021/12/02 02:12:29 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/02 02:12:29 App: TestDuplicateTimerCreationEventingSwapReb got un-deployed
2021/12/02 02:12:29 No eventing-con process running
2021/12/02 02:12:29 Function: TestDuplicateTimerCreationEventingSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationEventingSwapReb not found"
 }
}
2021/12/02 02:12:31 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:12:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 02:12:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 02:12:38 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:12:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 02:12:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:12:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 02:12:48 Deploying app: TestDuplicateTimerCreationEventingSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/02 02:12:49 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationEventingSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/02 02:12:54 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get deployed
2021/12/02 02:12:59 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get deployed
2021/12/02 02:13:04 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get deployed
2021/12/02 02:13:09 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get deployed
2021/12/02 02:13:09 App: TestDuplicateTimerCreationEventingSwapReb got deployed
2021/12/02 02:13:09 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/02 02:13:09 Starting bucket ops to source bucket
2021/12/02 02:13:09 Removing node(s): [] from the cluster
2021/12/02 02:13:14 Rebalance progress: 12.5
2021/12/02 02:13:19 Rebalance progress: 12.5
2021/12/02 02:13:24 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:13:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:13:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 02:13:24 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/02 02:13:25 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/02 02:13:30 Rebalance progress: 12.5
2021/12/02 02:13:35 Rebalance progress: 12.5
2021/12/02 02:13:40 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:13:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:13:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 02:13:40 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/02 02:13:45 Rebalance progress: 12.5
2021/12/02 02:13:50 Rebalance progress: 12.5
2021/12/02 02:13:55 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:13:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:13:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 02:13:55 Waiting for src bucket item count to get to: 50000 curr count: 46345
2021/12/02 02:14:00 Waiting for src bucket item count to get to: 50000 curr count: 46345
2021/12/02 02:14:10 Waiting for dst bucket item count to get to: 50000 curr count: 35194
2021/12/02 02:14:15 Waiting for dst bucket item count to get to: 50000 curr count: 35194
2021/12/02 02:14:20 Waiting for dst bucket item count to get to: 50000 curr count: 40936
2021/12/02 02:14:25 Waiting for dst bucket item count to get to: 50000 curr count: 40936
2021/12/02 02:14:25 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/02 02:14:25 Post SET ops. Expected item count: 50000 got 50000
2021/12/02 02:14:25 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 02:14:30 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/02 02:14:35 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/02 02:14:40 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/02 02:14:45 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/02 02:14:50 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/02 02:14:55 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/02 02:14:55 App: TestDuplicateTimerCreationEventingSwapReb got un-deployed
2021/12/02 02:14:55 No eventing-con process running
2021/12/02 02:14:55 Function: TestDuplicateTimerCreationEventingSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationEventingSwapReb deleting in the background"
}
2021/12/02 02:14:57 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:15:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 02:15:07 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/02 02:15:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 02:15:09 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:15:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 02:15:19 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/02 02:15:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationEventingSwapReb (179.67s)
=== RUN   TestDuplicateTimerCreationKVRebIn
2021/12/02 02:15:29 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/02 02:15:29 App: TestDuplicateTimerCreationKVRebIn got un-deployed
2021/12/02 02:15:29 No eventing-con process running
2021/12/02 02:15:29 Function: TestDuplicateTimerCreationKVRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationKVRebIn not found"
 }
}
2021/12/02 02:15:31 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:15:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 02:15:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 02:15:37 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:15:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 02:15:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:15:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 02:15:47 Deploying app: TestDuplicateTimerCreationKVRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/02 02:15:49 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationKVRebIn' in metakv",
  "warnings": null
 }
}
2021/12/02 02:15:54 Waiting for app: TestDuplicateTimerCreationKVRebIn to get deployed
2021/12/02 02:15:59 Waiting for app: TestDuplicateTimerCreationKVRebIn to get deployed
2021/12/02 02:16:04 Waiting for app: TestDuplicateTimerCreationKVRebIn to get deployed
2021/12/02 02:16:09 Waiting for app: TestDuplicateTimerCreationKVRebIn to get deployed
2021/12/02 02:16:09 App: TestDuplicateTimerCreationKVRebIn got deployed
2021/12/02 02:16:09 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/02 02:16:09 Starting bucket ops to source bucket
2021/12/02 02:16:09 Removing node(s): [] from the cluster
2021/12/02 02:16:14 Rebalance progress: 12.5
2021/12/02 02:16:19 Rebalance progress: 12.5
2021/12/02 02:16:24 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:16:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:16:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 02:16:24 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/02 02:16:29 Rebalance progress: 12.5
2021/12/02 02:16:34 Rebalance progress: 12.5
2021/12/02 02:16:39 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:16:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:16:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 02:16:39 Waiting for src bucket item count to get to: 50000 curr count: 25644
2021/12/02 02:16:44 Waiting for src bucket item count to get to: 50000 curr count: 35957
2021/12/02 02:16:49 Waiting for src bucket item count to get to: 50000 curr count: 35957
2021/12/02 02:16:54 Waiting for src bucket item count to get to: 50000 curr count: 46854
2021/12/02 02:16:59 Waiting for src bucket item count to get to: 50000 curr count: 46854
2021/12/02 02:17:09 Waiting for dst bucket item count to get to: 50000 curr count: 37107
2021/12/02 02:17:14 Waiting for dst bucket item count to get to: 50000 curr count: 37107
2021/12/02 02:17:19 Waiting for dst bucket item count to get to: 50000 curr count: 44329
2021/12/02 02:17:24 Waiting for dst bucket item count to get to: 50000 curr count: 44329
2021/12/02 02:17:24 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/02 02:17:24 Post SET ops. Expected item count: 50000 got 50000
2021/12/02 02:17:24 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 02:17:29 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/02 02:17:35 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/02 02:17:40 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/02 02:17:45 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/02 02:17:50 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/02 02:17:55 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/02 02:17:55 App: TestDuplicateTimerCreationKVRebIn got un-deployed
2021/12/02 02:17:55 No eventing-con process running
2021/12/02 02:17:55 Function: TestDuplicateTimerCreationKVRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationKVRebIn deleting in the background"
}
2021/12/02 02:17:56 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:18:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 02:18:06 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/02 02:18:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 02:18:08 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:18:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 02:18:18 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/02 02:18:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationKVRebIn (179.46s)
=== RUN   TestDuplicateTimerCreationKVRebOut
2021/12/02 02:18:23 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/02 02:18:24 Removing node(s): [] from the cluster
2021/12/02 02:18:29 Rebalance progress: 100
2021/12/02 02:18:34 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/02 02:18:34 App: TestDuplicateTimerCreationKVRebOut got un-deployed
2021/12/02 02:18:34 No eventing-con process running
2021/12/02 02:18:34 Function: TestDuplicateTimerCreationKVRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationKVRebOut not found"
 }
}
2021/12/02 02:18:36 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:18:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 02:18:41 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 02:18:42 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:18:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 02:18:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:18:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 02:18:52 Deploying app: TestDuplicateTimerCreationKVRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/02 02:18:54 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationKVRebOut' in metakv",
  "warnings": null
 }
}
2021/12/02 02:18:59 Waiting for app: TestDuplicateTimerCreationKVRebOut to get deployed
2021/12/02 02:19:04 Waiting for app: TestDuplicateTimerCreationKVRebOut to get deployed
2021/12/02 02:19:09 Waiting for app: TestDuplicateTimerCreationKVRebOut to get deployed
2021/12/02 02:19:14 Waiting for app: TestDuplicateTimerCreationKVRebOut to get deployed
2021/12/02 02:19:14 App: TestDuplicateTimerCreationKVRebOut got deployed
2021/12/02 02:19:14 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/02 02:19:14 Starting bucket ops to source bucket
2021/12/02 02:19:19 Rebalance progress: 12.5
2021/12/02 02:19:24 Rebalance progress: 12.5
2021/12/02 02:19:29 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:19:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:19:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 02:19:29 Waiting for src bucket item count to get to: 50000 curr count: 7027
2021/12/02 02:19:34 Waiting for src bucket item count to get to: 50000 curr count: 20629
2021/12/02 02:19:39 Waiting for src bucket item count to get to: 50000 curr count: 20629
2021/12/02 02:19:44 Waiting for src bucket item count to get to: 50000 curr count: 34716
2021/12/02 02:19:49 Waiting for src bucket item count to get to: 50000 curr count: 34716
2021/12/02 02:19:54 Waiting for src bucket item count to get to: 50000 curr count: 45449
2021/12/02 02:19:59 Waiting for src bucket item count to get to: 50000 curr count: 45449
2021/12/02 02:20:04 Waiting for src bucket item count to get to: 50000 curr count: 45449
2021/12/02 02:20:14 Waiting for dst bucket item count to get to: 50000 curr count: 29241
2021/12/02 02:20:19 Waiting for dst bucket item count to get to: 50000 curr count: 29241
2021/12/02 02:20:24 Waiting for dst bucket item count to get to: 50000 curr count: 38475
2021/12/02 02:20:29 Waiting for dst bucket item count to get to: 50000 curr count: 38475
2021/12/02 02:20:29 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/02 02:20:29 Post SET ops. Expected item count: 50000 got 50000
2021/12/02 02:20:29 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 02:20:34 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/02 02:20:39 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/02 02:20:44 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/02 02:20:49 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/02 02:20:54 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/02 02:20:54 App: TestDuplicateTimerCreationKVRebOut got un-deployed
2021/12/02 02:20:54 No eventing-con process running
2021/12/02 02:20:54 Function: TestDuplicateTimerCreationKVRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationKVRebOut deleting in the background"
}
2021/12/02 02:20:56 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:21:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 02:21:06 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/02 02:21:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 02:21:08 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:21:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 02:21:18 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/02 02:21:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationKVRebOut (179.52s)
=== RUN   TestDuplicateTimerCreationKVSwapReb
2021/12/02 02:21:28 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/02 02:21:28 App: TestDuplicateTimerCreationKVSwapReb got un-deployed
2021/12/02 02:21:28 No eventing-con process running
2021/12/02 02:21:28 Function: TestDuplicateTimerCreationKVSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationKVSwapReb not found"
 }
}
2021/12/02 02:21:30 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:21:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 02:21:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 02:21:36 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:21:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 02:21:41 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:21:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 02:21:46 Deploying app: TestDuplicateTimerCreationKVSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/02 02:21:48 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationKVSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/02 02:21:53 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get deployed
2021/12/02 02:21:58 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get deployed
2021/12/02 02:22:03 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get deployed
2021/12/02 02:22:08 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get deployed
2021/12/02 02:22:08 App: TestDuplicateTimerCreationKVSwapReb got deployed
2021/12/02 02:22:08 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/02 02:22:08 Starting bucket ops to source bucket
2021/12/02 02:22:08 Removing node(s): [] from the cluster
2021/12/02 02:22:13 Rebalance progress: 12.5
2021/12/02 02:22:18 Rebalance progress: 12.5
2021/12/02 02:22:23 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:22:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:22:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 02:22:23 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/02 02:22:23 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/02 02:22:28 Rebalance progress: 12.5
2021/12/02 02:22:33 Rebalance progress: 12.5
2021/12/02 02:22:38 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:22:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:22:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 02:22:38 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/02 02:22:43 Rebalance progress: 12.5
2021/12/02 02:22:48 Rebalance progress: 12.5
2021/12/02 02:22:53 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:22:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:22:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 02:22:54 Waiting for src bucket item count to get to: 50000 curr count: 48641
2021/12/02 02:22:59 Waiting for src bucket item count to get to: 50000 curr count: 48641
2021/12/02 02:23:09 Waiting for dst bucket item count to get to: 50000 curr count: 35434
2021/12/02 02:23:14 Waiting for dst bucket item count to get to: 50000 curr count: 35434
2021/12/02 02:23:19 Waiting for dst bucket item count to get to: 50000 curr count: 41353
2021/12/02 02:23:24 Waiting for dst bucket item count to get to: 50000 curr count: 41353
2021/12/02 02:23:24 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/02 02:23:24 Post SET ops. Expected item count: 50000 got 50000
2021/12/02 02:23:24 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 02:23:29 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/02 02:23:34 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/02 02:23:39 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/02 02:23:44 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/02 02:23:49 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/02 02:23:54 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/02 02:23:54 App: TestDuplicateTimerCreationKVSwapReb got un-deployed
2021/12/02 02:23:54 No eventing-con process running
2021/12/02 02:23:54 Function: TestDuplicateTimerCreationKVSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationKVSwapReb deleting in the background"
}
2021/12/02 02:23:56 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:24:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 02:24:06 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/02 02:24:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 02:24:07 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:24:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 02:24:17 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/02 02:24:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationKVSwapReb (179.65s)
=== RUN   TestDuplicateTimerFiringEventingRebIn
2021/12/02 02:24:27 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/02 02:24:27 App: TestDuplicateTimerFiringEventingRebIn got un-deployed
2021/12/02 02:24:27 No eventing-con process running
2021/12/02 02:24:27 Function: TestDuplicateTimerFiringEventingRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringEventingRebIn not found"
 }
}
2021/12/02 02:24:29 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:24:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 02:24:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 02:24:36 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:24:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 02:24:41 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:24:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 02:24:46 Deploying app: TestDuplicateTimerFiringEventingRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/02 02:24:47 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringEventingRebIn' in metakv",
  "warnings": null
 }
}
2021/12/02 02:24:52 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get deployed
2021/12/02 02:24:57 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get deployed
2021/12/02 02:25:02 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get deployed
2021/12/02 02:25:07 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get deployed
2021/12/02 02:25:07 App: TestDuplicateTimerFiringEventingRebIn got deployed
2021/12/02 02:25:07 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/02 02:25:07 Starting bucket ops to source bucket
2021/12/02 02:25:08 Removing node(s): [] from the cluster
2021/12/02 02:25:13 Rebalance progress: 12.5
2021/12/02 02:25:18 Rebalance progress: 12.5
2021/12/02 02:25:23 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:25:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:25:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 02:25:23 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/02 02:25:28 Rebalance progress: 12.5
2021/12/02 02:25:33 Rebalance progress: 12.5
2021/12/02 02:25:38 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:25:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:25:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 02:25:38 Waiting for src bucket item count to get to: 50000 curr count: 26296
2021/12/02 02:25:43 Waiting for src bucket item count to get to: 50000 curr count: 36739
2021/12/02 02:25:48 Waiting for src bucket item count to get to: 50000 curr count: 36739
2021/12/02 02:25:53 Waiting for src bucket item count to get to: 50000 curr count: 48804
2021/12/02 02:25:58 Waiting for src bucket item count to get to: 50000 curr count: 48804
2021/12/02 02:26:08 Waiting for dst bucket item count to get to: 50000 curr count: 46724
2021/12/02 02:26:13 Waiting for dst bucket item count to get to: 50000 curr count: 46724
2021/12/02 02:26:13 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/02 02:26:13 Post SET ops. Expected item count: 50000 got 50000
2021/12/02 02:26:13 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 02:26:18 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/02 02:26:23 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/02 02:26:28 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/02 02:26:34 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/02 02:26:39 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/02 02:26:39 App: TestDuplicateTimerFiringEventingRebIn got un-deployed
2021/12/02 02:26:39 No eventing-con process running
2021/12/02 02:26:39 Function: TestDuplicateTimerFiringEventingRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringEventingRebIn deleting in the background"
}
2021/12/02 02:26:40 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:26:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 02:26:50 Waiting for dst bucket item count to get to: 0 curr count: 20961
2021/12/02 02:26:55 Waiting for dst bucket item count to get to: 0 curr count: 20961
2021/12/02 02:26:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 02:26:57 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:27:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 02:27:07 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/02 02:27:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringEventingRebIn (169.76s)
=== RUN   TestDuplicateTimerFiringEventingRebOut
2021/12/02 02:27:12 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/02 02:27:12 Removing node(s): [] from the cluster
2021/12/02 02:27:17 Rebalance progress: 100
2021/12/02 02:27:22 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/02 02:27:23 App: TestDuplicateTimerFiringEventingRebOut got un-deployed
2021/12/02 02:27:23 No eventing-con process running
2021/12/02 02:27:23 Function: TestDuplicateTimerFiringEventingRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringEventingRebOut not found"
 }
}
2021/12/02 02:27:24 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:27:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 02:27:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 02:27:31 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:27:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 02:27:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:27:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 02:27:41 Deploying app: TestDuplicateTimerFiringEventingRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/02 02:27:42 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringEventingRebOut' in metakv",
  "warnings": null
 }
}
2021/12/02 02:27:47 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get deployed
2021/12/02 02:27:52 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get deployed
2021/12/02 02:27:57 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get deployed
2021/12/02 02:28:02 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get deployed
2021/12/02 02:28:02 App: TestDuplicateTimerFiringEventingRebOut got deployed
2021/12/02 02:28:02 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/02 02:28:02 Starting bucket ops to source bucket
2021/12/02 02:28:07 Rebalance progress: 12.5
2021/12/02 02:28:12 Rebalance progress: 12.5
2021/12/02 02:28:17 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:28:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:28:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 02:28:17 Waiting for src bucket item count to get to: 50000 curr count: 8824
2021/12/02 02:28:22 Waiting for src bucket item count to get to: 50000 curr count: 20924
2021/12/02 02:28:27 Waiting for src bucket item count to get to: 50000 curr count: 20924
2021/12/02 02:28:32 Waiting for src bucket item count to get to: 50000 curr count: 32355
2021/12/02 02:28:37 Waiting for src bucket item count to get to: 50000 curr count: 32355
2021/12/02 02:28:42 Waiting for src bucket item count to get to: 50000 curr count: 42353
2021/12/02 02:28:47 Waiting for src bucket item count to get to: 50000 curr count: 42353
2021/12/02 02:28:52 Waiting for src bucket item count to get to: 50000 curr count: 42353
2021/12/02 02:29:02 Waiting for dst bucket item count to get to: 50000 curr count: 38962
2021/12/02 02:29:07 Waiting for dst bucket item count to get to: 50000 curr count: 38962
2021/12/02 02:29:07 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/02 02:29:07 Post SET ops. Expected item count: 50000 got 50000
2021/12/02 02:29:08 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 02:29:13 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/02 02:29:18 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/02 02:29:23 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/02 02:29:28 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/02 02:29:33 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/02 02:29:38 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/02 02:29:38 App: TestDuplicateTimerFiringEventingRebOut got un-deployed
2021/12/02 02:29:38 No eventing-con process running
2021/12/02 02:29:38 Function: TestDuplicateTimerFiringEventingRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringEventingRebOut deleting in the background"
}
2021/12/02 02:29:40 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:29:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 02:29:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 02:29:46 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:29:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 02:29:56 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/02 02:29:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringEventingRebOut (169.25s)
=== RUN   TestDuplicateTimerFiringEventingSwapReb
2021/12/02 02:30:06 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/02 02:30:06 App: TestDuplicateTimerFiringEventingSwapReb got un-deployed
2021/12/02 02:30:06 No eventing-con process running
2021/12/02 02:30:06 Function: TestDuplicateTimerFiringEventingSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringEventingSwapReb not found"
 }
}
2021/12/02 02:30:08 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:30:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 02:30:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 02:30:15 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:30:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 02:30:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:30:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 02:30:25 Deploying app: TestDuplicateTimerFiringEventingSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/02 02:30:26 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringEventingSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/02 02:30:31 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get deployed
2021/12/02 02:30:36 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get deployed
2021/12/02 02:30:41 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get deployed
2021/12/02 02:30:46 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get deployed
2021/12/02 02:30:46 App: TestDuplicateTimerFiringEventingSwapReb got deployed
2021/12/02 02:30:46 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/02 02:30:46 Starting bucket ops to source bucket
2021/12/02 02:30:47 Removing node(s): [] from the cluster
2021/12/02 02:30:52 Rebalance progress: 12.5
2021/12/02 02:30:57 Rebalance progress: 12.5
2021/12/02 02:31:02 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:31:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:31:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 02:31:02 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/02 02:31:02 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/02 02:31:07 Rebalance progress: 12.5
2021/12/02 02:31:12 Rebalance progress: 12.5
2021/12/02 02:31:17 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:31:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:31:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 02:31:17 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/02 02:31:22 Rebalance progress: 12.5
2021/12/02 02:31:27 Rebalance progress: 12.5
2021/12/02 02:31:32 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:31:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:31:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 02:31:32 Waiting for src bucket item count to get to: 50000 curr count: 48889
2021/12/02 02:31:37 Waiting for src bucket item count to get to: 50000 curr count: 48889
2021/12/02 02:31:47 Waiting for dst bucket item count to get to: 50000 curr count: 41740
2021/12/02 02:31:52 Waiting for dst bucket item count to get to: 50000 curr count: 41740
2021/12/02 02:31:52 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/02 02:31:52 Post SET ops. Expected item count: 50000 got 50000
2021/12/02 02:31:53 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 02:31:58 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/02 02:32:03 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/02 02:32:08 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/02 02:32:13 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/02 02:32:18 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/02 02:32:18 App: TestDuplicateTimerFiringEventingSwapReb got un-deployed
2021/12/02 02:32:18 No eventing-con process running
2021/12/02 02:32:18 Function: TestDuplicateTimerFiringEventingSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringEventingSwapReb deleting in the background"
}
2021/12/02 02:32:20 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:32:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 02:32:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 02:32:26 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:32:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 02:32:36 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/02 02:32:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringEventingSwapReb (159.89s)
=== RUN   TestDuplicateTimerFiringKVRebIn
2021/12/02 02:32:46 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/02 02:32:46 App: TestDuplicateTimerFiringKVRebIn got un-deployed
2021/12/02 02:32:46 No eventing-con process running
2021/12/02 02:32:46 Function: TestDuplicateTimerFiringKVRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringKVRebIn not found"
 }
}
2021/12/02 02:32:48 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:32:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 02:32:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 02:32:55 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:33:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 02:33:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:33:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 02:33:05 Deploying app: TestDuplicateTimerFiringKVRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/02 02:33:06 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringKVRebIn' in metakv",
  "warnings": null
 }
}
2021/12/02 02:33:11 Waiting for app: TestDuplicateTimerFiringKVRebIn to get deployed
2021/12/02 02:33:16 Waiting for app: TestDuplicateTimerFiringKVRebIn to get deployed
2021/12/02 02:33:21 Waiting for app: TestDuplicateTimerFiringKVRebIn to get deployed
2021/12/02 02:33:26 Waiting for app: TestDuplicateTimerFiringKVRebIn to get deployed
2021/12/02 02:33:26 App: TestDuplicateTimerFiringKVRebIn got deployed
2021/12/02 02:33:26 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/02 02:33:26 Starting bucket ops to source bucket
2021/12/02 02:33:26 Removing node(s): [] from the cluster
2021/12/02 02:33:31 Rebalance progress: 12.5
2021/12/02 02:33:36 Rebalance progress: 12.5
2021/12/02 02:33:41 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:33:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:33:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 02:33:41 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/02 02:33:46 Rebalance progress: 12.5
2021/12/02 02:33:51 Rebalance progress: 12.5
2021/12/02 02:33:56 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:33:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:33:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 02:33:56 Waiting for src bucket item count to get to: 50000 curr count: 28060
2021/12/02 02:34:01 Waiting for src bucket item count to get to: 50000 curr count: 39259
2021/12/02 02:34:07 Waiting for src bucket item count to get to: 50000 curr count: 39259
2021/12/02 02:34:17 Waiting for dst bucket item count to get to: 50000 curr count: 34616
2021/12/02 02:34:22 Waiting for dst bucket item count to get to: 50000 curr count: 34616
2021/12/02 02:34:27 Waiting for dst bucket item count to get to: 50000 curr count: 44629
2021/12/02 02:34:32 Waiting for dst bucket item count to get to: 50000 curr count: 44629
2021/12/02 02:34:37 Waiting for dst bucket item count to get to: 50000 curr count: 44629
2021/12/02 02:34:37 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/02 02:34:37 Post SET ops. Expected item count: 50000 got 50000
2021/12/02 02:34:37 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 02:34:42 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/02 02:34:47 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/02 02:34:52 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/02 02:34:57 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/02 02:35:02 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/02 02:35:07 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/02 02:35:07 App: TestDuplicateTimerFiringKVRebIn got un-deployed
2021/12/02 02:35:07 No eventing-con process running
2021/12/02 02:35:07 Function: TestDuplicateTimerFiringKVRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringKVRebIn deleting in the background"
}
2021/12/02 02:35:09 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:35:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 02:35:19 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/02 02:35:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 02:35:21 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:35:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 02:35:31 Waiting for dst bucket item count to get to: 0 curr count: 32689
2021/12/02 02:35:36 Waiting for dst bucket item count to get to: 0 curr count: 32689
2021/12/02 02:35:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringKVRebIn (179.28s)
=== RUN   TestDuplicateTimerFiringKVRebOut
2021/12/02 02:35:41 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/02 02:35:41 Removing node(s): [] from the cluster
2021/12/02 02:35:46 Rebalance progress: 100
2021/12/02 02:35:51 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/02 02:35:51 App: TestDuplicateTimerFiringKVRebOut got un-deployed
2021/12/02 02:35:51 No eventing-con process running
2021/12/02 02:35:51 Function: TestDuplicateTimerFiringKVRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringKVRebOut not found"
 }
}
2021/12/02 02:35:53 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:35:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 02:35:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 02:36:00 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:36:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 02:36:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:36:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 02:36:10 Deploying app: TestDuplicateTimerFiringKVRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/02 02:36:11 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringKVRebOut' in metakv",
  "warnings": null
 }
}
2021/12/02 02:36:16 Waiting for app: TestDuplicateTimerFiringKVRebOut to get deployed
2021/12/02 02:36:21 Waiting for app: TestDuplicateTimerFiringKVRebOut to get deployed
2021/12/02 02:36:26 Waiting for app: TestDuplicateTimerFiringKVRebOut to get deployed
2021/12/02 02:36:31 Waiting for app: TestDuplicateTimerFiringKVRebOut to get deployed
2021/12/02 02:36:31 App: TestDuplicateTimerFiringKVRebOut got deployed
2021/12/02 02:36:31 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/02 02:36:31 Starting bucket ops to source bucket
2021/12/02 02:36:36 Rebalance progress: 12.5
2021/12/02 02:36:41 Rebalance progress: 12.5
2021/12/02 02:36:46 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:36:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:36:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 02:36:46 Waiting for src bucket item count to get to: 50000 curr count: 11257
2021/12/02 02:36:51 Waiting for src bucket item count to get to: 50000 curr count: 25159
2021/12/02 02:36:56 Waiting for src bucket item count to get to: 50000 curr count: 25159
2021/12/02 02:37:01 Waiting for src bucket item count to get to: 50000 curr count: 35918
2021/12/02 02:37:06 Waiting for src bucket item count to get to: 50000 curr count: 35918
2021/12/02 02:37:11 Waiting for src bucket item count to get to: 50000 curr count: 46211
2021/12/02 02:37:16 Waiting for src bucket item count to get to: 50000 curr count: 46211
2021/12/02 02:37:21 Waiting for src bucket item count to get to: 50000 curr count: 46211
2021/12/02 02:37:31 Waiting for dst bucket item count to get to: 50000 curr count: 35833
2021/12/02 02:37:36 Waiting for dst bucket item count to get to: 50000 curr count: 35833
2021/12/02 02:37:41 Waiting for dst bucket item count to get to: 50000 curr count: 49986
2021/12/02 02:37:46 Waiting for dst bucket item count to get to: 50000 curr count: 49986
2021/12/02 02:37:46 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/02 02:37:46 Post SET ops. Expected item count: 50000 got 50000
2021/12/02 02:37:46 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 02:37:51 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/02 02:37:56 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/02 02:38:01 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/02 02:38:06 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/02 02:38:11 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/02 02:38:16 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/02 02:38:16 App: TestDuplicateTimerFiringKVRebOut got un-deployed
2021/12/02 02:38:16 No eventing-con process running
2021/12/02 02:38:16 Function: TestDuplicateTimerFiringKVRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringKVRebOut deleting in the background"
}
2021/12/02 02:38:18 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:38:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 02:38:28 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/02 02:38:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 02:38:30 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:38:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 02:38:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringKVRebOut (179.23s)
=== RUN   TestDuplicateTimerFiringKVSwapReb
2021/12/02 02:38:45 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/02 02:38:45 App: TestDuplicateTimerFiringKVSwapReb got un-deployed
2021/12/02 02:38:45 No eventing-con process running
2021/12/02 02:38:45 Function: TestDuplicateTimerFiringKVSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringKVSwapReb not found"
 }
}
2021/12/02 02:38:47 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:38:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 02:38:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 02:38:53 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:38:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 02:38:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:39:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 02:39:04 Deploying app: TestDuplicateTimerFiringKVSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/02 02:39:05 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringKVSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/02 02:39:10 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get deployed
2021/12/02 02:39:15 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get deployed
2021/12/02 02:39:20 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get deployed
2021/12/02 02:39:25 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get deployed
2021/12/02 02:39:25 App: TestDuplicateTimerFiringKVSwapReb got deployed
2021/12/02 02:39:25 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/02 02:39:25 Starting bucket ops to source bucket
2021/12/02 02:39:25 Removing node(s): [] from the cluster
2021/12/02 02:39:30 Rebalance progress: 12.5
2021/12/02 02:39:35 Rebalance progress: 12.5
2021/12/02 02:39:40 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:39:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:39:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 02:39:40 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/02 02:39:40 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/02 02:39:46 Rebalance progress: 12.5
2021/12/02 02:39:51 Rebalance progress: 12.5
2021/12/02 02:39:56 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:39:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:39:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 02:39:56 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/02 02:40:01 Rebalance progress: 12.5
2021/12/02 02:40:06 Rebalance progress: 12.5
2021/12/02 02:40:11 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:40:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:40:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 02:40:16 Waiting for dst bucket item count to get to: 50000 curr count: 33591
2021/12/02 02:40:21 Waiting for dst bucket item count to get to: 50000 curr count: 33591
2021/12/02 02:40:26 Waiting for dst bucket item count to get to: 50000 curr count: 41675
2021/12/02 02:40:31 Waiting for dst bucket item count to get to: 50000 curr count: 41675
2021/12/02 02:40:31 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/02 02:40:31 Post SET ops. Expected item count: 50000 got 50000
2021/12/02 02:40:31 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 02:40:36 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/02 02:40:41 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/02 02:40:46 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/02 02:40:51 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/02 02:40:56 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/02 02:41:01 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/02 02:41:01 App: TestDuplicateTimerFiringKVSwapReb got un-deployed
2021/12/02 02:41:01 No eventing-con process running
2021/12/02 02:41:01 Function: TestDuplicateTimerFiringKVSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringKVSwapReb deleting in the background"
}
2021/12/02 02:41:03 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:41:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 02:41:13 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/02 02:41:18 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/02 02:41:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 02:41:20 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:41:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 02:41:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringKVSwapReb (169.86s)
=== RUN   TestKVRebOnUpdateBucketOpOneByOne
2021/12/02 02:41:35 Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 02:41:35 App: bucket_op_on_update got un-deployed
2021/12/02 02:41:35 No eventing-con process running
2021/12/02 02:41:35 Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/02 02:41:37 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:41:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 02:41:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 02:41:43 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:41:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 02:41:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:41:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 02:41:53 Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/02 02:41:54 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/02 02:41:59 Starting bucket ops to source bucket
2021/12/02 02:42:04 Waiting for app: bucket_op_on_update to get deployed
2021/12/02 02:42:09 Waiting for app: bucket_op_on_update to get deployed
2021/12/02 02:42:14 Waiting for app: bucket_op_on_update to get deployed
2021/12/02 02:42:14 App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:42:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:42:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 02:42:14 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/02 02:42:15 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/02 02:42:20 Rebalance progress: 12.5
2021/12/02 02:42:25 Rebalance progress: 12.5
2021/12/02 02:42:30 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:42:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:42:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 02:42:30 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/02 02:42:30 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/02 02:42:35 Rebalance progress: 12.5
2021/12/02 02:42:40 Rebalance progress: 12.5
2021/12/02 02:42:45 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:42:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:42:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 02:42:45 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: kv to the cluster
2021/12/02 02:42:45 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/02 02:42:50 Rebalance progress: 12.5
2021/12/02 02:42:55 Rebalance progress: 12.5
2021/12/02 02:43:00 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:43:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:43:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 02:43:00 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/02 02:43:05 Rebalance progress: 12.5
2021/12/02 02:43:10 Rebalance progress: 12.5
2021/12/02 02:43:15 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:43:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:43:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 02:43:15 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/02 02:43:21 Rebalance progress: 12.5
2021/12/02 02:43:26 Rebalance progress: 12.5
2021/12/02 02:43:31 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:43:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:43:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 02:43:31 clusterSetup.go:253: Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/02 02:43:36 Rebalance progress: 12.5
2021/12/02 02:43:41 Rebalance progress: 12.5
2021/12/02 02:43:46 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:43:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:43:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 02:43:46 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 02:43:51 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 02:43:56 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 02:44:01 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 02:44:06 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 02:44:11 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 02:44:11 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/02 02:44:11 eventingSetup.go:46: No eventing-con process running
2021/12/02 02:44:11 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/02 02:44:13 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:44:18 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 02:44:23 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 10608
2021/12/02 02:44:28 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 10608
2021/12/02 02:44:28 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 02:44:30 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:44:35 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 02:44:35 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebOnUpdateBucketOpOneByOne (189.88s)
=== RUN   TestKVForceFailover
2021/12/02 02:44:50 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 02:44:50 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/02 02:44:50 eventingSetup.go:46: No eventing-con process running
2021/12/02 02:44:50 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/02 02:44:51 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:44:56 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 02:44:56 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 02:44:58 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:45:03 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 02:45:03 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 02:45:03 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/02 02:45:03 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/02 02:45:08 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:45:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 02:45:08 Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/02 02:45:10 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/02 02:45:10 Starting bucket ops to source bucket
2021/12/02 02:45:15 Waiting for app: bucket_op_on_update to get deployed
2021/12/02 02:45:20 Waiting for app: bucket_op_on_update to get deployed
2021/12/02 02:45:25 Waiting for app: bucket_op_on_update to get deployed
2021/12/02 02:45:30 Waiting for app: bucket_op_on_update to get deployed
2021/12/02 02:45:30 App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:45:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:45:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 02:45:30 clusterSetup.go:229: Failing over: [127.0.0.1:19001]
2021/12/02 02:45:50 clusterSetup.go:237: Kicking off failover recovery, type: full
2021/12/02 02:45:50 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/02 02:45:55 Rebalance progress: 12.5
2021/12/02 02:46:00 Rebalance progress: 12.5
2021/12/02 02:46:05 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:46:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:46:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 02:46:05 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002] from the cluster
2021/12/02 02:46:10 Rebalance progress: 12.5
2021/12/02 02:46:15 Rebalance progress: 12.5
2021/12/02 02:46:20 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:46:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:46:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 02:46:20 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 02:46:25 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 02:46:30 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 02:46:35 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 02:46:40 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 02:46:45 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 02:46:50 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 02:46:50 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/02 02:46:50 eventingSetup.go:46: No eventing-con process running
2021/12/02 02:46:50 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/02 02:46:52 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:46:57 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 02:47:02 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7038
2021/12/02 02:47:07 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7038
2021/12/02 02:47:07 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 02:47:09 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:47:14 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 02:47:19 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7038
2021/12/02 02:47:19 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVForceFailover (164.48s)
=== RUN   TestKVHardFailoverRecovery
2021/12/02 02:47:29 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 02:47:29 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/02 02:47:29 eventingSetup.go:46: No eventing-con process running
2021/12/02 02:47:29 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/02 02:47:31 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:47:36 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 02:47:36 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 02:47:38 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:47:43 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 02:47:43 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:47:48 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 02:47:48 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/02 02:47:49 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/02 02:47:54 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/02 02:47:59 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/02 02:48:04 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/02 02:48:09 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/02 02:48:09 clusterSetup.go:354: App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:48:09 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:48:09 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 02:48:09 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/02 02:48:09 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/02 02:48:14 Rebalance progress: 12.5
2021/12/02 02:48:19 Rebalance progress: 12.5
2021/12/02 02:48:24 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:48:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:48:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 02:48:24 clusterSetup.go:229: Failing over: [127.0.0.1:19001]
2021/12/02 02:49:24 clusterSetup.go:237: Kicking off failover recovery, type: full
2021/12/02 02:49:29 Rebalance progress: 12.5
2021/12/02 02:49:34 Rebalance progress: 12.5
2021/12/02 02:49:39 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:49:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:49:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 02:49:39 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/02 02:49:45 Rebalance progress: 12.5
2021/12/02 02:49:50 Rebalance progress: 12.5
2021/12/02 02:49:55 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:49:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:49:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 02:49:55 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 02:50:00 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 02:50:05 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 02:50:10 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 02:50:15 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 02:50:20 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 02:50:25 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 02:50:25 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/02 02:50:25 eventingSetup.go:46: No eventing-con process running
2021/12/02 02:50:25 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/02 02:50:27 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:50:32 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 02:50:37 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12065
2021/12/02 02:50:37 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 02:50:39 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:50:44 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 02:50:49 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12065
2021/12/02 02:50:49 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVHardFailoverRecovery (209.60s)
=== RUN   TestKVRebOnUpdateBucketOpAllAtOnce
2021/12/02 02:50:59 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 02:50:59 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/02 02:50:59 eventingSetup.go:46: No eventing-con process running
2021/12/02 02:50:59 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/02 02:51:00 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:51:05 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 02:51:06 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 02:51:07 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:51:12 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 02:51:12 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:51:17 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 02:51:17 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/02 02:51:18 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/02 02:51:23 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/02 02:51:28 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/02 02:51:33 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/02 02:51:38 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/02 02:51:38 clusterSetup.go:354: App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:51:38 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:51:38 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 02:51:38 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/02 02:51:39 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/02 02:51:40 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: kv to the cluster
2021/12/02 02:51:40 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/02 02:51:45 Rebalance progress: 12.5
2021/12/02 02:51:50 Rebalance progress: 12.5
2021/12/02 02:51:55 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:51:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:51:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 02:51:55 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002 127.0.0.1:19003] from the cluster
2021/12/02 02:52:00 Rebalance progress: 12.5
2021/12/02 02:52:05 Rebalance progress: 12.5
2021/12/02 02:52:10 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:52:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:52:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 02:52:10 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 02:52:15 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 02:52:20 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 02:52:25 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 02:52:30 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 02:52:35 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 02:52:40 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 02:52:40 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/02 02:52:40 eventingSetup.go:46: No eventing-con process running
2021/12/02 02:52:40 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/02 02:52:42 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:52:47 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 02:52:52 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4636
2021/12/02 02:52:57 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4636
2021/12/02 02:52:57 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 02:52:59 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:53:04 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 02:53:09 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4636
2021/12/02 02:53:09 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebOnUpdateBucketOpAllAtOnce (140.19s)
=== RUN   TestKVSwapRebOnUpdateBucketOp
2021/12/02 02:53:19 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 02:53:19 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/02 02:53:19 eventingSetup.go:46: No eventing-con process running
2021/12/02 02:53:19 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/02 02:53:21 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:53:26 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 02:53:26 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 02:53:28 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:53:33 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 02:53:33 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:53:33 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 02:53:33 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/02 02:53:34 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/02 02:53:39 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/02 02:53:44 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/02 02:53:49 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/02 02:53:54 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/02 02:53:54 clusterSetup.go:354: App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:53:54 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:53:54 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 02:53:54 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/02 02:53:54 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/02 02:53:59 Rebalance progress: 12.5
2021/12/02 02:54:04 Rebalance progress: 12.5
2021/12/02 02:54:09 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:54:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:54:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 02:54:09 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/02 02:54:09 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/02 02:54:14 Rebalance progress: 12.5
2021/12/02 02:54:19 Rebalance progress: 12.5
2021/12/02 02:54:24 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:54:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:54:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 02:54:24 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/02 02:54:29 Rebalance progress: 12.5
2021/12/02 02:54:34 Rebalance progress: 12.5
2021/12/02 02:54:39 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:54:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:54:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 02:54:40 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 02:54:45 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 02:54:50 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 02:54:55 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 02:55:00 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 02:55:05 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 02:55:05 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/02 02:55:05 eventingSetup.go:46: No eventing-con process running
2021/12/02 02:55:05 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/02 02:55:07 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:55:12 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 02:55:17 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6075
2021/12/02 02:55:17 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 02:55:19 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:55:24 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 02:55:29 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6075
2021/12/02 02:55:29 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 02:55:34 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 02:55:34 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/02 02:55:34 eventingSetup.go:46: No eventing-con process running
2021/12/02 02:55:34 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/02 02:55:35 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:55:40 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 02:55:40 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 02:55:42 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:55:47 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 02:55:47 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVSwapRebOnUpdateBucketOp (158.30s)
=== RUN   TestKVRebalanceWithMultipleHandlers
2021/12/02 02:55:57 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 02:55:57 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/02 02:55:57 eventingSetup.go:46: No eventing-con process running
2021/12/02 02:55:57 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/02 02:55:59 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:56:04 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 02:56:04 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 02:56:06 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:56:11 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 02:56:11 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 02:56:16 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/02 02:56:16 clusterSetup.go:405: App: sys_test_bucket_op got un-deployed
2021/12/02 02:56:16 eventingSetup.go:46: No eventing-con process running
2021/12/02 02:56:16 eventingSetup.go:418: Function: sys_test_bucket_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: sys_test_bucket_op not found"
 }
}
2021/12/02 02:56:18 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:56:23 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 02:56:23 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 02:56:24 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:56:29 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 02:56:29 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:56:29 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 02:56:29 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/02 02:56:30 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:56:30 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 02:56:31 eventingSetup.go:125: Deploying app: sys_test_bucket_op. Item count src bucket: 0 dst bucket: 0
2021/12/02 02:56:32 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'sys_test_bucket_op' in metakv",
  "warnings": null
 }
}
2021/12/02 02:56:37 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/02 02:56:42 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/02 02:56:47 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/02 02:56:52 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/02 02:56:52 clusterSetup.go:354: App: bucket_op_on_update got deployed
2021/12/02 02:56:57 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/02 02:57:02 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/02 02:57:07 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/02 02:57:07 clusterSetup.go:354: App: sys_test_bucket_op got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:57:07 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:57:07 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 02:57:07 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/02 02:57:07 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/02 02:57:12 Rebalance progress: 12.5
2021/12/02 02:57:17 Rebalance progress: 12.5
2021/12/02 02:57:22 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:57:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:57:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 02:57:22 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/02 02:57:22 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/02 02:57:27 Rebalance progress: 12.5
2021/12/02 02:57:32 Rebalance progress: 12.5
2021/12/02 02:57:37 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:57:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:57:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 02:57:37 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: kv to the cluster
2021/12/02 02:57:38 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/02 02:57:43 Rebalance progress: 12.5
2021/12/02 02:57:48 Rebalance progress: 12.5
2021/12/02 02:57:53 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:57:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:57:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 02:57:53 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/02 02:57:58 Rebalance progress: 12.5
2021/12/02 02:58:03 Rebalance progress: 12.5
2021/12/02 02:58:08 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:58:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:58:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 02:58:08 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/02 02:58:13 Rebalance progress: 12.5
2021/12/02 02:58:18 Rebalance progress: 12.5
2021/12/02 02:58:23 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:58:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:58:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 02:58:24 clusterSetup.go:253: Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/02 02:58:29 Rebalance progress: 12.5
2021/12/02 02:58:34 Rebalance progress: 12.5
2021/12/02 02:58:39 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:58:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:58:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 02:58:39 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 02:58:44 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 02:58:49 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 02:58:54 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 02:58:59 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 02:59:04 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 02:59:04 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/02 02:59:04 eventingSetup.go:46: No eventing-con process running
2021/12/02 02:59:04 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/02 02:59:08 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:59:13 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 02:59:18 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12172
2021/12/02 02:59:18 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 02:59:20 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 02:59:25 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 02:59:30 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 872
2021/12/02 02:59:35 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 872
2021/12/02 02:59:35 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 02:59:35 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 02:59:40 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/02 02:59:45 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/02 02:59:50 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/02 02:59:55 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/02 03:00:00 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/02 03:00:05 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/02 03:00:05 clusterSetup.go:405: App: sys_test_bucket_op got un-deployed
2021/12/02 03:00:05 eventingSetup.go:46: No eventing-con process running
2021/12/02 03:00:05 eventingSetup.go:418: Function: sys_test_bucket_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: sys_test_bucket_op deleting in the background"
}
2021/12/02 03:00:07 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 03:00:12 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 03:00:12 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 03:00:14 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 03:00:19 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 03:00:19 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebalanceWithMultipleHandlers (271.83s)
=== RUN   TestKVSwapRebalanceWithMultipleHandlers
2021/12/02 03:00:29 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 03:00:29 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/02 03:00:29 eventingSetup.go:46: No eventing-con process running
2021/12/02 03:00:29 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/02 03:00:31 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 03:00:36 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 03:00:36 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 03:00:38 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 03:00:43 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 03:00:43 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 03:00:48 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/02 03:00:48 clusterSetup.go:405: App: sys_test_bucket_op got un-deployed
2021/12/02 03:00:48 eventingSetup.go:46: No eventing-con process running
2021/12/02 03:00:48 eventingSetup.go:418: Function: sys_test_bucket_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: sys_test_bucket_op not found"
 }
}
2021/12/02 03:00:49 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 03:00:54 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 03:00:54 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 03:00:56 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 03:01:01 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 03:01:01 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 03:01:01 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 03:01:01 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/02 03:01:02 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 03:01:02 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 03:01:02 eventingSetup.go:125: Deploying app: sys_test_bucket_op. Item count src bucket: 0 dst bucket: 0
2021/12/02 03:01:03 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'sys_test_bucket_op' in metakv",
  "warnings": null
 }
}
2021/12/02 03:01:08 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/02 03:01:13 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/02 03:01:19 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/02 03:01:24 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/02 03:01:24 clusterSetup.go:354: App: bucket_op_on_update got deployed
2021/12/02 03:01:29 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/02 03:01:34 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/02 03:01:39 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/02 03:01:39 clusterSetup.go:354: App: sys_test_bucket_op got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 03:01:39 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 03:01:39 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 03:01:39 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/02 03:01:39 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/02 03:01:44 Rebalance progress: 12.5
2021/12/02 03:01:49 Rebalance progress: 12.5
2021/12/02 03:01:54 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 03:01:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 03:01:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 03:01:54 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/02 03:01:55 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/02 03:02:00 Rebalance progress: 12.5
2021/12/02 03:02:05 Rebalance progress: 12.5
2021/12/02 03:02:10 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 03:02:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 03:02:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 03:02:10 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/02 03:02:15 Rebalance progress: 12.5
2021/12/02 03:02:20 Rebalance progress: 12.5
2021/12/02 03:02:25 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 03:02:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 03:02:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 03:02:25 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 03:02:30 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 03:02:35 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 03:02:40 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 03:02:45 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 03:02:50 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 03:02:55 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 03:02:55 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/02 03:02:55 eventingSetup.go:46: No eventing-con process running
2021/12/02 03:02:55 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/02 03:02:59 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 03:03:04 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 03:03:09 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7640
2021/12/02 03:03:09 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 03:03:10 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 03:03:15 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 03:03:20 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3627
2021/12/02 03:03:25 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3627
2021/12/02 03:03:25 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 03:03:26 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 03:03:31 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/02 03:03:36 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/02 03:03:41 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/02 03:03:46 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/02 03:03:51 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/02 03:03:56 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/02 03:03:56 clusterSetup.go:405: App: sys_test_bucket_op got un-deployed
2021/12/02 03:03:56 eventingSetup.go:46: No eventing-con process running
2021/12/02 03:03:56 eventingSetup.go:418: Function: sys_test_bucket_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: sys_test_bucket_op deleting in the background"
}
2021/12/02 03:03:58 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 03:04:03 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 03:04:03 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 03:04:04 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 03:04:09 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 03:04:09 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVSwapRebalanceWithMultipleHandlers (230.48s)
=== RUN   TestKVRebStopStartKVOpsOnUpdateBucketOpOneByOne
2021/12/02 03:04:19 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 03:04:19 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/02 03:04:19 eventingSetup.go:46: No eventing-con process running
2021/12/02 03:04:19 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/02 03:04:21 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 03:04:26 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 03:04:26 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 03:04:28 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 03:04:33 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 03:04:33 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 03:04:38 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 03:04:38 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/02 03:04:39 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/02 03:04:44 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/02 03:04:49 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/02 03:04:54 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/02 03:04:59 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/02 03:04:59 clusterSetup.go:354: App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 03:04:59 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 03:04:59 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 03:04:59 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/02 03:04:59 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/02 03:05:19 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 03:05:20 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 03:05:20 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 03:05:30 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 03:05:30 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 03:05:30 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/02 03:05:35 Rebalance progress: 12.5
2021/12/02 03:05:40 Rebalance progress: 12.5
2021/12/02 03:05:45 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 03:05:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 03:05:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 03:05:45 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/02 03:05:45 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/02 03:05:50 Rebalance progress: 12.5
2021/12/02 03:05:55 Rebalance progress: 12.5
2021/12/02 03:06:00 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 03:06:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 03:06:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 03:06:00 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/02 03:06:06 Rebalance progress: 12.5
2021/12/02 03:06:11 Rebalance progress: 12.5
2021/12/02 03:06:16 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 03:06:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 03:06:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 03:06:16 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 03:06:21 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 03:06:26 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 03:06:31 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 03:06:36 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 03:06:41 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 03:06:46 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 03:06:46 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/02 03:06:46 eventingSetup.go:46: No eventing-con process running
2021/12/02 03:06:46 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/02 03:06:48 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 03:06:53 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 03:06:58 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 9121
2021/12/02 03:06:58 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 03:07:00 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 03:07:05 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 03:07:05 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebStopStartKVOpsOnUpdateBucketOpOneByOne (175.11s)
=== RUN   TestKVFailoverOnUpdateBucketOp
2021/12/02 03:07:15 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 03:07:15 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/02 03:07:15 eventingSetup.go:46: No eventing-con process running
2021/12/02 03:07:15 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/02 03:07:16 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 03:07:21 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 03:07:21 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 03:07:23 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 03:07:28 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 03:07:28 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 03:07:33 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 03:07:33 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/02 03:07:34 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/02 03:07:39 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/02 03:07:44 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/02 03:07:49 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/02 03:07:54 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/02 03:07:54 clusterSetup.go:354: App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 03:07:54 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 03:07:54 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 03:07:54 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/02 03:07:55 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/02 03:08:00 Rebalance progress: 12.5
2021/12/02 03:08:05 Rebalance progress: 12.5
2021/12/02 03:08:10 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 03:08:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 03:08:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 03:08:10 clusterSetup.go:229: Failing over: [127.0.0.1:19001]
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 03:08:20 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 03:08:20 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 03:08:20 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/02 03:08:25 Rebalance progress: 12.5
2021/12/02 03:08:30 Rebalance progress: 12.5
2021/12/02 03:08:35 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 03:08:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 03:08:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 03:08:35 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/02 03:08:35 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/02 03:08:40 Rebalance progress: 12.5
2021/12/02 03:08:45 Rebalance progress: 12.5
2021/12/02 03:08:50 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 03:08:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 03:08:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 03:08:50 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/02 03:08:55 Rebalance progress: 12.5
2021/12/02 03:09:00 Rebalance progress: 12.5
2021/12/02 03:09:05 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 03:09:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 03:09:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 03:09:05 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 03:09:10 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 03:09:15 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 03:09:21 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 03:09:26 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 03:09:31 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 03:09:36 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 03:09:36 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/02 03:09:36 eventingSetup.go:46: No eventing-con process running
2021/12/02 03:09:36 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/02 03:09:38 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 03:09:43 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 03:09:48 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 8592
2021/12/02 03:09:48 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 03:09:49 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 03:09:54 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 03:09:59 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 8592
2021/12/02 03:09:59 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVFailoverOnUpdateBucketOp (174.71s)
=== RUN   TestBootstrapAfterKVHardFailover
2021/12/02 03:10:04 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/02 03:10:05 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/02 03:10:10 Rebalance progress: 100
2021/12/02 03:10:10 Failing over: [127.0.0.1:19001]
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 03:10:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 03:10:40 Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/02 03:10:41 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/02 03:10:46 Removing node(s): [] from the cluster
2021/12/02 03:10:46 Starting bucket ops to source bucket
2021/12/02 03:10:51 Rebalance failed. See logs for detailed reason. You can try again.
2021/12/02 03:10:56 Waiting for app: bucket_op_on_update to get deployed
2021/12/02 03:11:01 Waiting for app: bucket_op_on_update to get deployed
2021/12/02 03:11:01 App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 03:11:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 03:11:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 03:11:01 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 03:11:06 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 03:11:11 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 03:11:16 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 03:11:21 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 03:11:26 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 03:11:31 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 03:11:31 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/02 03:11:31 eventingSetup.go:46: No eventing-con process running
2021/12/02 03:11:32 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/02 03:11:33 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 03:11:38 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 03:11:43 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1507
2021/12/02 03:11:48 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1507
2021/12/02 03:11:48 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 03:11:50 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 03:11:55 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 03:12:00 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 313
2021/12/02 03:12:05 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 313
2021/12/02 03:12:05 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestBootstrapAfterKVHardFailover (125.84s)
PASS
ok  	github.com/couchbase/eventing/tests/functional_tests	12185.316s

Rebalance tests took 3 hours, 23 minutes and 7 seconds.

Testing Succeeded


Version: versions-01.12.2021-19.27.cfg
Build Log: make-01.12.2021-19.27.log
Server Log: logs-01.12.2021-19.27.tar.gz

Finished