Building

Started building at 2021/12/14 02:46:14
Calculating base
Updating mirror
Basing run on 7.1.0-1903 075dfde62
Updating tree for run 14.12.2021-02.46
gocb is at 635fa6c, changes since last good build: none
libcouchbase is at f6822fc, changes since last good build: none
indexing is at 888c51e, changes since last good build: none
kv_engine is at 2dc5ee9, changes since last good build: 
 2dc5ee9c4 MB-49858: BloomFilter tidy up
 1114e8a7b MB-49702: Track backfill runtime and disk bytes read
ns_server is at 6e2fba0, changes since last good build: 
 6e2fba06a formatting
 d036a4815 MB-49788: Add ellipsis to bucket names in failover/rebalance popup
couchstore is at 5aa7c4c, changes since last good build: none
couchdb is at fca2e6b, changes since last good build: none
testrunner is at df47214, changes since last good build: 
 df4721468 CBQE-7253: Add maximum version flags for test suites
Switching eventing to unstable
eventing is at 9fce322, changes since last good build: none
Switching eventing-ee to unstable
eventing-ee is at 3ae49c5, changes since last good build: none
Pulling in uncommitted change 160397 at refs/changes/97/160397/3
Total 23 (delta 15), reused 17 (delta 15)
[unstable 932b3295] MB-100 : Do not fail n1ql timeout test if we are catching timeout exceptions successfully
 Author: abhijpes 
 Date: Mon Aug 30 08:24:38 2021 +0530
 1 file changed, 2 insertions(+), 2 deletions(-)
Pulling in uncommitted change 163449 at refs/changes/49/163449/1
Total 48 (delta 27), reused 39 (delta 27)
[unstable 9ffb7bc8] MB-100: collect data when test fails
 Author: AnkitPrabhu 
 Date: Tue Oct 12 14:02:31 2021 +0530
 1 file changed, 2 insertions(+)
Pulling in uncommitted change 165976 at refs/changes/76/165976/17
Total 4 (delta 3), reused 4 (delta 3)
[unstable b1154baa] MB-48538: Add new audit ids for different events
 Author: AnkitPrabhu 
 Date: Tue Nov 23 14:29:10 2021 +0530
 1 file changed, 337 insertions(+), 61 deletions(-)
Pulling in uncommitted change 166697 at refs/changes/97/166697/4
Total 10 (delta 9), reused 10 (delta 9)
[unstable 159603e3] MB-49547: Check bucket instance to get kv node for dcp request
 Author: AnkitPrabhu 
 Date: Thu Dec 2 12:22:17 2021 +0530
 6 files changed, 38 insertions(+), 109 deletions(-)
Pulling in uncommitted change 166698 at refs/changes/98/166698/3
Total 4 (delta 3), reused 4 (delta 3)
[unstable 295a531b] MB-48334 : Fail rebalance unless all functions deployed on current encryption level
 Author: abhijpes 
 Date: Wed Dec 1 14:51:29 2021 +0530
 1 file changed, 9 insertions(+)
Pulling in uncommitted change 166812 at refs/changes/12/166812/5
Total 24 (delta 20), reused 24 (delta 20)
[unstable aa25c318] MB-49547: Refresh bucket info from streaming endpoint instance
 Author: AnkitPrabhu 
 Date: Thu Dec 2 13:55:55 2021 +0530
 7 files changed, 147 insertions(+), 235 deletions(-)
Pulling in uncommitted change 166999 at refs/changes/99/166999/5
Total 56 (delta 47), reused 56 (delta 47)
[unstable 7307dcf0] MB-46304: Add functionality to disable eventing exposed javascript features
 Author: AnkitPrabhu 
 Date: Mon Dec 6 13:26:12 2021 +0530
 20 files changed, 217 insertions(+), 30 deletions(-)
Pulling in uncommitted change 167196 at refs/changes/96/167196/6
Total 67 (delta 57), reused 67 (delta 57)
[unstable 373da81d] MB-49720: Store failover log in checkpoint blob
 Author: AnkitPrabhu 
 Date: Wed Dec 8 15:16:22 2021 +0530
 5 files changed, 147 insertions(+), 72 deletions(-)
Pulling in uncommitted change 167183 at refs/changes/83/167183/2
Total 5 (delta 2), reused 4 (delta 2)
[unstable 391d751] MB-46304: Throw runtime exception when curl feature is disabled
 Author: AnkitPrabhu 
 Date: Wed Dec 8 10:39:19 2021 +0530
 1 file changed, 9 insertions(+), 3 deletions(-)
Building cmakefiles and deps
Building main product

Testing

Started testing at 2021/12/14 03:03:37
Testing mode: rebalance,n1ql,functional
Starting server
Starting server: attempt 1

Functional tests

2021/12/14 03:05:06 Started functional
2021/12/14 03:05:14 Adding node: https://127.0.0.1:19001 with role: kv,index,n1ql to the cluster
2021/12/14 03:05:20 Error in adding nodes : , response: {"otpNode":"n_1@127.0.0.1"}
2021/12/14 03:05:20 Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/14 03:05:25 Removing node(s): [] from the cluster
2021/12/14 03:05:30 Rebalance progress: 0
2021/12/14 03:05:35 Rebalance progress: 0
2021/12/14 03:05:40 Rebalance progress: 4.539207175925926
2021/12/14 03:05:45 Rebalance progress: 9.168836805555555
2021/12/14 03:05:50 Rebalance progress: 12.51446759259259
2021/12/14 03:05:55 Rebalance progress: 17.10792824074074
2021/12/14 03:06:00 Rebalance progress: 20.52589699074074
2021/12/14 03:06:05 Rebalance progress: 25.11935763888889
2021/12/14 03:06:10 Rebalance progress: 100
2021/12/14 03:06:13 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/14 03:06:28 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/14 03:06:43 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/14 03:06:58 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/14 03:07:13 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/14 03:07:28 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/14 03:07:43 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/14 03:07:58 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/14 03:08:13 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/14 03:08:28 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/14 03:08:43 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/14 03:08:58 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/14 03:09:13 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/14 03:09:28 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/14 03:09:43 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/14 03:09:58 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
=== RUN   TestAdvancedGetOps
2021/12/14 03:10:16 Deploying app: TestAdvancedGetOps. Item count src bucket: 0 dst bucket: 0
2021/12/14 03:10:17 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedGetOps' in metakv",
  "warnings": null
 }
}
2021/12/14 03:10:22 Waiting for app: TestAdvancedGetOps to get deployed
2021/12/14 03:10:27 Waiting for app: TestAdvancedGetOps to get deployed
2021/12/14 03:10:32 Waiting for app: TestAdvancedGetOps to get deployed
2021/12/14 03:10:37 Waiting for app: TestAdvancedGetOps to get deployed
2021/12/14 03:10:37 App: TestAdvancedGetOps got deployed
2021/12/14 03:10:37 Starting bucket ops to source bucket
2021/12/14 03:10:42 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/14 03:10:42 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/14 03:10:42 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 03:10:42 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/14 03:10:42 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 03:10:47 Waiting for app: TestAdvancedGetOps to get un-deployed
2021/12/14 03:10:52 Waiting for app: TestAdvancedGetOps to get un-deployed
2021/12/14 03:10:57 Waiting for app: TestAdvancedGetOps to get un-deployed
2021/12/14 03:11:02 Waiting for app: TestAdvancedGetOps to get un-deployed
2021/12/14 03:11:02 App: TestAdvancedGetOps got un-deployed
2021/12/14 03:11:02 No eventing-con process running
2021/12/14 03:11:02 Function: TestAdvancedGetOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedGetOps deleting in the background"
}
2021/12/14 03:11:04 Context: Bucket flush, response code: 200 dump: 
2021/12/14 03:11:14 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/14 03:11:19 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/14 03:11:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 03:11:21 Context: Bucket flush, response code: 200 dump: 
2021/12/14 03:11:31 Waiting for dst bucket item count to get to: 0 curr count: 87
2021/12/14 03:11:36 Waiting for dst bucket item count to get to: 0 curr count: 87
2021/12/14 03:11:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedGetOps (80.15s)
=== RUN   TestAdvancedInsertOps
2021/12/14 03:11:36 Deploying app: TestAdvancedInsertOps. Item count src bucket: 0 dst bucket: 0
2021/12/14 03:11:37 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedInsertOps' in metakv",
  "warnings": null
 }
}
2021/12/14 03:11:42 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/14 03:11:47 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/14 03:11:52 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/14 03:11:57 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/14 03:11:57 App: TestAdvancedInsertOps got deployed
2021/12/14 03:11:57 Starting bucket ops to source bucket
2021/12/14 03:12:03 Waiting for dst bucket item count to get to: 1024 curr count: 0
2021/12/14 03:12:08 Waiting for dst bucket item count to get to: 1024 curr count: 653
2021/12/14 03:12:13 Waiting for dst bucket item count to get to: 1024 curr count: 653
2021/12/14 03:12:13 src & dst bucket item count matched up. src bucket count: 1024 dst bucket count: 1024
2021/12/14 03:12:13 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 03:12:13 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/14 03:12:13 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 03:12:18 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/14 03:12:23 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/14 03:12:28 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/14 03:12:33 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/14 03:12:33 App: TestAdvancedInsertOps got un-deployed
2021/12/14 03:12:33 No eventing-con process running
2021/12/14 03:12:33 Function: TestAdvancedInsertOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedInsertOps deleting in the background"
}
2021/12/14 03:12:35 Context: Bucket flush, response code: 200 dump: 
2021/12/14 03:12:45 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/14 03:12:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 03:12:46 Context: Bucket flush, response code: 200 dump: 
2021/12/14 03:12:56 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/14 03:12:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 03:12:56 Testing insert operation on source bucket
2021/12/14 03:12:56 Deploying app: TestAdvancedInsertOps. Item count src bucket: 0 dst bucket: 0
2021/12/14 03:12:58 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedInsertOps' in metakv",
  "warnings": null
 }
}
2021/12/14 03:13:03 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/14 03:13:08 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/14 03:13:13 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/14 03:13:18 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/14 03:13:18 App: TestAdvancedInsertOps got deployed
2021/12/14 03:13:18 Starting bucket ops to source bucket
2021/12/14 03:13:24 Waiting for dst bucket item count to get to: 2048 curr count: 0
2021/12/14 03:13:29 Waiting for dst bucket item count to get to: 2048 curr count: 1024
2021/12/14 03:13:34 Waiting for dst bucket item count to get to: 2048 curr count: 1024
2021/12/14 03:13:34 src & dst bucket item count matched up. src bucket count: 2048 dst bucket count: 2048
2021/12/14 03:13:34 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 03:13:34 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/14 03:13:34 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 03:13:39 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/14 03:13:44 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/14 03:13:49 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/14 03:13:54 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/14 03:13:54 App: TestAdvancedInsertOps got un-deployed
2021/12/14 03:13:54 No eventing-con process running
2021/12/14 03:13:54 Function: TestAdvancedInsertOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedInsertOps deleting in the background"
}
2021/12/14 03:13:56 Context: Bucket flush, response code: 200 dump: 
2021/12/14 03:14:06 Waiting for dst bucket item count to get to: 0 curr count: 2048
2021/12/14 03:14:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 03:14:07 Context: Bucket flush, response code: 200 dump: 
2021/12/14 03:14:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedInsertOps (156.55s)
=== RUN   TestAdvancedUpsertOps
2021/12/14 03:14:12 Starting bucket ops to source bucket
2021/12/14 03:14:13 Deploying app: TestAdvancedUpsertOps. Item count src bucket: 0 dst bucket: 0
2021/12/14 03:14:14 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedUpsertOps' in metakv",
  "warnings": null
 }
}
2021/12/14 03:14:19 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/14 03:14:24 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/14 03:14:29 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/14 03:14:34 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/14 03:14:34 App: TestAdvancedUpsertOps got deployed
2021/12/14 03:14:34 Starting bucket ops to source bucket
2021/12/14 03:14:40 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/14 03:14:45 Waiting for dst bucket item count to get to: 0 curr count: 512
2021/12/14 03:14:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 03:14:45 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 03:14:45 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/14 03:14:45 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 03:14:50 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/14 03:14:55 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/14 03:15:00 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/14 03:15:05 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/14 03:15:05 App: TestAdvancedUpsertOps got un-deployed
2021/12/14 03:15:05 No eventing-con process running
2021/12/14 03:15:05 Function: TestAdvancedUpsertOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedUpsertOps deleting in the background"
}
2021/12/14 03:15:07 Context: Bucket flush, response code: 200 dump: 
2021/12/14 03:15:17 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/14 03:15:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 03:15:19 Context: Bucket flush, response code: 200 dump: 
2021/12/14 03:15:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 03:15:24 Testing upsert operation on source bucket
2021/12/14 03:15:24 Deploying app: TestAdvancedUpsertOps. Item count src bucket: 0 dst bucket: 0
2021/12/14 03:15:25 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedUpsertOps' in metakv",
  "warnings": null
 }
}
2021/12/14 03:15:30 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/14 03:15:35 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/14 03:15:40 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/14 03:15:45 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/14 03:15:45 App: TestAdvancedUpsertOps got deployed
2021/12/14 03:15:45 Starting bucket ops to source bucket
2021/12/14 03:15:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 03:15:46 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 03:15:46 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/14 03:15:46 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 03:15:51 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/14 03:15:56 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/14 03:16:01 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/14 03:16:06 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/14 03:16:06 App: TestAdvancedUpsertOps got un-deployed
2021/12/14 03:16:06 No eventing-con process running
2021/12/14 03:16:06 Function: TestAdvancedUpsertOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedUpsertOps deleting in the background"
}
2021/12/14 03:16:08 Context: Bucket flush, response code: 200 dump: 
2021/12/14 03:16:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 03:16:15 Context: Bucket flush, response code: 200 dump: 
2021/12/14 03:16:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedUpsertOps (128.00s)
=== RUN   TestAdvancedReplaceOps
2021/12/14 03:16:20 Starting bucket ops to source bucket
2021/12/14 03:16:21 Deploying app: TestAdvancedReplaceOps. Item count src bucket: 0 dst bucket: 0
2021/12/14 03:16:22 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedReplaceOps' in metakv",
  "warnings": null
 }
}
2021/12/14 03:16:27 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/14 03:16:32 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/14 03:16:37 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/14 03:16:42 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/14 03:16:42 App: TestAdvancedReplaceOps got deployed
2021/12/14 03:16:42 Starting bucket ops to source bucket
2021/12/14 03:16:49 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/14 03:16:54 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/14 03:16:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 03:16:54 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 03:16:54 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/14 03:16:54 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 03:16:59 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/14 03:17:04 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/14 03:17:09 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/14 03:17:14 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/14 03:17:14 App: TestAdvancedReplaceOps got un-deployed
2021/12/14 03:17:14 No eventing-con process running
2021/12/14 03:17:14 Function: TestAdvancedReplaceOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedReplaceOps deleting in the background"
}
2021/12/14 03:17:15 Context: Bucket flush, response code: 200 dump: 
2021/12/14 03:17:25 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/14 03:17:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 03:17:27 Context: Bucket flush, response code: 200 dump: 
2021/12/14 03:17:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 03:17:32 Testing upsert operation on source bucket
2021/12/14 03:17:32 Deploying app: TestAdvancedReplaceOps. Item count src bucket: 0 dst bucket: 0
2021/12/14 03:17:33 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedReplaceOps' in metakv",
  "warnings": null
 }
}
2021/12/14 03:17:38 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/14 03:17:43 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/14 03:17:48 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/14 03:17:53 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/14 03:17:53 App: TestAdvancedReplaceOps got deployed
2021/12/14 03:17:53 Starting bucket ops to source bucket
2021/12/14 03:17:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 03:17:54 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 03:17:54 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/14 03:17:55 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 03:18:00 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/14 03:18:05 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/14 03:18:10 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/14 03:18:15 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/14 03:18:15 App: TestAdvancedReplaceOps got un-deployed
2021/12/14 03:18:15 No eventing-con process running
2021/12/14 03:18:15 Function: TestAdvancedReplaceOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedReplaceOps deleting in the background"
}
2021/12/14 03:18:16 Context: Bucket flush, response code: 200 dump: 
2021/12/14 03:18:26 Waiting for dst bucket item count to get to: 0 curr count: 357
2021/12/14 03:18:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 03:18:28 Context: Bucket flush, response code: 200 dump: 
2021/12/14 03:18:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedReplaceOps (132.81s)
=== RUN   TestAdvancedDeleteOps
2021/12/14 03:18:33 Starting bucket ops to source bucket
2021/12/14 03:18:33 Deploying app: TestAdvancedDeleteOps. Item count src bucket: 0 dst bucket: 0
2021/12/14 03:18:34 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedDeleteOps' in metakv",
  "warnings": null
 }
}
2021/12/14 03:18:39 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/14 03:18:44 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/14 03:18:49 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/14 03:18:54 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/14 03:18:54 App: TestAdvancedDeleteOps got deployed
2021/12/14 03:18:54 Starting bucket ops to source bucket
2021/12/14 03:19:00 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/14 03:19:05 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/14 03:19:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 03:19:05 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 03:19:05 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/14 03:19:05 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 03:19:10 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/14 03:19:15 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/14 03:19:20 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/14 03:19:25 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/14 03:19:25 App: TestAdvancedDeleteOps got un-deployed
2021/12/14 03:19:25 No eventing-con process running
2021/12/14 03:19:25 Function: TestAdvancedDeleteOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedDeleteOps deleting in the background"
}
2021/12/14 03:19:27 Context: Bucket flush, response code: 200 dump: 
2021/12/14 03:19:37 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/14 03:19:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 03:19:38 Context: Bucket flush, response code: 200 dump: 
2021/12/14 03:19:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 03:19:43 Testing delete operation on source bucket
2021/12/14 03:19:43 Deploying app: TestAdvancedDeleteOps. Item count src bucket: 0 dst bucket: 0
2021/12/14 03:19:44 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedDeleteOps' in metakv",
  "warnings": null
 }
}
2021/12/14 03:19:49 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/14 03:19:55 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/14 03:20:00 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/14 03:20:05 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/14 03:20:05 App: TestAdvancedDeleteOps got deployed
2021/12/14 03:20:05 Starting bucket ops to source bucket
2021/12/14 03:20:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 03:20:05 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 03:20:05 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/14 03:20:05 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 03:20:10 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/14 03:20:15 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/14 03:20:20 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/14 03:20:25 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/14 03:20:25 App: TestAdvancedDeleteOps got un-deployed
2021/12/14 03:20:25 No eventing-con process running
2021/12/14 03:20:25 Function: TestAdvancedDeleteOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedDeleteOps deleting in the background"
}
2021/12/14 03:20:27 Context: Bucket flush, response code: 200 dump: 
2021/12/14 03:20:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 03:20:34 Context: Bucket flush, response code: 200 dump: 
2021/12/14 03:20:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedDeleteOps (125.83s)
=== RUN   TestEnoentAdvancedGet
2021/12/14 03:20:39 Deploying app: TestEnoentAdvancedGet. Item count src bucket: 0 dst bucket: 0
2021/12/14 03:20:40 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedGet' in metakv",
  "warnings": null
 }
}
2021/12/14 03:20:45 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/14 03:20:50 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/14 03:20:55 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/14 03:21:00 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/14 03:21:00 App: TestEnoentAdvancedGet got deployed
2021/12/14 03:21:00 Starting bucket ops to source bucket
2021/12/14 03:21:05 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/14 03:21:10 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/14 03:21:10 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/14 03:21:10 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 03:21:10 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/14 03:21:10 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 03:21:15 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/14 03:21:20 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/14 03:21:25 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/14 03:21:30 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/14 03:21:30 App: TestEnoentAdvancedGet got un-deployed
2021/12/14 03:21:30 No eventing-con process running
2021/12/14 03:21:30 Function: TestEnoentAdvancedGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedGet deleting in the background"
}
2021/12/14 03:21:32 Context: Bucket flush, response code: 200 dump: 
2021/12/14 03:21:42 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/14 03:21:47 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/14 03:21:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 03:21:49 Context: Bucket flush, response code: 200 dump: 
2021/12/14 03:21:59 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/14 03:22:04 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/14 03:22:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 03:22:04 Testing get enoent operation on source bucket
2021/12/14 03:22:04 Deploying app: TestEnoentAdvancedGet. Item count src bucket: 0 dst bucket: 0
2021/12/14 03:22:05 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedGet' in metakv",
  "warnings": null
 }
}
2021/12/14 03:22:10 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/14 03:22:15 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/14 03:22:20 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/14 03:22:25 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/14 03:22:25 App: TestEnoentAdvancedGet got deployed
2021/12/14 03:22:25 Starting bucket ops to source bucket
2021/12/14 03:22:30 Waiting for dst bucket item count to get to: 200 curr count: 0
2021/12/14 03:22:30 src & dst bucket item count matched up. src bucket count: 200 dst bucket count: 200
2021/12/14 03:22:30 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 03:22:30 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/14 03:22:30 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 03:22:35 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/14 03:22:40 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/14 03:22:45 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/14 03:22:50 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/14 03:22:50 App: TestEnoentAdvancedGet got un-deployed
2021/12/14 03:22:50 No eventing-con process running
2021/12/14 03:22:50 Function: TestEnoentAdvancedGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedGet deleting in the background"
}
2021/12/14 03:22:52 Context: Bucket flush, response code: 200 dump: 
2021/12/14 03:23:02 Waiting for dst bucket item count to get to: 0 curr count: 200
2021/12/14 03:23:07 Waiting for dst bucket item count to get to: 0 curr count: 200
2021/12/14 03:23:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 03:23:09 Context: Bucket flush, response code: 200 dump: 
2021/12/14 03:23:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEnoentAdvancedGet (154.87s)
=== RUN   TestEnoentAdvancedReplace
2021/12/14 03:23:14 Deploying app: TestEnoentAdvancedReplace. Item count src bucket: 0 dst bucket: 0
2021/12/14 03:23:15 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedReplace' in metakv",
  "warnings": null
 }
}
2021/12/14 03:23:20 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/14 03:23:25 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/14 03:23:30 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/14 03:23:35 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/14 03:23:35 App: TestEnoentAdvancedReplace got deployed
2021/12/14 03:23:35 Starting bucket ops to source bucket
2021/12/14 03:23:40 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/14 03:23:45 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/14 03:23:45 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/14 03:23:45 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 03:23:45 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/14 03:23:45 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 03:23:50 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/14 03:23:55 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/14 03:24:00 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/14 03:24:05 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/14 03:24:05 App: TestEnoentAdvancedReplace got un-deployed
2021/12/14 03:24:05 No eventing-con process running
2021/12/14 03:24:05 Function: TestEnoentAdvancedReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedReplace deleting in the background"
}
2021/12/14 03:24:07 Context: Bucket flush, response code: 200 dump: 
2021/12/14 03:24:17 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/14 03:24:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 03:24:19 Context: Bucket flush, response code: 200 dump: 
2021/12/14 03:24:29 Waiting for dst bucket item count to get to: 0 curr count: 97
2021/12/14 03:24:34 Waiting for dst bucket item count to get to: 0 curr count: 47
2021/12/14 03:24:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 03:24:34 Testing get enoent operation on source bucket
2021/12/14 03:24:34 Deploying app: TestEnoentAdvancedReplace. Item count src bucket: 0 dst bucket: 0
2021/12/14 03:24:35 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedReplace' in metakv",
  "warnings": null
 }
}
2021/12/14 03:24:40 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/14 03:24:45 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/14 03:24:50 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/14 03:24:55 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/14 03:24:55 App: TestEnoentAdvancedReplace got deployed
2021/12/14 03:24:55 Starting bucket ops to source bucket
2021/12/14 03:25:00 Waiting for dst bucket item count to get to: 200 curr count: 0
2021/12/14 03:25:05 Waiting for dst bucket item count to get to: 200 curr count: 100
2021/12/14 03:25:05 src & dst bucket item count matched up. src bucket count: 200 dst bucket count: 200
2021/12/14 03:25:05 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 03:25:05 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/14 03:25:05 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 03:25:10 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/14 03:25:15 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/14 03:25:20 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/14 03:25:25 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/14 03:25:25 App: TestEnoentAdvancedReplace got un-deployed
2021/12/14 03:25:25 No eventing-con process running
2021/12/14 03:25:25 Function: TestEnoentAdvancedReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedReplace deleting in the background"
}
2021/12/14 03:25:27 Context: Bucket flush, response code: 200 dump: 
2021/12/14 03:25:37 Waiting for dst bucket item count to get to: 0 curr count: 200
2021/12/14 03:25:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 03:25:39 Context: Bucket flush, response code: 200 dump: 
2021/12/14 03:25:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEnoentAdvancedReplace (150.00s)
=== RUN   TestEnoentAdvancedDelete
2021/12/14 03:25:44 Deploying app: TestEnoentAdvancedDelete. Item count src bucket: 0 dst bucket: 0
2021/12/14 03:25:45 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedDelete' in metakv",
  "warnings": null
 }
}
2021/12/14 03:25:50 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/14 03:25:55 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/14 03:26:00 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/14 03:26:05 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/14 03:26:05 App: TestEnoentAdvancedDelete got deployed
2021/12/14 03:26:05 Starting bucket ops to source bucket
2021/12/14 03:26:10 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/14 03:26:15 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/14 03:26:15 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/14 03:26:15 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 03:26:15 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/14 03:26:15 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 03:26:20 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/14 03:26:25 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/14 03:26:30 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/14 03:26:35 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/14 03:26:35 App: TestEnoentAdvancedDelete got un-deployed
2021/12/14 03:26:35 No eventing-con process running
2021/12/14 03:26:35 Function: TestEnoentAdvancedDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedDelete deleting in the background"
}
2021/12/14 03:26:37 Context: Bucket flush, response code: 200 dump: 
2021/12/14 03:26:47 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/14 03:26:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 03:26:49 Context: Bucket flush, response code: 200 dump: 
2021/12/14 03:26:59 Waiting for dst bucket item count to get to: 0 curr count: 96
2021/12/14 03:27:04 Waiting for dst bucket item count to get to: 0 curr count: 46
2021/12/14 03:27:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 03:27:04 Testing get enoent operation on source bucket
2021/12/14 03:27:04 Deploying app: TestEnoentAdvancedDelete. Item count src bucket: 0 dst bucket: 0
2021/12/14 03:27:05 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedDelete' in metakv",
  "warnings": null
 }
}
2021/12/14 03:27:10 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/14 03:27:15 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/14 03:27:20 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/14 03:27:25 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/14 03:27:25 App: TestEnoentAdvancedDelete got deployed
2021/12/14 03:27:25 Starting bucket ops to source bucket
2021/12/14 03:27:30 Waiting for dst bucket item count to get to: 200 curr count: 0
2021/12/14 03:27:35 Waiting for dst bucket item count to get to: 200 curr count: 100
2021/12/14 03:27:35 src & dst bucket item count matched up. src bucket count: 200 dst bucket count: 200
2021/12/14 03:27:35 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 03:27:35 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/14 03:27:35 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 03:27:40 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/14 03:27:45 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/14 03:27:50 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/14 03:27:55 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/14 03:27:55 App: TestEnoentAdvancedDelete got un-deployed
2021/12/14 03:27:55 No eventing-con process running
2021/12/14 03:27:55 Function: TestEnoentAdvancedDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedDelete deleting in the background"
}
2021/12/14 03:27:57 Context: Bucket flush, response code: 200 dump: 
2021/12/14 03:28:07 Waiting for dst bucket item count to get to: 0 curr count: 200
2021/12/14 03:28:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 03:28:09 Context: Bucket flush, response code: 200 dump: 
2021/12/14 03:28:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEnoentAdvancedDelete (149.97s)
=== RUN   TestAdvancedInsertKeyExist
2021/12/14 03:28:14 Deploying app: TestAdvancedInsertKeyExist. Item count src bucket: 0 dst bucket: 0
2021/12/14 03:28:15 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedInsertKeyExist' in metakv",
  "warnings": null
 }
}
2021/12/14 03:28:20 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/14 03:28:25 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/14 03:28:30 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/14 03:28:35 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/14 03:28:35 App: TestAdvancedInsertKeyExist got deployed
2021/12/14 03:28:35 Starting bucket ops to source bucket
2021/12/14 03:28:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 03:28:35 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 03:28:35 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/14 03:28:35 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 03:28:40 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/14 03:28:45 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/14 03:28:50 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/14 03:28:55 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/14 03:28:55 App: TestAdvancedInsertKeyExist got un-deployed
2021/12/14 03:28:55 No eventing-con process running
2021/12/14 03:28:55 Function: TestAdvancedInsertKeyExist delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedInsertKeyExist deleting in the background"
}
2021/12/14 03:28:57 Context: Bucket flush, response code: 200 dump: 
2021/12/14 03:29:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 03:29:04 Context: Bucket flush, response code: 200 dump: 
2021/12/14 03:29:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 03:29:09 Pumping document in the destination bucket
2021/12/14 03:29:09 Starting bucket ops to source bucket
2021/12/14 03:29:10 Deploying app: TestAdvancedInsertKeyExist. Item count src bucket: 0 dst bucket: 0
2021/12/14 03:29:11 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedInsertKeyExist' in metakv",
  "warnings": null
 }
}
2021/12/14 03:29:16 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/14 03:29:21 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/14 03:29:26 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/14 03:29:31 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/14 03:29:31 App: TestAdvancedInsertKeyExist got deployed
2021/12/14 03:29:31 Starting bucket ops to source bucket
2021/12/14 03:29:36 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/14 03:29:41 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/14 03:29:41 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 03:29:41 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 03:29:41 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/14 03:29:41 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 03:29:46 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/14 03:29:51 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/14 03:29:56 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/14 03:30:01 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/14 03:30:01 App: TestAdvancedInsertKeyExist got un-deployed
2021/12/14 03:30:01 No eventing-con process running
2021/12/14 03:30:01 Function: TestAdvancedInsertKeyExist delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedInsertKeyExist deleting in the background"
}
2021/12/14 03:30:03 Context: Bucket flush, response code: 200 dump: 
2021/12/14 03:30:13 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/14 03:30:18 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/14 03:30:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 03:30:19 Context: Bucket flush, response code: 200 dump: 
2021/12/14 03:30:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedInsertKeyExist (130.69s)
=== RUN   TestExipryGet
2021/12/14 03:30:24 Deploying app: TestExipryGet. Item count src bucket: 0 dst bucket: 0
2021/12/14 03:30:26 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryGet' in metakv",
  "warnings": null
 }
}
2021/12/14 03:30:31 Waiting for app: TestExipryGet to get deployed
2021/12/14 03:30:36 Waiting for app: TestExipryGet to get deployed
2021/12/14 03:30:41 Waiting for app: TestExipryGet to get deployed
2021/12/14 03:30:46 Waiting for app: TestExipryGet to get deployed
2021/12/14 03:30:46 App: TestExipryGet got deployed
2021/12/14 03:30:46 Starting bucket ops to source bucket
2021/12/14 03:31:31 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/14 03:31:31 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 03:31:31 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 03:31:31 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/14 03:31:31 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 03:31:36 Waiting for app: TestExipryGet to get un-deployed
2021/12/14 03:31:41 Waiting for app: TestExipryGet to get un-deployed
2021/12/14 03:31:46 Waiting for app: TestExipryGet to get un-deployed
2021/12/14 03:31:51 Waiting for app: TestExipryGet to get un-deployed
2021/12/14 03:31:51 App: TestExipryGet got un-deployed
2021/12/14 03:31:51 No eventing-con process running
2021/12/14 03:31:51 Function: TestExipryGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryGet deleting in the background"
}
2021/12/14 03:31:53 Context: Bucket flush, response code: 200 dump: 
2021/12/14 03:31:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 03:32:00 Context: Bucket flush, response code: 200 dump: 
2021/12/14 03:32:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 03:32:05 Pumping document in the destination bucket for expiry get
2021/12/14 03:32:05 Starting bucket ops to source bucket
2021/12/14 03:32:05 Deploying app: TestExipryGet. Item count src bucket: 0 dst bucket: 0
2021/12/14 03:32:06 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryGet' in metakv",
  "warnings": null
 }
}
2021/12/14 03:32:11 Waiting for app: TestExipryGet to get deployed
2021/12/14 03:32:16 Waiting for app: TestExipryGet to get deployed
2021/12/14 03:32:21 Waiting for app: TestExipryGet to get deployed
2021/12/14 03:32:26 Waiting for app: TestExipryGet to get deployed
2021/12/14 03:32:26 App: TestExipryGet got deployed
2021/12/14 03:32:26 Starting bucket ops to source bucket
2021/12/14 03:33:12 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/14 03:33:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 03:33:12 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 03:33:12 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/14 03:33:12 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 03:33:17 Waiting for app: TestExipryGet to get un-deployed
2021/12/14 03:33:22 Waiting for app: TestExipryGet to get un-deployed
2021/12/14 03:33:27 Waiting for app: TestExipryGet to get un-deployed
2021/12/14 03:33:32 Waiting for app: TestExipryGet to get un-deployed
2021/12/14 03:33:32 App: TestExipryGet got un-deployed
2021/12/14 03:33:32 No eventing-con process running
2021/12/14 03:33:32 Function: TestExipryGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryGet deleting in the background"
}
2021/12/14 03:33:33 Context: Bucket flush, response code: 200 dump: 
2021/12/14 03:33:43 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/14 03:33:48 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/14 03:33:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 03:33:50 Context: Bucket flush, response code: 200 dump: 
2021/12/14 03:33:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestExipryGet (210.73s)
=== RUN   TestExipryInsert
2021/12/14 03:33:55 Deploying app: TestExipryInsert. Item count src bucket: 0 dst bucket: 0
2021/12/14 03:33:56 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryInsert' in metakv",
  "warnings": null
 }
}
2021/12/14 03:34:01 Waiting for app: TestExipryInsert to get deployed
2021/12/14 03:34:06 Waiting for app: TestExipryInsert to get deployed
2021/12/14 03:34:11 Waiting for app: TestExipryInsert to get deployed
2021/12/14 03:34:17 Waiting for app: TestExipryInsert to get deployed
2021/12/14 03:34:17 App: TestExipryInsert got deployed
2021/12/14 03:34:17 Starting bucket ops to source bucket
2021/12/14 03:35:02 Waiting for dst bucket item count to get to: 0 curr count: 200
2021/12/14 03:35:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 03:35:02 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 03:35:02 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/14 03:35:02 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 03:35:07 Waiting for app: TestExipryInsert to get un-deployed
2021/12/14 03:35:12 Waiting for app: TestExipryInsert to get un-deployed
2021/12/14 03:35:17 Waiting for app: TestExipryInsert to get un-deployed
2021/12/14 03:35:22 Waiting for app: TestExipryInsert to get un-deployed
2021/12/14 03:35:22 App: TestExipryInsert got un-deployed
2021/12/14 03:35:22 No eventing-con process running
2021/12/14 03:35:22 Function: TestExipryInsert delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryInsert deleting in the background"
}
2021/12/14 03:35:24 Context: Bucket flush, response code: 200 dump: 
2021/12/14 03:35:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 03:35:31 Context: Bucket flush, response code: 200 dump: 
2021/12/14 03:35:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 03:35:36 Testing on the destination bucket for expiry insert
2021/12/14 03:35:36 Deploying app: TestExipryInsert. Item count src bucket: 0 dst bucket: 0
2021/12/14 03:35:37 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryInsert' in metakv",
  "warnings": null
 }
}
2021/12/14 03:35:42 Waiting for app: TestExipryInsert to get deployed
2021/12/14 03:35:47 Waiting for app: TestExipryInsert to get deployed
2021/12/14 03:35:52 Waiting for app: TestExipryInsert to get deployed
2021/12/14 03:35:57 Waiting for app: TestExipryInsert to get deployed
2021/12/14 03:35:57 App: TestExipryInsert got deployed
2021/12/14 03:35:57 Starting bucket ops to source bucket
2021/12/14 03:36:42 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/14 03:36:47 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/14 03:36:52 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/14 03:36:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 03:36:52 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 03:36:52 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/14 03:36:52 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 03:36:57 Waiting for app: TestExipryInsert to get un-deployed
2021/12/14 03:37:02 Waiting for app: TestExipryInsert to get un-deployed
2021/12/14 03:37:07 Waiting for app: TestExipryInsert to get un-deployed
2021/12/14 03:37:12 Waiting for app: TestExipryInsert to get un-deployed
2021/12/14 03:37:12 App: TestExipryInsert got un-deployed
2021/12/14 03:37:12 No eventing-con process running
2021/12/14 03:37:13 Function: TestExipryInsert delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryInsert deleting in the background"
}
2021/12/14 03:37:14 Context: Bucket flush, response code: 200 dump: 
2021/12/14 03:37:24 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/14 03:37:29 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/14 03:37:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 03:37:31 Context: Bucket flush, response code: 200 dump: 
2021/12/14 03:37:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestExipryInsert (220.79s)
=== RUN   TestExipryReplace
2021/12/14 03:37:36 Deploying app: TestExipryReplace. Item count src bucket: 0 dst bucket: 0
2021/12/14 03:37:37 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryReplace' in metakv",
  "warnings": null
 }
}
2021/12/14 03:37:42 Waiting for app: TestExipryReplace to get deployed
2021/12/14 03:37:47 Waiting for app: TestExipryReplace to get deployed
2021/12/14 03:37:52 Waiting for app: TestExipryReplace to get deployed
2021/12/14 03:37:57 Waiting for app: TestExipryReplace to get deployed
2021/12/14 03:37:57 App: TestExipryReplace got deployed
2021/12/14 03:37:57 Starting bucket ops to source bucket
2021/12/14 03:38:43 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/14 03:38:48 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/14 03:38:53 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/14 03:38:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 03:38:53 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 03:38:53 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/14 03:38:53 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 03:38:58 Waiting for app: TestExipryReplace to get un-deployed
2021/12/14 03:39:03 Waiting for app: TestExipryReplace to get un-deployed
2021/12/14 03:39:08 Waiting for app: TestExipryReplace to get un-deployed
2021/12/14 03:39:13 Waiting for app: TestExipryReplace to get un-deployed
2021/12/14 03:39:13 App: TestExipryReplace got un-deployed
2021/12/14 03:39:13 No eventing-con process running
2021/12/14 03:39:13 Function: TestExipryReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryReplace deleting in the background"
}
2021/12/14 03:39:14 Context: Bucket flush, response code: 200 dump: 
2021/12/14 03:39:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 03:39:22 Context: Bucket flush, response code: 200 dump: 
2021/12/14 03:39:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 03:39:27 Testing on the destination bucket for expiry insert
2021/12/14 03:39:27 Deploying app: TestExipryReplace. Item count src bucket: 0 dst bucket: 0
2021/12/14 03:39:28 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryReplace' in metakv",
  "warnings": null
 }
}
2021/12/14 03:39:33 Waiting for app: TestExipryReplace to get deployed
2021/12/14 03:39:38 Waiting for app: TestExipryReplace to get deployed
2021/12/14 03:39:43 Waiting for app: TestExipryReplace to get deployed
2021/12/14 03:39:48 Waiting for app: TestExipryReplace to get deployed
2021/12/14 03:39:48 App: TestExipryReplace got deployed
2021/12/14 03:39:48 Starting bucket ops to source bucket
2021/12/14 03:40:33 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/14 03:40:38 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/14 03:40:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 03:40:38 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 03:40:38 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/14 03:40:38 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 03:40:43 Waiting for app: TestExipryReplace to get un-deployed
2021/12/14 03:40:48 Waiting for app: TestExipryReplace to get un-deployed
2021/12/14 03:40:53 Waiting for app: TestExipryReplace to get un-deployed
2021/12/14 03:40:58 Waiting for app: TestExipryReplace to get un-deployed
2021/12/14 03:40:58 App: TestExipryReplace got un-deployed
2021/12/14 03:40:58 No eventing-con process running
2021/12/14 03:40:58 Function: TestExipryReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryReplace deleting in the background"
}
2021/12/14 03:41:00 Context: Bucket flush, response code: 200 dump: 
2021/12/14 03:41:10 Waiting for dst bucket item count to get to: 0 curr count: 54
2021/12/14 03:41:15 Waiting for dst bucket item count to get to: 0 curr count: 54
2021/12/14 03:41:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 03:41:17 Context: Bucket flush, response code: 200 dump: 
2021/12/14 03:41:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestExipryReplace (225.66s)
=== RUN   TestExipryUpsert
2021/12/14 03:41:22 Deploying app: TestExipryUpsert. Item count src bucket: 0 dst bucket: 0
2021/12/14 03:41:23 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryUpsert' in metakv",
  "warnings": null
 }
}
2021/12/14 03:41:28 Waiting for app: TestExipryUpsert to get deployed
2021/12/14 03:41:33 Waiting for app: TestExipryUpsert to get deployed
2021/12/14 03:41:38 Waiting for app: TestExipryUpsert to get deployed
2021/12/14 03:41:43 Waiting for app: TestExipryUpsert to get deployed
2021/12/14 03:41:43 App: TestExipryUpsert got deployed
2021/12/14 03:41:43 Starting bucket ops to source bucket
2021/12/14 03:42:28 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/14 03:42:33 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/14 03:42:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 03:42:33 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 03:42:33 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/14 03:42:33 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 03:42:38 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/14 03:42:43 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/14 03:42:48 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/14 03:42:53 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/14 03:42:53 App: TestExipryUpsert got un-deployed
2021/12/14 03:42:53 No eventing-con process running
2021/12/14 03:42:53 Function: TestExipryUpsert delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryUpsert deleting in the background"
}
2021/12/14 03:42:55 Context: Bucket flush, response code: 200 dump: 
2021/12/14 03:43:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 03:43:02 Context: Bucket flush, response code: 200 dump: 
2021/12/14 03:43:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 03:43:07 Testing on the destination bucket for expiry insert
2021/12/14 03:43:07 Deploying app: TestExipryUpsert. Item count src bucket: 0 dst bucket: 0
2021/12/14 03:43:08 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryUpsert' in metakv",
  "warnings": null
 }
}
2021/12/14 03:43:13 Waiting for app: TestExipryUpsert to get deployed
2021/12/14 03:43:18 Waiting for app: TestExipryUpsert to get deployed
2021/12/14 03:43:23 Waiting for app: TestExipryUpsert to get deployed
2021/12/14 03:43:28 Waiting for app: TestExipryUpsert to get deployed
2021/12/14 03:43:28 App: TestExipryUpsert got deployed
2021/12/14 03:43:28 Starting bucket ops to source bucket
2021/12/14 03:44:14 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/14 03:44:19 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/14 03:44:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 03:44:19 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 03:44:19 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/14 03:44:19 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 03:44:24 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/14 03:44:29 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/14 03:44:34 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/14 03:44:39 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/14 03:44:39 App: TestExipryUpsert got un-deployed
2021/12/14 03:44:39 No eventing-con process running
2021/12/14 03:44:39 Function: TestExipryUpsert delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryUpsert deleting in the background"
}
2021/12/14 03:44:41 Context: Bucket flush, response code: 200 dump: 
2021/12/14 03:44:51 Waiting for dst bucket item count to get to: 0 curr count: 84
2021/12/14 03:44:56 Waiting for dst bucket item count to get to: 0 curr count: 84
2021/12/14 03:44:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 03:44:57 Context: Bucket flush, response code: 200 dump: 
2021/12/14 03:45:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestExipryUpsert (220.69s)
=== RUN   TestCasReplace
2021/12/14 03:45:02 Deploying app: TestCasReplace. Item count src bucket: 0 dst bucket: 0
2021/12/14 03:45:04 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCasReplace' in metakv",
  "warnings": null
 }
}
2021/12/14 03:45:09 Waiting for app: TestCasReplace to get deployed
2021/12/14 03:45:14 Waiting for app: TestCasReplace to get deployed
2021/12/14 03:45:19 Waiting for app: TestCasReplace to get deployed
2021/12/14 03:45:24 Waiting for app: TestCasReplace to get deployed
2021/12/14 03:45:24 App: TestCasReplace got deployed
2021/12/14 03:45:24 Starting bucket ops to source bucket
2021/12/14 03:45:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 03:45:24 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 03:45:24 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/14 03:45:24 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 03:45:29 Waiting for app: TestCasReplace to get un-deployed
2021/12/14 03:45:34 Waiting for app: TestCasReplace to get un-deployed
2021/12/14 03:45:39 Waiting for app: TestCasReplace to get un-deployed
2021/12/14 03:45:44 Waiting for app: TestCasReplace to get un-deployed
2021/12/14 03:45:44 App: TestCasReplace got un-deployed
2021/12/14 03:45:44 No eventing-con process running
2021/12/14 03:45:44 Function: TestCasReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCasReplace deleting in the background"
}
2021/12/14 03:45:46 Context: Bucket flush, response code: 200 dump: 
2021/12/14 03:45:56 Waiting for dst bucket item count to get to: 0 curr count: 7
2021/12/14 03:45:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 03:45:57 Context: Bucket flush, response code: 200 dump: 
2021/12/14 03:46:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 03:46:02 Testing on the destination bucket for upsert with cas
2021/12/14 03:46:02 Starting bucket ops to source bucket
2021/12/14 03:46:03 Deploying app: TestCasReplace. Item count src bucket: 0 dst bucket: 0
2021/12/14 03:46:04 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCasReplace' in metakv",
  "warnings": null
 }
}
2021/12/14 03:46:09 Waiting for app: TestCasReplace to get deployed
2021/12/14 03:46:14 Waiting for app: TestCasReplace to get deployed
2021/12/14 03:46:19 Waiting for app: TestCasReplace to get deployed
2021/12/14 03:46:24 Waiting for app: TestCasReplace to get deployed
2021/12/14 03:46:24 App: TestCasReplace got deployed
2021/12/14 03:46:24 Starting bucket ops to source bucket
2021/12/14 03:46:29 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/14 03:46:34 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/14 03:46:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 03:46:34 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 03:46:34 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/14 03:46:34 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 03:46:39 Waiting for app: TestCasReplace to get un-deployed
2021/12/14 03:46:44 Waiting for app: TestCasReplace to get un-deployed
2021/12/14 03:46:49 Waiting for app: TestCasReplace to get un-deployed
2021/12/14 03:46:54 Waiting for app: TestCasReplace to get un-deployed
2021/12/14 03:46:54 App: TestCasReplace got un-deployed
2021/12/14 03:46:54 No eventing-con process running
2021/12/14 03:46:54 Function: TestCasReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCasReplace deleting in the background"
}
2021/12/14 03:46:56 Context: Bucket flush, response code: 200 dump: 
2021/12/14 03:47:06 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/14 03:47:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 03:47:07 Context: Bucket flush, response code: 200 dump: 
2021/12/14 03:47:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCasReplace (130.13s)
=== RUN   TestCasDelete
2021/12/14 03:47:12 Deploying app: TestCasDelete. Item count src bucket: 0 dst bucket: 0
2021/12/14 03:47:14 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCasDelete' in metakv",
  "warnings": null
 }
}
2021/12/14 03:47:19 Waiting for app: TestCasDelete to get deployed
2021/12/14 03:47:24 Waiting for app: TestCasDelete to get deployed
2021/12/14 03:47:29 Waiting for app: TestCasDelete to get deployed
2021/12/14 03:47:34 Waiting for app: TestCasDelete to get deployed
2021/12/14 03:47:34 App: TestCasDelete got deployed
2021/12/14 03:47:34 Starting bucket ops to source bucket
2021/12/14 03:47:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 03:47:34 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 03:47:34 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/14 03:47:34 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 03:47:39 Waiting for app: TestCasDelete to get un-deployed
2021/12/14 03:47:44 Waiting for app: TestCasDelete to get un-deployed
2021/12/14 03:47:49 Waiting for app: TestCasDelete to get un-deployed
2021/12/14 03:47:54 Waiting for app: TestCasDelete to get un-deployed
2021/12/14 03:47:54 App: TestCasDelete got un-deployed
2021/12/14 03:47:54 No eventing-con process running
2021/12/14 03:47:54 Function: TestCasDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCasDelete deleting in the background"
}
2021/12/14 03:47:56 Context: Bucket flush, response code: 200 dump: 
2021/12/14 03:48:06 Waiting for dst bucket item count to get to: 0 curr count: 4
2021/12/14 03:48:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 03:48:08 Context: Bucket flush, response code: 200 dump: 
2021/12/14 03:48:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 03:48:13 Testing on the destination bucket for delete with cas
2021/12/14 03:48:13 Starting bucket ops to source bucket
2021/12/14 03:48:13 Deploying app: TestCasDelete. Item count src bucket: 0 dst bucket: 0
2021/12/14 03:48:14 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCasDelete' in metakv",
  "warnings": null
 }
}
2021/12/14 03:48:19 Waiting for app: TestCasDelete to get deployed
2021/12/14 03:48:24 Waiting for app: TestCasDelete to get deployed
2021/12/14 03:48:29 Waiting for app: TestCasDelete to get deployed
2021/12/14 03:48:34 Waiting for app: TestCasDelete to get deployed
2021/12/14 03:48:34 App: TestCasDelete got deployed
2021/12/14 03:48:34 Starting bucket ops to source bucket
2021/12/14 03:48:39 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/14 03:48:44 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/14 03:48:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 03:48:44 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 03:48:44 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/14 03:48:44 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 03:48:49 Waiting for app: TestCasDelete to get un-deployed
2021/12/14 03:48:54 Waiting for app: TestCasDelete to get un-deployed
2021/12/14 03:48:59 Waiting for app: TestCasDelete to get un-deployed
2021/12/14 03:49:04 Waiting for app: TestCasDelete to get un-deployed
2021/12/14 03:49:04 App: TestCasDelete got un-deployed
2021/12/14 03:49:04 No eventing-con process running
2021/12/14 03:49:04 Function: TestCasDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCasDelete deleting in the background"
}
2021/12/14 03:49:06 Context: Bucket flush, response code: 200 dump: 
2021/12/14 03:49:16 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/14 03:49:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 03:49:18 Context: Bucket flush, response code: 200 dump: 
2021/12/14 03:49:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCasDelete (130.11s)
=== RUN   TestCountersIncrement
2021/12/14 03:49:23 Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/14 03:49:28 Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/14 03:49:29 Removing node(s): [] from the cluster
2021/12/14 03:49:34 Rebalance progress: 100
2021/12/14 03:49:34 Deploying app: TestCountersIncrement. Item count src bucket: 0 dst bucket: 0
2021/12/14 03:49:35 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCountersIncrement' in metakv",
  "warnings": null
 }
}
2021/12/14 03:49:40 Waiting for app: TestCountersIncrement to get deployed
2021/12/14 03:49:45 Waiting for app: TestCountersIncrement to get deployed
2021/12/14 03:49:50 Waiting for app: TestCountersIncrement to get deployed
2021/12/14 03:49:55 Waiting for app: TestCountersIncrement to get deployed
2021/12/14 03:49:55 App: TestCountersIncrement got deployed
2021/12/14 03:49:55 Starting bucket ops to source bucket
2021/12/14 03:50:01 Waiting for dst bucket item count to get to: 1024 curr count: 0
2021/12/14 03:50:06 Waiting for dst bucket item count to get to: 1024 curr count: 512
2021/12/14 03:50:06 src & dst bucket item count matched up. src bucket count: 1024 dst bucket count: 1024
2021/12/14 03:50:06 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 03:50:06 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 03:50:11 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/14 03:50:16 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/14 03:50:21 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/14 03:50:26 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/14 03:50:26 App: TestCountersIncrement got un-deployed
2021/12/14 03:50:26 No eventing-con process running
2021/12/14 03:50:26 Function: TestCountersIncrement delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCountersIncrement deleting in the background"
}
2021/12/14 03:50:28 Context: Bucket flush, response code: 200 dump: 
2021/12/14 03:50:38 Waiting for dst bucket item count to get to: 0 curr count: 681
2021/12/14 03:50:43 Waiting for dst bucket item count to get to: 0 curr count: 168
2021/12/14 03:50:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 03:50:45 Context: Bucket flush, response code: 200 dump: 
2021/12/14 03:50:55 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/14 03:50:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 03:50:55 Deploying app: TestCountersIncrement. Item count src bucket: 0 dst bucket: 0
2021/12/14 03:50:56 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCountersIncrement' in metakv",
  "warnings": null
 }
}
2021/12/14 03:51:01 Waiting for app: TestCountersIncrement to get deployed
2021/12/14 03:51:06 Waiting for app: TestCountersIncrement to get deployed
2021/12/14 03:51:11 Waiting for app: TestCountersIncrement to get deployed
2021/12/14 03:51:16 Waiting for app: TestCountersIncrement to get deployed
2021/12/14 03:51:16 App: TestCountersIncrement got deployed
2021/12/14 03:51:16 Starting bucket ops to source bucket
2021/12/14 03:51:22 Waiting for dst bucket item count to get to: 2048 curr count: 0
2021/12/14 03:51:22 src & dst bucket item count matched up. src bucket count: 2048 dst bucket count: 2048
2021/12/14 03:51:22 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 03:51:22 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 03:51:27 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/14 03:51:32 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/14 03:51:37 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/14 03:51:42 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/14 03:51:42 App: TestCountersIncrement got un-deployed
2021/12/14 03:51:42 No eventing-con process running
2021/12/14 03:51:42 Function: TestCountersIncrement delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCountersIncrement deleting in the background"
}
2021/12/14 03:51:44 Context: Bucket flush, response code: 200 dump: 
2021/12/14 03:51:54 Waiting for dst bucket item count to get to: 0 curr count: 2048
2021/12/14 03:51:59 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/14 03:51:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 03:52:01 Context: Bucket flush, response code: 200 dump: 
2021/12/14 03:52:11 Waiting for dst bucket item count to get to: 0 curr count: 1
2021/12/14 03:52:16 Waiting for dst bucket item count to get to: 0 curr count: 1
2021/12/14 03:52:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 03:52:16 Removing node(s): [http://127.0.0.1:19002 http://127.0.0.1:19003] from the cluster
2021/12/14 03:52:21 Rebalance progress: 100
--- PASS: TestCountersIncrement (178.34s)
=== RUN   TestCountersDecrement
2021/12/14 03:52:21 Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/14 03:52:21 Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/14 03:52:22 Removing node(s): [] from the cluster
2021/12/14 03:52:27 Rebalance progress: 100
2021/12/14 03:52:27 Deploying app: TestCountersDecrement. Item count src bucket: 0 dst bucket: 0
2021/12/14 03:52:28 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCountersDecrement' in metakv",
  "warnings": null
 }
}
2021/12/14 03:52:33 Waiting for app: TestCountersDecrement to get deployed
2021/12/14 03:52:38 Waiting for app: TestCountersDecrement to get deployed
2021/12/14 03:52:43 Waiting for app: TestCountersDecrement to get deployed
2021/12/14 03:52:48 Waiting for app: TestCountersDecrement to get deployed
2021/12/14 03:52:48 App: TestCountersDecrement got deployed
2021/12/14 03:52:48 Starting bucket ops to source bucket
2021/12/14 03:52:54 Waiting for dst bucket item count to get to: 1024 curr count: 0
2021/12/14 03:52:59 Waiting for dst bucket item count to get to: 1024 curr count: 512
2021/12/14 03:53:04 Waiting for dst bucket item count to get to: 1024 curr count: 512
2021/12/14 03:53:04 src & dst bucket item count matched up. src bucket count: 1024 dst bucket count: 1024
2021/12/14 03:53:04 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 03:53:04 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 03:53:09 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/14 03:53:14 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/14 03:53:19 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/14 03:53:24 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/14 03:53:24 App: TestCountersDecrement got un-deployed
2021/12/14 03:53:24 No eventing-con process running
2021/12/14 03:53:24 Function: TestCountersDecrement delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCountersDecrement deleting in the background"
}
2021/12/14 03:53:26 Context: Bucket flush, response code: 200 dump: 
2021/12/14 03:53:36 Waiting for dst bucket item count to get to: 0 curr count: 1025
2021/12/14 03:53:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 03:53:38 Context: Bucket flush, response code: 200 dump: 
2021/12/14 03:53:48 Waiting for dst bucket item count to get to: 0 curr count: 663
2021/12/14 03:53:53 Waiting for dst bucket item count to get to: 0 curr count: 151
2021/12/14 03:53:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 03:53:53 Deploying app: TestCountersDecrement. Item count src bucket: 0 dst bucket: 0
2021/12/14 03:53:54 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCountersDecrement' in metakv",
  "warnings": null
 }
}
2021/12/14 03:53:59 Waiting for app: TestCountersDecrement to get deployed
2021/12/14 03:54:04 Waiting for app: TestCountersDecrement to get deployed
2021/12/14 03:54:09 Waiting for app: TestCountersDecrement to get deployed
2021/12/14 03:54:14 Waiting for app: TestCountersDecrement to get deployed
2021/12/14 03:54:14 App: TestCountersDecrement got deployed
2021/12/14 03:54:14 Starting bucket ops to source bucket
2021/12/14 03:54:20 Waiting for dst bucket item count to get to: 2048 curr count: 0
2021/12/14 03:54:25 Waiting for dst bucket item count to get to: 2048 curr count: 1024
2021/12/14 03:54:25 src & dst bucket item count matched up. src bucket count: 2048 dst bucket count: 2048
2021/12/14 03:54:25 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 03:54:25 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 03:54:30 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/14 03:54:35 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/14 03:54:40 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/14 03:54:45 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/14 03:54:45 App: TestCountersDecrement got un-deployed
2021/12/14 03:54:45 No eventing-con process running
2021/12/14 03:54:46 Function: TestCountersDecrement delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCountersDecrement deleting in the background"
}
2021/12/14 03:54:47 Context: Bucket flush, response code: 200 dump: 
2021/12/14 03:54:57 Waiting for dst bucket item count to get to: 0 curr count: 2048
2021/12/14 03:54:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 03:54:59 Context: Bucket flush, response code: 200 dump: 
2021/12/14 03:55:09 Waiting for dst bucket item count to get to: 0 curr count: 1
2021/12/14 03:55:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 03:55:09 Removing node(s): [http://127.0.0.1:19002 http://127.0.0.1:19003] from the cluster
2021/12/14 03:55:14 Rebalance progress: 100
--- PASS: TestCountersDecrement (173.15s)
=== RUN   TestCacheGet
2021/12/14 03:55:14 Deploying app: TestCacheGet. Item count src bucket: 0 dst bucket: 0
2021/12/14 03:55:15 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCacheGet' in metakv",
  "warnings": null
 }
}
2021/12/14 03:55:20 Waiting for app: TestCacheGet to get deployed
2021/12/14 03:55:25 Waiting for app: TestCacheGet to get deployed
2021/12/14 03:55:30 Waiting for app: TestCacheGet to get deployed
2021/12/14 03:55:35 Waiting for app: TestCacheGet to get deployed
2021/12/14 03:55:35 App: TestCacheGet got deployed
2021/12/14 03:55:35 Starting bucket ops to source bucket
2021/12/14 03:55:41 Waiting for dst bucket item count to get to: 20 curr count: 0
2021/12/14 03:55:46 Waiting for dst bucket item count to get to: 20 curr count: 0
2021/12/14 03:55:51 Waiting for dst bucket item count to get to: 20 curr count: 0
2021/12/14 03:55:56 Waiting for dst bucket item count to get to: 20 curr count: 12
2021/12/14 03:56:01 Waiting for dst bucket item count to get to: 20 curr count: 12
2021/12/14 03:56:01 src & dst bucket item count matched up. src bucket count: 20 dst bucket count: 20
2021/12/14 03:56:01 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 03:56:01 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 03:56:06 Waiting for app: TestCacheGet to get un-deployed
2021/12/14 03:56:11 Waiting for app: TestCacheGet to get un-deployed
2021/12/14 03:56:16 Waiting for app: TestCacheGet to get un-deployed
2021/12/14 03:56:21 Waiting for app: TestCacheGet to get un-deployed
2021/12/14 03:56:21 App: TestCacheGet got un-deployed
2021/12/14 03:56:21 No eventing-con process running
2021/12/14 03:56:21 Function: TestCacheGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCacheGet deleting in the background"
}
2021/12/14 03:56:23 Context: Bucket flush, response code: 200 dump: 
2021/12/14 03:56:33 Waiting for dst bucket item count to get to: 0 curr count: 20
2021/12/14 03:56:38 Waiting for dst bucket item count to get to: 0 curr count: 10
2021/12/14 03:56:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 03:56:39 Context: Bucket flush, response code: 200 dump: 
2021/12/14 03:56:49 Waiting for dst bucket item count to get to: 0 curr count: 10
2021/12/14 03:56:54 Waiting for dst bucket item count to get to: 0 curr count: 10
2021/12/14 03:56:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCacheGet (100.29s)
=== RUN   TestCacheRYOW
2021/12/14 03:56:54 Deploying app: TestCacheRYOW. Item count src bucket: 0 dst bucket: 0
2021/12/14 03:56:56 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCacheRYOW' in metakv",
  "warnings": null
 }
}
2021/12/14 03:57:01 Waiting for app: TestCacheRYOW to get deployed
2021/12/14 03:57:06 Waiting for app: TestCacheRYOW to get deployed
2021/12/14 03:57:11 Waiting for app: TestCacheRYOW to get deployed
2021/12/14 03:57:16 Waiting for app: TestCacheRYOW to get deployed
2021/12/14 03:57:16 App: TestCacheRYOW got deployed
2021/12/14 03:57:16 Starting bucket ops to source bucket
2021/12/14 03:57:21 Waiting for dst bucket item count to get to: 50 curr count: 0
2021/12/14 03:57:21 src & dst bucket item count matched up. src bucket count: 50 dst bucket count: 50
2021/12/14 03:57:21 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 03:57:21 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 03:57:26 Waiting for app: TestCacheRYOW to get un-deployed
2021/12/14 03:57:31 Waiting for app: TestCacheRYOW to get un-deployed
2021/12/14 03:57:36 Waiting for app: TestCacheRYOW to get un-deployed
2021/12/14 03:57:41 Waiting for app: TestCacheRYOW to get un-deployed
2021/12/14 03:57:41 App: TestCacheRYOW got un-deployed
2021/12/14 03:57:41 No eventing-con process running
2021/12/14 03:57:41 Function: TestCacheRYOW delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCacheRYOW deleting in the background"
}
2021/12/14 03:57:43 Context: Bucket flush, response code: 200 dump: 
2021/12/14 03:57:53 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/14 03:57:58 Waiting for dst bucket item count to get to: 0 curr count: 25
2021/12/14 03:57:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 03:58:00 Context: Bucket flush, response code: 200 dump: 
2021/12/14 03:58:10 Waiting for dst bucket item count to get to: 0 curr count: 25
2021/12/14 03:58:15 Waiting for dst bucket item count to get to: 0 curr count: 25
2021/12/14 03:58:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCacheRYOW (80.24s)
=== RUN   TestCacheExpiry
2021/12/14 03:58:15 Deploying app: TestCacheExpiry. Item count src bucket: 0 dst bucket: 0
2021/12/14 03:58:16 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCacheExpiry' in metakv",
  "warnings": null
 }
}
2021/12/14 03:58:21 Waiting for app: TestCacheExpiry to get deployed
2021/12/14 03:58:26 Waiting for app: TestCacheExpiry to get deployed
2021/12/14 03:58:31 Waiting for app: TestCacheExpiry to get deployed
2021/12/14 03:58:36 Waiting for app: TestCacheExpiry to get deployed
2021/12/14 03:58:36 App: TestCacheExpiry got deployed
2021/12/14 03:58:36 Starting bucket ops to source bucket
2021/12/14 03:58:41 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/14 03:58:46 Waiting for dst bucket item count to get to: 100 curr count: 19
2021/12/14 03:58:51 Waiting for dst bucket item count to get to: 100 curr count: 19
2021/12/14 03:58:56 Waiting for dst bucket item count to get to: 100 curr count: 94
2021/12/14 03:59:01 Waiting for dst bucket item count to get to: 100 curr count: 94
2021/12/14 03:59:01 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/14 03:59:01 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 03:59:01 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 03:59:06 Waiting for app: TestCacheExpiry to get un-deployed
2021/12/14 03:59:11 Waiting for app: TestCacheExpiry to get un-deployed
2021/12/14 03:59:16 Waiting for app: TestCacheExpiry to get un-deployed
2021/12/14 03:59:21 Waiting for app: TestCacheExpiry to get un-deployed
2021/12/14 03:59:21 App: TestCacheExpiry got un-deployed
2021/12/14 03:59:21 No eventing-con process running
2021/12/14 03:59:21 Function: TestCacheExpiry delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCacheExpiry deleting in the background"
}
2021/12/14 03:59:23 Context: Bucket flush, response code: 200 dump: 
2021/12/14 03:59:33 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/14 03:59:38 Waiting for dst bucket item count to get to: 0 curr count: 25
2021/12/14 03:59:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 03:59:40 Context: Bucket flush, response code: 200 dump: 
2021/12/14 03:59:50 Waiting for dst bucket item count to get to: 0 curr count: 51
2021/12/14 03:59:55 Waiting for dst bucket item count to get to: 0 curr count: 51
2021/12/14 03:59:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCacheExpiry (100.22s)
=== RUN   TestCacheExpiryCustom
2021/12/14 03:59:55 Deploying app: TestCacheExpiryCustom. Item count src bucket: 0 dst bucket: 0
2021/12/14 03:59:56 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCacheExpiryCustom' in metakv",
  "warnings": null
 }
}
2021/12/14 04:00:01 Waiting for app: TestCacheExpiryCustom to get deployed
2021/12/14 04:00:06 Waiting for app: TestCacheExpiryCustom to get deployed
2021/12/14 04:00:11 Waiting for app: TestCacheExpiryCustom to get deployed
2021/12/14 04:00:16 Waiting for app: TestCacheExpiryCustom to get deployed
2021/12/14 04:00:16 App: TestCacheExpiryCustom got deployed
2021/12/14 04:00:16 Starting bucket ops to source bucket
2021/12/14 04:00:21 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/14 04:00:26 Waiting for dst bucket item count to get to: 100 curr count: 19
2021/12/14 04:00:31 Waiting for dst bucket item count to get to: 100 curr count: 19
2021/12/14 04:00:36 Waiting for dst bucket item count to get to: 100 curr count: 94
2021/12/14 04:00:41 Waiting for dst bucket item count to get to: 100 curr count: 94
2021/12/14 04:00:41 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/14 04:00:41 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 04:00:41 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 04:00:46 Waiting for app: TestCacheExpiryCustom to get un-deployed
2021/12/14 04:00:51 Waiting for app: TestCacheExpiryCustom to get un-deployed
2021/12/14 04:00:56 Waiting for app: TestCacheExpiryCustom to get un-deployed
2021/12/14 04:01:01 Waiting for app: TestCacheExpiryCustom to get un-deployed
2021/12/14 04:01:01 App: TestCacheExpiryCustom got un-deployed
2021/12/14 04:01:01 No eventing-con process running
2021/12/14 04:01:01 Function: TestCacheExpiryCustom delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCacheExpiryCustom deleting in the background"
}
2021/12/14 04:01:03 Context: Bucket flush, response code: 200 dump: 
2021/12/14 04:01:13 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/14 04:01:18 Waiting for dst bucket item count to get to: 0 curr count: 25
2021/12/14 04:01:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 04:01:20 Context: Bucket flush, response code: 200 dump: 
2021/12/14 04:01:30 Waiting for dst bucket item count to get to: 0 curr count: 58
2021/12/14 04:01:35 Waiting for dst bucket item count to get to: 0 curr count: 58
2021/12/14 04:01:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCacheExpiryCustom (100.22s)
=== RUN   TestCacheOverflowCustom
2021/12/14 04:01:35 Deploying app: TestCacheOverflowCustom. Item count src bucket: 0 dst bucket: 0
2021/12/14 04:01:36 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCacheOverflowCustom' in metakv",
  "warnings": null
 }
}
2021/12/14 04:01:41 Waiting for app: TestCacheOverflowCustom to get deployed
2021/12/14 04:01:46 Waiting for app: TestCacheOverflowCustom to get deployed
2021/12/14 04:01:51 Waiting for app: TestCacheOverflowCustom to get deployed
2021/12/14 04:01:56 Waiting for app: TestCacheOverflowCustom to get deployed
2021/12/14 04:01:56 App: TestCacheOverflowCustom got deployed
2021/12/14 04:01:56 Starting bucket ops to source bucket
2021/12/14 04:02:01 Waiting for dst bucket item count to get to: 35 curr count: 0
2021/12/14 04:02:06 Waiting for dst bucket item count to get to: 35 curr count: 10
2021/12/14 04:02:11 Waiting for dst bucket item count to get to: 35 curr count: 10
2021/12/14 04:02:16 Waiting for dst bucket item count to get to: 35 curr count: 27
2021/12/14 04:02:21 Waiting for dst bucket item count to get to: 35 curr count: 27
2021/12/14 04:02:21 src & dst bucket item count matched up. src bucket count: 35 dst bucket count: 35
2021/12/14 04:02:21 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 04:02:22 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 04:02:27 Waiting for app: TestCacheOverflowCustom to get un-deployed
2021/12/14 04:02:32 Waiting for app: TestCacheOverflowCustom to get un-deployed
2021/12/14 04:02:37 Waiting for app: TestCacheOverflowCustom to get un-deployed
2021/12/14 04:02:42 Waiting for app: TestCacheOverflowCustom to get un-deployed
2021/12/14 04:02:42 App: TestCacheOverflowCustom got un-deployed
2021/12/14 04:02:42 No eventing-con process running
2021/12/14 04:02:42 Function: TestCacheOverflowCustom delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCacheOverflowCustom deleting in the background"
}
2021/12/14 04:02:43 Context: Bucket flush, response code: 200 dump: 
2021/12/14 04:02:53 Waiting for dst bucket item count to get to: 0 curr count: 5
2021/12/14 04:02:58 Waiting for dst bucket item count to get to: 0 curr count: 3
2021/12/14 04:02:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 04:03:00 Context: Bucket flush, response code: 200 dump: 
2021/12/14 04:03:10 Waiting for dst bucket item count to get to: 0 curr count: 26
2021/12/14 04:03:15 Waiting for dst bucket item count to get to: 0 curr count: 26
2021/12/14 04:03:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCacheOverflowCustom (100.28s)
=== RUN   TestStrictMode
2021/12/14 04:03:15 Deploying app: TestStrictMode. Item count src bucket: 0 dst bucket: 0
2021/12/14 04:03:16 Context: Post to main store, response code: 422 dump: {
 "name": "ERR_HANDLER_COMPILATION",
 "code": 27,
 "description": "Function compilation failed",
 "attributes": null,
 "runtime_info": {
  "code": 27,
  "info": {
   "area": "handlerCode",
   "column_number": 63,
   "compile_success": false,
   "description": "        destination[meta.id] = 00000000000000000000000000000101;\n                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nSyntaxError: Octal literals are not allowed in strict mode.\n",
   "index": 132,
   "language": "JavaScript",
   "line_number": 4
  }
 }
}
--- PASS: TestStrictMode (1.06s)
=== RUN   TestDataTypes
2021/12/14 04:03:16 Deploying app: TestDataTypes. Item count src bucket: 0 dst bucket: 0
2021/12/14 04:03:18 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDataTypes' in metakv",
  "warnings": null
 }
}
2021/12/14 04:03:23 Waiting for app: TestDataTypes to get deployed
2021/12/14 04:03:28 Waiting for app: TestDataTypes to get deployed
2021/12/14 04:03:33 Waiting for app: TestDataTypes to get deployed
2021/12/14 04:03:38 Waiting for app: TestDataTypes to get deployed
2021/12/14 04:03:38 App: TestDataTypes got deployed
2021/12/14 04:03:38 Starting bucket ops to source bucket
2021/12/14 04:03:43 Waiting for dst bucket item count to get to: 18 curr count: 0
2021/12/14 04:03:48 Waiting for dst bucket item count to get to: 18 curr count: 10
2021/12/14 04:03:53 Waiting for dst bucket item count to get to: 18 curr count: 10
2021/12/14 04:03:53 src & dst bucket item count matched up. src bucket count: 18 dst bucket count: 18
2021/12/14 04:03:53 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 04:03:53 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 04:03:58 Waiting for app: TestDataTypes to get un-deployed
2021/12/14 04:04:03 Waiting for app: TestDataTypes to get un-deployed
2021/12/14 04:04:08 Waiting for app: TestDataTypes to get un-deployed
2021/12/14 04:04:13 Waiting for app: TestDataTypes to get un-deployed
2021/12/14 04:04:13 App: TestDataTypes got un-deployed
2021/12/14 04:04:13 No eventing-con process running
2021/12/14 04:04:13 Function: TestDataTypes delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDataTypes deleting in the background"
}
2021/12/14 04:04:15 Context: Bucket flush, response code: 200 dump: 
2021/12/14 04:04:25 Waiting for dst bucket item count to get to: 0 curr count: 1
2021/12/14 04:04:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 04:04:26 Context: Bucket flush, response code: 200 dump: 
2021/12/14 04:04:36 Waiting for dst bucket item count to get to: 0 curr count: 18
2021/12/14 04:04:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDataTypes (80.12s)
=== RUN   TestEnoentGet
2021/12/14 04:04:37 Deploying app: TestEnoentGet. Item count src bucket: 0 dst bucket: 0
2021/12/14 04:04:38 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentGet' in metakv",
  "warnings": null
 }
}
2021/12/14 04:04:43 Waiting for app: TestEnoentGet to get deployed
2021/12/14 04:04:48 Waiting for app: TestEnoentGet to get deployed
2021/12/14 04:04:53 Waiting for app: TestEnoentGet to get deployed
2021/12/14 04:04:58 Waiting for app: TestEnoentGet to get deployed
2021/12/14 04:04:58 App: TestEnoentGet got deployed
2021/12/14 04:04:58 Starting bucket ops to source bucket
2021/12/14 04:05:03 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/14 04:05:08 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/14 04:05:13 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/14 04:05:13 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/14 04:05:13 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 04:05:13 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 04:05:18 Waiting for app: TestEnoentGet to get un-deployed
2021/12/14 04:05:23 Waiting for app: TestEnoentGet to get un-deployed
2021/12/14 04:05:28 Waiting for app: TestEnoentGet to get un-deployed
2021/12/14 04:05:33 Waiting for app: TestEnoentGet to get un-deployed
2021/12/14 04:05:33 App: TestEnoentGet got un-deployed
2021/12/14 04:05:33 No eventing-con process running
2021/12/14 04:05:33 Function: TestEnoentGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentGet deleting in the background"
}
2021/12/14 04:05:35 Context: Bucket flush, response code: 200 dump: 
2021/12/14 04:05:45 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/14 04:05:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 04:05:47 Context: Bucket flush, response code: 200 dump: 
2021/12/14 04:05:57 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/14 04:05:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 04:05:57 Deploying app: TestEnoentGet. Item count src bucket: 0 dst bucket: 0
2021/12/14 04:05:58 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentGet' in metakv",
  "warnings": null
 }
}
2021/12/14 04:06:03 Waiting for app: TestEnoentGet to get deployed
2021/12/14 04:06:08 Waiting for app: TestEnoentGet to get deployed
2021/12/14 04:06:13 Waiting for app: TestEnoentGet to get deployed
2021/12/14 04:06:18 Waiting for app: TestEnoentGet to get deployed
2021/12/14 04:06:18 App: TestEnoentGet got deployed
2021/12/14 04:06:18 Starting bucket ops to source bucket
2021/12/14 04:06:23 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/14 04:06:28 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/14 04:06:33 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/14 04:06:33 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/14 04:06:33 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 04:06:33 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 04:06:38 Waiting for app: TestEnoentGet to get un-deployed
2021/12/14 04:06:43 Waiting for app: TestEnoentGet to get un-deployed
2021/12/14 04:06:48 Waiting for app: TestEnoentGet to get un-deployed
2021/12/14 04:06:53 Waiting for app: TestEnoentGet to get un-deployed
2021/12/14 04:06:53 App: TestEnoentGet got un-deployed
2021/12/14 04:06:53 No eventing-con process running
2021/12/14 04:06:53 Function: TestEnoentGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentGet deleting in the background"
}
2021/12/14 04:06:55 Context: Bucket flush, response code: 200 dump: 
2021/12/14 04:07:05 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/14 04:07:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 04:07:07 Context: Bucket flush, response code: 200 dump: 
2021/12/14 04:07:17 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/14 04:07:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEnoentGet (160.27s)
=== RUN   TestEnoentDelete
2021/12/14 04:07:17 Deploying app: TestEnoentDelete. Item count src bucket: 0 dst bucket: 0
2021/12/14 04:07:18 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentDelete' in metakv",
  "warnings": null
 }
}
2021/12/14 04:07:23 Waiting for app: TestEnoentDelete to get deployed
2021/12/14 04:07:28 Waiting for app: TestEnoentDelete to get deployed
2021/12/14 04:07:33 Waiting for app: TestEnoentDelete to get deployed
2021/12/14 04:07:38 Waiting for app: TestEnoentDelete to get deployed
2021/12/14 04:07:38 App: TestEnoentDelete got deployed
2021/12/14 04:07:38 Starting bucket ops to source bucket
2021/12/14 04:07:43 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/14 04:07:48 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/14 04:07:53 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/14 04:07:53 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/14 04:07:53 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 04:07:53 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 04:07:58 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/14 04:08:03 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/14 04:08:08 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/14 04:08:13 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/14 04:08:13 App: TestEnoentDelete got un-deployed
2021/12/14 04:08:13 No eventing-con process running
2021/12/14 04:08:13 Function: TestEnoentDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentDelete deleting in the background"
}
2021/12/14 04:08:15 Context: Bucket flush, response code: 200 dump: 
2021/12/14 04:08:25 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/14 04:08:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 04:08:27 Context: Bucket flush, response code: 200 dump: 
2021/12/14 04:08:37 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/14 04:08:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 04:08:37 Deploying app: TestEnoentDelete. Item count src bucket: 0 dst bucket: 0
2021/12/14 04:08:38 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentDelete' in metakv",
  "warnings": null
 }
}
2021/12/14 04:08:43 Waiting for app: TestEnoentDelete to get deployed
2021/12/14 04:08:48 Waiting for app: TestEnoentDelete to get deployed
2021/12/14 04:08:53 Waiting for app: TestEnoentDelete to get deployed
2021/12/14 04:08:58 Waiting for app: TestEnoentDelete to get deployed
2021/12/14 04:08:58 App: TestEnoentDelete got deployed
2021/12/14 04:08:58 Starting bucket ops to source bucket
2021/12/14 04:09:03 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/14 04:09:08 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/14 04:09:08 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/14 04:09:08 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 04:09:09 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 04:09:14 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/14 04:09:19 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/14 04:09:24 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/14 04:09:29 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/14 04:09:29 App: TestEnoentDelete got un-deployed
2021/12/14 04:09:29 No eventing-con process running
2021/12/14 04:09:29 Function: TestEnoentDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentDelete deleting in the background"
}
2021/12/14 04:09:30 Context: Bucket flush, response code: 200 dump: 
2021/12/14 04:09:40 Waiting for dst bucket item count to get to: 0 curr count: 78
2021/12/14 04:09:45 Waiting for dst bucket item count to get to: 0 curr count: 78
2021/12/14 04:09:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 04:09:47 Context: Bucket flush, response code: 200 dump: 
2021/12/14 04:09:57 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/14 04:09:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEnoentDelete (160.41s)
=== RUN   TestError
2021/12/14 04:10:07 Waiting for app: TestError to get un-deployed
2021/12/14 04:10:07 App: TestError got un-deployed
2021/12/14 04:10:07 No eventing-con process running
2021/12/14 04:10:07 Function: TestError delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestError not found"
 }
}
2021/12/14 04:10:09 Context: Bucket flush, response code: 200 dump: 
2021/12/14 04:10:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 04:10:16 Context: Bucket flush, response code: 200 dump: 
2021/12/14 04:10:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 04:10:21 Deploying app: TestError. Item count src bucket: 0 dst bucket: 0
2021/12/14 04:10:22 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestError' in metakv",
  "warnings": null
 }
}
2021/12/14 04:10:27 Waiting for app: TestError to get deployed
2021/12/14 04:10:32 Waiting for app: TestError to get deployed
2021/12/14 04:10:37 Waiting for app: TestError to get deployed
2021/12/14 04:10:42 Waiting for app: TestError to get deployed
2021/12/14 04:10:42 App: TestError got deployed
2021/12/14 04:10:42 Starting bucket ops to source bucket
2021/12/14 04:10:47 Waiting for dst bucket item count to get to: 300 curr count: 0
2021/12/14 04:10:52 Waiting for dst bucket item count to get to: 300 curr count: 0
2021/12/14 04:10:52 src & dst bucket item count matched up. src bucket count: 300 dst bucket count: 300
2021/12/14 04:10:52 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 04:10:52 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 04:10:57 Waiting for app: TestError to get un-deployed
2021/12/14 04:11:02 Waiting for app: TestError to get un-deployed
2021/12/14 04:11:07 Waiting for app: TestError to get un-deployed
2021/12/14 04:11:12 Waiting for app: TestError to get un-deployed
2021/12/14 04:11:12 App: TestError got un-deployed
2021/12/14 04:11:12 No eventing-con process running
2021/12/14 04:11:12 Function: TestError delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestError deleting in the background"
}
2021/12/14 04:11:14 Context: Bucket flush, response code: 200 dump: 
2021/12/14 04:11:24 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/14 04:11:29 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/14 04:11:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 04:11:31 Context: Bucket flush, response code: 200 dump: 
2021/12/14 04:11:41 Waiting for dst bucket item count to get to: 0 curr count: 294
2021/12/14 04:11:46 Waiting for dst bucket item count to get to: 0 curr count: 294
2021/12/14 04:11:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestError (108.80s)
=== RUN   TestCRLF
2021/12/14 04:11:56 Waiting for app: TestCRLF to get un-deployed
2021/12/14 04:11:56 App: TestCRLF got un-deployed
2021/12/14 04:11:56 No eventing-con process running
2021/12/14 04:11:56 Function: TestCRLF delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestCRLF not found"
 }
}
2021/12/14 04:11:58 Context: Bucket flush, response code: 200 dump: 
2021/12/14 04:12:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 04:12:05 Context: Bucket flush, response code: 200 dump: 
2021/12/14 04:12:10 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 04:12:10 Deploying app: TestCRLF. Item count src bucket: 0 dst bucket: 0
2021/12/14 04:12:11 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCRLF' in metakv",
  "warnings": null
 }
}
2021/12/14 04:12:16 Waiting for app: TestCRLF to get deployed
2021/12/14 04:12:21 Waiting for app: TestCRLF to get deployed
2021/12/14 04:12:26 Waiting for app: TestCRLF to get deployed
2021/12/14 04:12:31 Waiting for app: TestCRLF to get deployed
2021/12/14 04:12:31 App: TestCRLF got deployed
2021/12/14 04:12:31 Starting bucket ops to source bucket
2021/12/14 04:12:36 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/14 04:12:41 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/14 04:12:41 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/14 04:12:41 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 04:12:41 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 04:12:46 Waiting for app: TestCRLF to get un-deployed
2021/12/14 04:12:51 Waiting for app: TestCRLF to get un-deployed
2021/12/14 04:12:56 Waiting for app: TestCRLF to get un-deployed
2021/12/14 04:13:01 Waiting for app: TestCRLF to get un-deployed
2021/12/14 04:13:01 App: TestCRLF got un-deployed
2021/12/14 04:13:01 No eventing-con process running
2021/12/14 04:13:01 Function: TestCRLF delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCRLF deleting in the background"
}
2021/12/14 04:13:03 Context: Bucket flush, response code: 200 dump: 
2021/12/14 04:13:13 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/14 04:13:18 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/14 04:13:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 04:13:20 Context: Bucket flush, response code: 200 dump: 
2021/12/14 04:13:30 Waiting for dst bucket item count to get to: 0 curr count: 60
2021/12/14 04:13:35 Waiting for dst bucket item count to get to: 0 curr count: 60
2021/12/14 04:13:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCRLF (108.81s)
=== RUN   TestImportExport
2021/12/14 04:13:45 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/14 04:13:45 App: TestImportExport_n1ql_insert_on_update got un-deployed
2021/12/14 04:13:45 No eventing-con process running
2021/12/14 04:13:45 Function: TestImportExport_n1ql_insert_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestImportExport_n1ql_insert_on_update not found"
 }
}
2021/12/14 04:13:47 Context: Bucket flush, response code: 200 dump: 
2021/12/14 04:13:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 04:13:53 Context: Bucket flush, response code: 200 dump: 
2021/12/14 04:13:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 04:14:03 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/14 04:14:03 App: TestImportExport_bucket_op_on_update got un-deployed
2021/12/14 04:14:03 No eventing-con process running
2021/12/14 04:14:03 Function: TestImportExport_bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestImportExport_bucket_op_on_update not found"
 }
}
2021/12/14 04:14:05 Context: Bucket flush, response code: 200 dump: 
2021/12/14 04:14:10 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 04:14:12 Context: Bucket flush, response code: 200 dump: 
2021/12/14 04:14:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 04:14:17 Deploying app: TestImportExport_n1ql_insert_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/14 04:14:18 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestImportExport_n1ql_insert_on_update' in metakv",
  "warnings": null
 }
}
2021/12/14 04:14:18 Deploying app: TestImportExport_bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/14 04:14:19 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestImportExport_bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/14 04:14:24 Waiting for app: TestImportExport_n1ql_insert_on_update to get deployed
2021/12/14 04:14:29 Waiting for app: TestImportExport_n1ql_insert_on_update to get deployed
2021/12/14 04:14:34 Waiting for app: TestImportExport_n1ql_insert_on_update to get deployed
2021/12/14 04:14:39 Waiting for app: TestImportExport_n1ql_insert_on_update to get deployed
2021/12/14 04:14:39 App: TestImportExport_n1ql_insert_on_update got deployed
2021/12/14 04:14:44 Waiting for app: TestImportExport_bucket_op_on_update to get deployed
2021/12/14 04:14:49 Waiting for app: TestImportExport_bucket_op_on_update to get deployed
2021/12/14 04:14:54 Waiting for app: TestImportExport_bucket_op_on_update to get deployed
2021/12/14 04:14:54 App: TestImportExport_bucket_op_on_update got deployed
2021/12/14 04:14:55 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 04:15:00 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/14 04:15:05 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/14 04:15:10 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/14 04:15:15 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/14 04:15:15 App: TestImportExport_n1ql_insert_on_update got un-deployed
2021/12/14 04:15:15 No eventing-con process running
2021/12/14 04:15:15 Function: TestImportExport_n1ql_insert_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestImportExport_n1ql_insert_on_update deleting in the background"
}
2021/12/14 04:15:15 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 04:15:20 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/14 04:15:25 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/14 04:15:30 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/14 04:15:35 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/14 04:15:35 App: TestImportExport_bucket_op_on_update got un-deployed
2021/12/14 04:15:35 No eventing-con process running
2021/12/14 04:15:35 Function: TestImportExport_bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestImportExport_bucket_op_on_update deleting in the background"
}
2021/12/14 04:15:52 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/14 04:15:52 App: TestImportExport_bucket_op_on_update got un-deployed
2021/12/14 04:15:52 No eventing-con process running
2021/12/14 04:15:52 Function: TestImportExport_bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestImportExport_bucket_op_on_update deleting in the background"
}
2021/12/14 04:15:54 Context: Bucket flush, response code: 200 dump: 
2021/12/14 04:15:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 04:16:01 Context: Bucket flush, response code: 200 dump: 
2021/12/14 04:16:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 04:16:11 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/14 04:16:11 App: TestImportExport_n1ql_insert_on_update got un-deployed
2021/12/14 04:16:11 No eventing-con process running
2021/12/14 04:16:11 Function: TestImportExport_n1ql_insert_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestImportExport_n1ql_insert_on_update deleting in the background"
}
2021/12/14 04:16:13 Context: Bucket flush, response code: 200 dump: 
2021/12/14 04:16:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 04:16:19 Context: Bucket flush, response code: 200 dump: 
2021/12/14 04:16:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestImportExport (169.56s)
=== RUN   TestDeployUndeployLoopNonDefaultSettings
2021/12/14 04:16:34 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/14 04:16:34 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/14 04:16:34 No eventing-con process running
2021/12/14 04:16:34 Function: TestDeployUndeployLoopNonDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDeployUndeployLoopNonDefaultSettings not found"
 }
}
2021/12/14 04:16:36 Context: Bucket flush, response code: 200 dump: 
2021/12/14 04:16:41 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 04:16:43 Context: Bucket flush, response code: 200 dump: 
2021/12/14 04:16:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 04:16:48 Deploying app: TestDeployUndeployLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/14 04:16:49 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/14 04:16:49 Starting bucket ops to source bucket
2021/12/14 04:16:57 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 04:17:02 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 04:17:07 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 04:17:12 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 04:17:17 Waiting for dst bucket item count to get to: 5000 curr count: 2500
2021/12/14 04:17:17 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/14 04:17:22 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get deployed
2021/12/14 04:17:22 App: TestDeployUndeployLoopNonDefaultSettings got deployed
2021/12/14 04:17:22 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 04:17:22 Undeploying app: bucket_op_on_update
2021/12/14 04:17:22 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 04:17:27 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/14 04:17:32 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/14 04:17:37 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/14 04:17:42 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/14 04:17:42 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/14 04:17:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 04:17:42 No eventing-con process running
2021/12/14 04:17:44 Context: Bucket flush, response code: 200 dump: 
2021/12/14 04:17:52 Context: Bucket flush, response code: 200 dump: 
2021/12/14 04:18:27 Deploying app: TestDeployUndeployLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/14 04:18:28 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/14 04:18:28 Starting bucket ops to source bucket
2021/12/14 04:18:36 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 04:18:41 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 04:18:46 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 04:18:51 Waiting for dst bucket item count to get to: 5000 curr count: 536
2021/12/14 04:18:56 Waiting for dst bucket item count to get to: 5000 curr count: 3036
2021/12/14 04:18:56 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/14 04:19:01 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get deployed
2021/12/14 04:19:01 App: TestDeployUndeployLoopNonDefaultSettings got deployed
2021/12/14 04:19:01 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 04:19:01 Undeploying app: bucket_op_on_update
2021/12/14 04:19:01 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 04:19:06 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/14 04:19:11 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/14 04:19:16 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/14 04:19:21 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/14 04:19:21 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/14 04:19:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 04:19:21 No eventing-con process running
2021/12/14 04:19:23 Context: Bucket flush, response code: 200 dump: 
2021/12/14 04:19:30 Context: Bucket flush, response code: 200 dump: 
2021/12/14 04:20:05 Deploying app: TestDeployUndeployLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/14 04:20:06 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/14 04:20:06 Starting bucket ops to source bucket
2021/12/14 04:20:14 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 04:20:19 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 04:20:24 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 04:20:29 Waiting for dst bucket item count to get to: 5000 curr count: 1968
2021/12/14 04:20:34 Waiting for dst bucket item count to get to: 5000 curr count: 1968
2021/12/14 04:20:34 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/14 04:20:39 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get deployed
2021/12/14 04:20:39 App: TestDeployUndeployLoopNonDefaultSettings got deployed
2021/12/14 04:20:39 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 04:20:39 Undeploying app: bucket_op_on_update
2021/12/14 04:20:39 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 04:20:44 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/14 04:20:49 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/14 04:20:54 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/14 04:20:59 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/14 04:20:59 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/14 04:20:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 04:20:59 No eventing-con process running
2021/12/14 04:21:01 Context: Bucket flush, response code: 200 dump: 
2021/12/14 04:21:08 Context: Bucket flush, response code: 200 dump: 
2021/12/14 04:21:43 Deploying app: TestDeployUndeployLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/14 04:21:44 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/14 04:21:44 Starting bucket ops to source bucket
2021/12/14 04:21:52 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 04:21:57 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 04:22:02 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 04:22:07 Waiting for dst bucket item count to get to: 5000 curr count: 1057
2021/12/14 04:22:12 Waiting for dst bucket item count to get to: 5000 curr count: 3557
2021/12/14 04:22:12 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/14 04:22:17 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get deployed
2021/12/14 04:22:17 App: TestDeployUndeployLoopNonDefaultSettings got deployed
2021/12/14 04:22:17 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 04:22:17 Undeploying app: bucket_op_on_update
2021/12/14 04:22:17 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 04:22:22 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/14 04:22:28 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/14 04:22:33 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/14 04:22:38 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/14 04:22:38 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/14 04:22:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 04:22:38 No eventing-con process running
2021/12/14 04:22:39 Context: Bucket flush, response code: 200 dump: 
2021/12/14 04:22:47 Context: Bucket flush, response code: 200 dump: 
2021/12/14 04:23:22 Deploying app: TestDeployUndeployLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/14 04:23:23 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/14 04:23:23 Starting bucket ops to source bucket
2021/12/14 04:23:31 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 04:23:36 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 04:23:41 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 04:23:46 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 04:23:46 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/14 04:23:51 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get deployed
2021/12/14 04:23:51 App: TestDeployUndeployLoopNonDefaultSettings got deployed
2021/12/14 04:23:51 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 04:23:51 Undeploying app: bucket_op_on_update
2021/12/14 04:23:51 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 04:23:56 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/14 04:24:01 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/14 04:24:06 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/14 04:24:11 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/14 04:24:11 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/14 04:24:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 04:24:11 No eventing-con process running
2021/12/14 04:24:13 Context: Bucket flush, response code: 200 dump: 
2021/12/14 04:24:20 Context: Bucket flush, response code: 200 dump: 
2021/12/14 04:24:55 Function: TestDeployUndeployLoopNonDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDeployUndeployLoopNonDefaultSettings deleting in the background"
}
--- PASS: TestDeployUndeployLoopNonDefaultSettings (510.71s)
=== RUN   TestOnUpdateN1QLOp
2021/12/14 04:25:05 Waiting for app: TestOnUpdateN1QLOp to get un-deployed
2021/12/14 04:25:05 App: TestOnUpdateN1QLOp got un-deployed
2021/12/14 04:25:05 No eventing-con process running
2021/12/14 04:25:05 Function: TestOnUpdateN1QLOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateN1QLOp not found"
 }
}
2021/12/14 04:25:07 Context: Bucket flush, response code: 200 dump: 
2021/12/14 04:25:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 04:25:14 Context: Bucket flush, response code: 200 dump: 
2021/12/14 04:25:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 04:25:19 Deploying app: TestOnUpdateN1QLOp. Item count src bucket: 0 dst bucket: 0
2021/12/14 04:25:20 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateN1QLOp' in metakv",
  "warnings": null
 }
}
2021/12/14 04:25:20 Starting bucket ops to source bucket
2021/12/14 04:25:28 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 04:25:33 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 04:25:38 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 04:25:43 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 04:25:48 Waiting for dst bucket item count to get to: 5000 curr count: 2584
2021/12/14 04:25:48 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/14 04:25:48 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 04:25:48 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 04:25:53 Waiting for app: TestOnUpdateN1QLOp to get un-deployed
2021/12/14 04:25:58 Waiting for app: TestOnUpdateN1QLOp to get un-deployed
2021/12/14 04:26:03 Waiting for app: TestOnUpdateN1QLOp to get un-deployed
2021/12/14 04:26:08 Waiting for app: TestOnUpdateN1QLOp to get un-deployed
2021/12/14 04:26:08 App: TestOnUpdateN1QLOp got un-deployed
2021/12/14 04:26:08 No eventing-con process running
2021/12/14 04:26:08 Function: TestOnUpdateN1QLOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateN1QLOp deleting in the background"
}
2021/12/14 04:26:10 Context: Bucket flush, response code: 200 dump: 
2021/12/14 04:26:20 Waiting for dst bucket item count to get to: 0 curr count: 2903
2021/12/14 04:26:25 Waiting for dst bucket item count to get to: 0 curr count: 2903
2021/12/14 04:26:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 04:26:27 Context: Bucket flush, response code: 200 dump: 
2021/12/14 04:26:37 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/14 04:26:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateN1QLOp (101.68s)
=== RUN   TestOnUpdateBucketOpDefaultSettings
2021/12/14 04:26:47 Waiting for app: TestOnUpdateBucketOpDefaultSettings to get un-deployed
2021/12/14 04:26:47 App: TestOnUpdateBucketOpDefaultSettings got un-deployed
2021/12/14 04:26:47 No eventing-con process running
2021/12/14 04:26:47 Function: TestOnUpdateBucketOpDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateBucketOpDefaultSettings not found"
 }
}
2021/12/14 04:26:49 Context: Bucket flush, response code: 200 dump: 
2021/12/14 04:26:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 04:26:55 Context: Bucket flush, response code: 200 dump: 
2021/12/14 04:27:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 04:27:00 Deploying app: TestOnUpdateBucketOpDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/14 04:27:01 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateBucketOpDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/14 04:27:01 Starting bucket ops to source bucket
2021/12/14 04:27:09 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 04:27:14 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 04:27:20 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 04:27:25 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 04:27:25 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/14 04:27:25 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 04:27:25 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 04:27:30 Waiting for app: TestOnUpdateBucketOpDefaultSettings to get un-deployed
2021/12/14 04:27:35 Waiting for app: TestOnUpdateBucketOpDefaultSettings to get un-deployed
2021/12/14 04:27:40 Waiting for app: TestOnUpdateBucketOpDefaultSettings to get un-deployed
2021/12/14 04:27:45 Waiting for app: TestOnUpdateBucketOpDefaultSettings to get un-deployed
2021/12/14 04:27:45 App: TestOnUpdateBucketOpDefaultSettings got un-deployed
2021/12/14 04:27:45 No eventing-con process running
2021/12/14 04:27:45 Function: TestOnUpdateBucketOpDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateBucketOpDefaultSettings deleting in the background"
}
2021/12/14 04:27:47 Context: Bucket flush, response code: 200 dump: 
2021/12/14 04:27:57 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/14 04:27:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 04:27:58 Context: Bucket flush, response code: 200 dump: 
2021/12/14 04:28:08 Waiting for dst bucket item count to get to: 0 curr count: 4098
2021/12/14 04:28:13 Waiting for dst bucket item count to get to: 0 curr count: 1598
2021/12/14 04:28:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateBucketOpDefaultSettings (96.60s)
=== RUN   TestOnUpdateBucketOpNonDefaultSettings
2021/12/14 04:28:23 Waiting for app: TestOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/14 04:28:23 App: TestOnUpdateBucketOpNonDefaultSettings got un-deployed
2021/12/14 04:28:23 No eventing-con process running
2021/12/14 04:28:23 Function: TestOnUpdateBucketOpNonDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateBucketOpNonDefaultSettings not found"
 }
}
2021/12/14 04:28:25 Context: Bucket flush, response code: 200 dump: 
2021/12/14 04:28:30 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 04:28:32 Context: Bucket flush, response code: 200 dump: 
2021/12/14 04:28:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 04:28:37 Deploying app: TestOnUpdateBucketOpNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/14 04:28:38 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateBucketOpNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/14 04:28:38 Starting bucket ops to source bucket
2021/12/14 04:28:46 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 04:28:51 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 04:28:56 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 04:29:01 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 04:29:06 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 04:29:06 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/14 04:29:06 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 04:29:06 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 04:29:11 Waiting for app: TestOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/14 04:29:16 Waiting for app: TestOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/14 04:29:21 Waiting for app: TestOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/14 04:29:26 Waiting for app: TestOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/14 04:29:26 App: TestOnUpdateBucketOpNonDefaultSettings got un-deployed
2021/12/14 04:29:26 No eventing-con process running
2021/12/14 04:29:26 Function: TestOnUpdateBucketOpNonDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateBucketOpNonDefaultSettings deleting in the background"
}
2021/12/14 04:29:28 Context: Bucket flush, response code: 200 dump: 
2021/12/14 04:29:38 Waiting for dst bucket item count to get to: 0 curr count: 3653
2021/12/14 04:29:43 Waiting for dst bucket item count to get to: 0 curr count: 1153
2021/12/14 04:29:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 04:29:45 Context: Bucket flush, response code: 200 dump: 
2021/12/14 04:29:55 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/14 04:29:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateBucketOpNonDefaultSettings (101.70s)
=== RUN   TestOnUpdateBucketOpDefaultSettings10K
2021/12/14 04:30:05 Waiting for app: TestOnUpdateBucketOpDefaultSettings10K to get un-deployed
2021/12/14 04:30:05 App: TestOnUpdateBucketOpDefaultSettings10K got un-deployed
2021/12/14 04:30:05 No eventing-con process running
2021/12/14 04:30:05 Function: TestOnUpdateBucketOpDefaultSettings10K delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateBucketOpDefaultSettings10K not found"
 }
}
2021/12/14 04:30:07 Context: Bucket flush, response code: 200 dump: 
2021/12/14 04:30:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 04:30:14 Context: Bucket flush, response code: 200 dump: 
2021/12/14 04:30:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 04:30:19 Deploying app: TestOnUpdateBucketOpDefaultSettings10K. Item count src bucket: 0 dst bucket: 0
2021/12/14 04:30:20 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateBucketOpDefaultSettings10K' in metakv",
  "warnings": null
 }
}
2021/12/14 04:30:20 Starting bucket ops to source bucket
2021/12/14 04:30:28 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 04:30:33 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 04:30:38 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 04:30:43 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 04:30:48 Waiting for dst bucket item count to get to: 5000 curr count: 2500
2021/12/14 04:30:48 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/14 04:30:48 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 04:30:48 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 04:30:53 Waiting for app: TestOnUpdateBucketOpDefaultSettings10K to get un-deployed
2021/12/14 04:30:58 Waiting for app: TestOnUpdateBucketOpDefaultSettings10K to get un-deployed
2021/12/14 04:31:03 Waiting for app: TestOnUpdateBucketOpDefaultSettings10K to get un-deployed
2021/12/14 04:31:08 Waiting for app: TestOnUpdateBucketOpDefaultSettings10K to get un-deployed
2021/12/14 04:31:08 App: TestOnUpdateBucketOpDefaultSettings10K got un-deployed
2021/12/14 04:31:08 No eventing-con process running
2021/12/14 04:31:08 Function: TestOnUpdateBucketOpDefaultSettings10K delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateBucketOpDefaultSettings10K deleting in the background"
}
2021/12/14 04:31:10 Context: Bucket flush, response code: 200 dump: 
2021/12/14 04:31:20 Waiting for dst bucket item count to get to: 0 curr count: 2863
2021/12/14 04:31:25 Waiting for dst bucket item count to get to: 0 curr count: 2863
2021/12/14 04:31:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 04:31:27 Context: Bucket flush, response code: 200 dump: 
2021/12/14 04:31:37 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/14 04:31:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateBucketOpDefaultSettings10K (101.67s)
=== RUN   TestOnUpdateBucketOpDefaultSettings100K
2021/12/14 04:31:47 Waiting for app: TestOnUpdateBucketOpDefaultSettings100K to get un-deployed
2021/12/14 04:31:47 App: TestOnUpdateBucketOpDefaultSettings100K got un-deployed
2021/12/14 04:31:47 No eventing-con process running
2021/12/14 04:31:47 Function: TestOnUpdateBucketOpDefaultSettings100K delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateBucketOpDefaultSettings100K not found"
 }
}
2021/12/14 04:31:49 Context: Bucket flush, response code: 200 dump: 
2021/12/14 04:31:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 04:31:55 Context: Bucket flush, response code: 200 dump: 
2021/12/14 04:32:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 04:32:00 Deploying app: TestOnUpdateBucketOpDefaultSettings100K. Item count src bucket: 0 dst bucket: 0
2021/12/14 04:32:02 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateBucketOpDefaultSettings100K' in metakv",
  "warnings": null
 }
}
2021/12/14 04:32:02 Starting bucket ops to source bucket
2021/12/14 04:32:10 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 04:32:15 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 04:32:20 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 04:32:25 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 04:32:25 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/14 04:32:25 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 04:32:25 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 04:32:30 Waiting for app: TestOnUpdateBucketOpDefaultSettings100K to get un-deployed
2021/12/14 04:32:35 Waiting for app: TestOnUpdateBucketOpDefaultSettings100K to get un-deployed
2021/12/14 04:32:40 Waiting for app: TestOnUpdateBucketOpDefaultSettings100K to get un-deployed
2021/12/14 04:32:45 Waiting for app: TestOnUpdateBucketOpDefaultSettings100K to get un-deployed
2021/12/14 04:32:45 App: TestOnUpdateBucketOpDefaultSettings100K got un-deployed
2021/12/14 04:32:45 No eventing-con process running
2021/12/14 04:32:45 Function: TestOnUpdateBucketOpDefaultSettings100K delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateBucketOpDefaultSettings100K deleting in the background"
}
2021/12/14 04:32:47 Context: Bucket flush, response code: 200 dump: 
2021/12/14 04:32:57 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/14 04:32:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 04:32:59 Context: Bucket flush, response code: 200 dump: 
2021/12/14 04:33:09 Waiting for dst bucket item count to get to: 0 curr count: 4868
2021/12/14 04:33:14 Waiting for dst bucket item count to get to: 0 curr count: 2368
2021/12/14 04:33:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateBucketOpDefaultSettings100K (97.13s)
=== RUN   TestOnDeleteBucketOp
2021/12/14 04:33:24 Waiting for app: TestOnDeleteBucketOp to get un-deployed
2021/12/14 04:33:24 App: TestOnDeleteBucketOp got un-deployed
2021/12/14 04:33:24 No eventing-con process running
2021/12/14 04:33:24 Function: TestOnDeleteBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnDeleteBucketOp not found"
 }
}
2021/12/14 04:33:26 Context: Bucket flush, response code: 200 dump: 
2021/12/14 04:33:31 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 04:33:32 Context: Bucket flush, response code: 200 dump: 
2021/12/14 04:33:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 04:33:37 Deploying app: TestOnDeleteBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/14 04:33:39 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnDeleteBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/14 04:33:39 Starting bucket ops to source bucket
2021/12/14 04:33:49 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 04:33:54 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 04:33:59 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 04:34:04 Waiting for dst bucket item count to get to: 5000 curr count: 2504
2021/12/14 04:34:04 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/14 04:34:04 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 04:34:05 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 04:34:10 Waiting for app: TestOnDeleteBucketOp to get un-deployed
2021/12/14 04:34:15 Waiting for app: TestOnDeleteBucketOp to get un-deployed
2021/12/14 04:34:20 Waiting for app: TestOnDeleteBucketOp to get un-deployed
2021/12/14 04:34:25 Waiting for app: TestOnDeleteBucketOp to get un-deployed
2021/12/14 04:34:25 App: TestOnDeleteBucketOp got un-deployed
2021/12/14 04:34:25 No eventing-con process running
2021/12/14 04:34:25 Function: TestOnDeleteBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnDeleteBucketOp deleting in the background"
}
2021/12/14 04:34:26 Context: Bucket flush, response code: 200 dump: 
2021/12/14 04:34:31 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 04:34:34 Context: Bucket flush, response code: 200 dump: 
2021/12/14 04:34:44 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/14 04:34:49 Waiting for dst bucket item count to get to: 0 curr count: 2496
2021/12/14 04:34:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnDeleteBucketOp (94.90s)
=== RUN   TestOnDeleteBucketOp5K
2021/12/14 04:34:59 Waiting for app: TestOnDeleteBucketOp5K to get un-deployed
2021/12/14 04:34:59 App: TestOnDeleteBucketOp5K got un-deployed
2021/12/14 04:34:59 No eventing-con process running
2021/12/14 04:34:59 Function: TestOnDeleteBucketOp5K delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnDeleteBucketOp5K not found"
 }
}
2021/12/14 04:35:01 Context: Bucket flush, response code: 200 dump: 
2021/12/14 04:35:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 04:35:07 Context: Bucket flush, response code: 200 dump: 
2021/12/14 04:35:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 04:35:12 Deploying app: TestOnDeleteBucketOp5K. Item count src bucket: 0 dst bucket: 0
2021/12/14 04:35:14 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnDeleteBucketOp5K' in metakv",
  "warnings": null
 }
}
2021/12/14 04:35:14 Starting bucket ops to source bucket
2021/12/14 04:35:25 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 04:35:30 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 04:35:35 Waiting for dst bucket item count to get to: 5000 curr count: 2249
2021/12/14 04:35:40 Waiting for dst bucket item count to get to: 5000 curr count: 4745
2021/12/14 04:35:40 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/14 04:35:40 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 04:35:40 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 04:35:45 Waiting for app: TestOnDeleteBucketOp5K to get un-deployed
2021/12/14 04:35:50 Waiting for app: TestOnDeleteBucketOp5K to get un-deployed
2021/12/14 04:35:55 Waiting for app: TestOnDeleteBucketOp5K to get un-deployed
2021/12/14 04:36:00 Waiting for app: TestOnDeleteBucketOp5K to get un-deployed
2021/12/14 04:36:00 App: TestOnDeleteBucketOp5K got un-deployed
2021/12/14 04:36:00 No eventing-con process running
2021/12/14 04:36:00 Function: TestOnDeleteBucketOp5K delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnDeleteBucketOp5K deleting in the background"
}
2021/12/14 04:36:01 Context: Bucket flush, response code: 200 dump: 
2021/12/14 04:36:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 04:36:09 Context: Bucket flush, response code: 200 dump: 
2021/12/14 04:36:19 Waiting for dst bucket item count to get to: 0 curr count: 4294
2021/12/14 04:36:24 Waiting for dst bucket item count to get to: 0 curr count: 1798
2021/12/14 04:36:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnDeleteBucketOp5K (95.09s)
=== RUN   TestTimerBucketOp
2021/12/14 04:36:34 Waiting for app: TestTimerBucketOp to get un-deployed
2021/12/14 04:36:34 App: TestTimerBucketOp got un-deployed
2021/12/14 04:36:34 No eventing-con process running
2021/12/14 04:36:34 Function: TestTimerBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestTimerBucketOp not found"
 }
}
2021/12/14 04:36:36 Context: Bucket flush, response code: 200 dump: 
2021/12/14 04:36:41 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 04:36:42 Context: Bucket flush, response code: 200 dump: 
2021/12/14 04:36:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 04:36:47 Deploying app: TestTimerBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/14 04:36:49 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestTimerBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/14 04:36:49 Starting bucket ops to source bucket
2021/12/14 04:36:57 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 04:37:02 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 04:37:07 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 04:37:12 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 04:37:17 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 04:37:22 Waiting for dst bucket item count to get to: 5000 curr count: 527
2021/12/14 04:37:27 Waiting for dst bucket item count to get to: 5000 curr count: 3027
2021/12/14 04:37:27 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/14 04:37:27 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 04:37:27 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 04:37:32 Waiting for app: TestTimerBucketOp to get un-deployed
2021/12/14 04:37:37 Waiting for app: TestTimerBucketOp to get un-deployed
2021/12/14 04:37:42 Waiting for app: TestTimerBucketOp to get un-deployed
2021/12/14 04:37:47 Waiting for app: TestTimerBucketOp to get un-deployed
2021/12/14 04:37:47 App: TestTimerBucketOp got un-deployed
2021/12/14 04:37:47 No eventing-con process running
2021/12/14 04:37:47 Function: TestTimerBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestTimerBucketOp deleting in the background"
}
2021/12/14 04:37:49 Context: Bucket flush, response code: 200 dump: 
2021/12/14 04:37:59 Waiting for dst bucket item count to get to: 0 curr count: 4338
2021/12/14 04:38:04 Waiting for dst bucket item count to get to: 0 curr count: 1838
2021/12/14 04:38:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 04:38:05 Context: Bucket flush, response code: 200 dump: 
2021/12/14 04:38:15 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/14 04:38:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestTimerBucketOp (111.47s)
=== RUN   TestTimerInPastBucketOp
2021/12/14 04:38:25 Waiting for app: TestTimerInPastBucketOp to get un-deployed
2021/12/14 04:38:25 App: TestTimerInPastBucketOp got un-deployed
2021/12/14 04:38:25 No eventing-con process running
2021/12/14 04:38:25 Function: TestTimerInPastBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestTimerInPastBucketOp not found"
 }
}
2021/12/14 04:38:27 Context: Bucket flush, response code: 200 dump: 
2021/12/14 04:38:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 04:38:34 Context: Bucket flush, response code: 200 dump: 
2021/12/14 04:38:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 04:38:39 Deploying app: TestTimerInPastBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/14 04:38:40 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestTimerInPastBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/14 04:38:40 Starting bucket ops to source bucket
2021/12/14 04:38:48 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 04:38:53 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 04:38:58 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 04:39:03 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 04:39:08 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 04:39:13 Waiting for dst bucket item count to get to: 5000 curr count: 29
2021/12/14 04:39:18 Waiting for dst bucket item count to get to: 5000 curr count: 4407
2021/12/14 04:39:23 Waiting for dst bucket item count to get to: 5000 curr count: 4407
2021/12/14 04:39:23 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/14 04:39:23 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 04:39:23 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 04:39:28 Waiting for app: TestTimerInPastBucketOp to get un-deployed
2021/12/14 04:39:33 Waiting for app: TestTimerInPastBucketOp to get un-deployed
2021/12/14 04:39:38 Waiting for app: TestTimerInPastBucketOp to get un-deployed
2021/12/14 04:39:43 Waiting for app: TestTimerInPastBucketOp to get un-deployed
2021/12/14 04:39:43 App: TestTimerInPastBucketOp got un-deployed
2021/12/14 04:39:43 No eventing-con process running
2021/12/14 04:39:43 Function: TestTimerInPastBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestTimerInPastBucketOp deleting in the background"
}
2021/12/14 04:39:45 Context: Bucket flush, response code: 200 dump: 
2021/12/14 04:39:55 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/14 04:39:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 04:39:57 Context: Bucket flush, response code: 200 dump: 
2021/12/14 04:40:07 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/14 04:40:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestTimerInPastBucketOp (111.57s)
=== RUN   TestDeployUndeployLoopTimer
2021/12/14 04:40:17 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/14 04:40:17 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/14 04:40:17 No eventing-con process running
2021/12/14 04:40:17 Function: TestDeployUndeployLoopTimer delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDeployUndeployLoopTimer not found"
 }
}
2021/12/14 04:40:19 Context: Bucket flush, response code: 200 dump: 
2021/12/14 04:40:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 04:40:26 Context: Bucket flush, response code: 200 dump: 
2021/12/14 04:40:31 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 04:40:31 Deploying app: TestDeployUndeployLoopTimer. Item count src bucket: 0 dst bucket: 0
2021/12/14 04:40:32 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopTimer' in metakv",
  "warnings": null
 }
}
2021/12/14 04:40:32 Starting bucket ops to source bucket
2021/12/14 04:40:40 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 04:40:45 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 04:40:50 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 04:40:55 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 04:41:00 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 04:41:05 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 04:41:10 Waiting for dst bucket item count to get to: 5000 curr count: 3770
2021/12/14 04:41:15 Waiting for dst bucket item count to get to: 5000 curr count: 3770
2021/12/14 04:41:15 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/14 04:41:20 Waiting for app: TestDeployUndeployLoopTimer to get deployed
2021/12/14 04:41:20 App: TestDeployUndeployLoopTimer got deployed
2021/12/14 04:41:20 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 04:41:20 Undeploying app: bucket_op_with_timer
2021/12/14 04:41:20 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 04:41:25 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/14 04:41:30 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/14 04:41:35 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/14 04:41:40 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/14 04:41:40 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/14 04:41:40 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 04:41:40 No eventing-con process running
2021/12/14 04:41:42 Context: Bucket flush, response code: 200 dump: 
2021/12/14 04:41:49 Context: Bucket flush, response code: 200 dump: 
2021/12/14 04:42:24 Deploying app: TestDeployUndeployLoopTimer. Item count src bucket: 0 dst bucket: 0
2021/12/14 04:42:25 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopTimer' in metakv",
  "warnings": null
 }
}
2021/12/14 04:42:25 Starting bucket ops to source bucket
2021/12/14 04:42:33 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 04:42:38 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 04:42:43 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 04:42:48 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 04:42:53 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 04:42:58 Waiting for dst bucket item count to get to: 5000 curr count: 236
2021/12/14 04:43:03 Waiting for dst bucket item count to get to: 5000 curr count: 236
2021/12/14 04:43:03 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/14 04:43:08 Waiting for app: TestDeployUndeployLoopTimer to get deployed
2021/12/14 04:43:08 App: TestDeployUndeployLoopTimer got deployed
2021/12/14 04:43:08 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 04:43:08 Undeploying app: bucket_op_with_timer
2021/12/14 04:43:08 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 04:43:13 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/14 04:43:18 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/14 04:43:23 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/14 04:43:28 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/14 04:43:28 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/14 04:43:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 04:43:28 No eventing-con process running
2021/12/14 04:43:30 Context: Bucket flush, response code: 200 dump: 
2021/12/14 04:43:37 Context: Bucket flush, response code: 200 dump: 
2021/12/14 04:44:12 Deploying app: TestDeployUndeployLoopTimer. Item count src bucket: 0 dst bucket: 0
2021/12/14 04:44:13 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopTimer' in metakv",
  "warnings": null
 }
}
2021/12/14 04:44:13 Starting bucket ops to source bucket
2021/12/14 04:44:21 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 04:44:26 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 04:44:31 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 04:44:36 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 04:44:41 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 04:44:46 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 04:44:51 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 04:44:56 Waiting for dst bucket item count to get to: 5000 curr count: 2500
2021/12/14 04:44:56 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/14 04:45:01 Waiting for app: TestDeployUndeployLoopTimer to get deployed
2021/12/14 04:45:01 App: TestDeployUndeployLoopTimer got deployed
2021/12/14 04:45:01 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 04:45:01 Undeploying app: bucket_op_with_timer
2021/12/14 04:45:01 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 04:45:06 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/14 04:45:11 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/14 04:45:16 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/14 04:45:22 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/14 04:45:22 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/14 04:45:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 04:45:22 No eventing-con process running
2021/12/14 04:45:23 Context: Bucket flush, response code: 200 dump: 
2021/12/14 04:45:31 Context: Bucket flush, response code: 200 dump: 
2021/12/14 04:46:06 Deploying app: TestDeployUndeployLoopTimer. Item count src bucket: 0 dst bucket: 0
2021/12/14 04:46:07 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopTimer' in metakv",
  "warnings": null
 }
}
2021/12/14 04:46:07 Starting bucket ops to source bucket
2021/12/14 04:46:15 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 04:46:20 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 04:46:25 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 04:46:30 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 04:46:35 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 04:46:40 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 04:46:45 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 04:46:50 Waiting for dst bucket item count to get to: 5000 curr count: 4047
2021/12/14 04:46:55 Waiting for dst bucket item count to get to: 5000 curr count: 4047
2021/12/14 04:46:55 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/14 04:47:00 Waiting for app: TestDeployUndeployLoopTimer to get deployed
2021/12/14 04:47:00 App: TestDeployUndeployLoopTimer got deployed
2021/12/14 04:47:00 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 04:47:00 Undeploying app: bucket_op_with_timer
2021/12/14 04:47:00 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 04:47:05 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/14 04:47:10 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/14 04:47:15 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/14 04:47:20 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/14 04:47:20 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/14 04:47:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 04:47:20 No eventing-con process running
2021/12/14 04:47:22 Context: Bucket flush, response code: 200 dump: 
2021/12/14 04:47:29 Context: Bucket flush, response code: 200 dump: 
2021/12/14 04:48:04 Deploying app: TestDeployUndeployLoopTimer. Item count src bucket: 0 dst bucket: 0
2021/12/14 04:48:05 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopTimer' in metakv",
  "warnings": null
 }
}
2021/12/14 04:48:05 Starting bucket ops to source bucket
2021/12/14 04:48:13 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 04:48:18 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 04:48:23 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 04:48:28 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 04:48:33 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 04:48:38 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 04:48:43 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 04:48:43 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/14 04:48:48 Waiting for app: TestDeployUndeployLoopTimer to get deployed
2021/12/14 04:48:48 App: TestDeployUndeployLoopTimer got deployed
2021/12/14 04:48:48 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 04:48:48 Undeploying app: bucket_op_with_timer
2021/12/14 04:48:48 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 04:48:53 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/14 04:48:58 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/14 04:49:03 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/14 04:49:08 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/14 04:49:08 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/14 04:49:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 04:49:08 No eventing-con process running
2021/12/14 04:49:10 Context: Bucket flush, response code: 200 dump: 
2021/12/14 04:49:17 Context: Bucket flush, response code: 200 dump: 
2021/12/14 04:49:53 Function: TestDeployUndeployLoopTimer delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDeployUndeployLoopTimer deleting in the background"
}
--- PASS: TestDeployUndeployLoopTimer (585.63s)
=== RUN   TestMultipleHandlers
2021/12/14 04:50:03 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get un-deployed
2021/12/14 04:50:03 App: TestMultipleHandlers_bucket_op_on_update got un-deployed
2021/12/14 04:50:03 No eventing-con process running
2021/12/14 04:50:03 Function: TestMultipleHandlers_bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMultipleHandlers_bucket_op_on_update not found"
 }
}
2021/12/14 04:50:04 Context: Bucket flush, response code: 200 dump: 
2021/12/14 04:50:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 04:50:11 Context: Bucket flush, response code: 200 dump: 
2021/12/14 04:50:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 04:50:21 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get un-deployed
2021/12/14 04:50:21 App: TestMultipleHandlers_bucket_op_on_delete got un-deployed
2021/12/14 04:50:21 No eventing-con process running
2021/12/14 04:50:21 Function: TestMultipleHandlers_bucket_op_on_delete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMultipleHandlers_bucket_op_on_delete not found"
 }
}
2021/12/14 04:50:23 Context: Bucket flush, response code: 200 dump: 
2021/12/14 04:50:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 04:50:30 Context: Bucket flush, response code: 200 dump: 
2021/12/14 04:50:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 04:50:35 Deploying app: TestMultipleHandlers_bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/14 04:50:36 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMultipleHandlers_bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/14 04:50:36 Deploying app: TestMultipleHandlers_bucket_op_on_delete. Item count src bucket: 0 dst bucket: 0
2021/12/14 04:50:37 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMultipleHandlers_bucket_op_on_delete' in metakv",
  "warnings": null
 }
}
2021/12/14 04:50:42 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get deployed
2021/12/14 04:50:47 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get deployed
2021/12/14 04:50:52 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get deployed
2021/12/14 04:50:57 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get deployed
2021/12/14 04:50:57 App: TestMultipleHandlers_bucket_op_on_update got deployed
2021/12/14 04:51:02 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get deployed
2021/12/14 04:51:07 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get deployed
2021/12/14 04:51:12 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get deployed
2021/12/14 04:51:12 App: TestMultipleHandlers_bucket_op_on_delete got deployed
2021/12/14 04:51:12 Starting bucket ops to source bucket
2021/12/14 04:51:22 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 04:51:22 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/14 04:51:22 Starting bucket ops to source bucket
2021/12/14 04:51:37 Waiting for dst bucket item count to get to: 10000 curr count: 6150
2021/12/14 04:51:42 Waiting for dst bucket item count to get to: 10000 curr count: 6150
2021/12/14 04:51:42 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/14 04:51:42 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 04:51:42 Function: TestMultipleHandlers_bucket_op_on_update update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/14 04:51:42 Function: TestMultipleHandlers_bucket_op_on_delete update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/14 04:51:42 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 04:51:47 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get un-deployed
2021/12/14 04:51:52 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get un-deployed
2021/12/14 04:51:57 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get un-deployed
2021/12/14 04:52:02 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get un-deployed
2021/12/14 04:52:02 App: TestMultipleHandlers_bucket_op_on_update got un-deployed
2021/12/14 04:52:02 No eventing-con process running
2021/12/14 04:52:02 Function: TestMultipleHandlers_bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMultipleHandlers_bucket_op_on_update deleting in the background"
}
2021/12/14 04:52:04 Context: Bucket flush, response code: 200 dump: 
2021/12/14 04:52:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 04:52:11 Context: Bucket flush, response code: 200 dump: 
2021/12/14 04:52:21 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/14 04:52:26 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/14 04:52:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 04:52:27 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 04:52:32 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get un-deployed
2021/12/14 04:52:37 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get un-deployed
2021/12/14 04:52:42 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get un-deployed
2021/12/14 04:52:47 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get un-deployed
2021/12/14 04:52:47 App: TestMultipleHandlers_bucket_op_on_delete got un-deployed
2021/12/14 04:52:47 No eventing-con process running
2021/12/14 04:52:47 Function: TestMultipleHandlers_bucket_op_on_delete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMultipleHandlers_bucket_op_on_delete deleting in the background"
}
2021/12/14 04:52:48 Context: Bucket flush, response code: 200 dump: 
2021/12/14 04:52:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 04:52:55 Context: Bucket flush, response code: 200 dump: 
2021/12/14 04:53:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestMultipleHandlers (187.71s)
=== RUN   TestPauseResumeLoopDefaultSettings
2021/12/14 04:53:10 Waiting for app: TestPauseResumeLoopDefaultSettings to get un-deployed
2021/12/14 04:53:10 App: TestPauseResumeLoopDefaultSettings got un-deployed
2021/12/14 04:53:10 No eventing-con process running
2021/12/14 04:53:10 Function: TestPauseResumeLoopDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestPauseResumeLoopDefaultSettings not found"
 }
}
2021/12/14 04:53:12 Context: Bucket flush, response code: 200 dump: 
2021/12/14 04:53:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 04:53:19 Context: Bucket flush, response code: 200 dump: 
2021/12/14 04:53:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 04:53:24 Deploying app: TestPauseResumeLoopDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/14 04:53:25 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestPauseResumeLoopDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/14 04:53:25 Starting bucket ops to source bucket
2021/12/14 04:53:33 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 04:53:38 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 04:53:43 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 04:53:48 Waiting for dst bucket item count to get to: 5000 curr count: 163
2021/12/14 04:53:53 Waiting for dst bucket item count to get to: 5000 curr count: 2663
2021/12/14 04:53:53 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/14 04:53:53 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 04:53:58 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to deployed
2021/12/14 04:53:58 Function: TestPauseResumeLoopDefaultSettings status changed to deployed
2021/12/14 04:53:58 Pausing app: bucket_op_on_update
2021/12/14 04:53:58 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/14 04:54:03 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to paused
2021/12/14 04:54:03 Function: TestPauseResumeLoopDefaultSettings status changed to paused
2021/12/14 04:54:03 Resuming app: bucket_op_on_update
2021/12/14 04:54:04 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/14 04:54:04 Starting bucket ops to source bucket
2021/12/14 04:54:12 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/14 04:54:17 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/14 04:54:22 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/14 04:54:22 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/14 04:54:22 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 04:54:27 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to deployed
2021/12/14 04:54:27 Function: TestPauseResumeLoopDefaultSettings status changed to deployed
2021/12/14 04:54:27 Pausing app: bucket_op_on_update
2021/12/14 04:54:27 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/14 04:54:32 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to paused
2021/12/14 04:54:32 Function: TestPauseResumeLoopDefaultSettings status changed to paused
2021/12/14 04:54:32 Resuming app: bucket_op_on_update
2021/12/14 04:54:34 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/14 04:54:34 Starting bucket ops to source bucket
2021/12/14 04:54:42 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/14 04:54:47 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/14 04:54:52 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/14 04:54:52 src & dst bucket item count matched up. src bucket count: 15000 dst bucket count: 15000
2021/12/14 04:54:52 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 04:54:57 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to deployed
2021/12/14 04:54:57 Function: TestPauseResumeLoopDefaultSettings status changed to deployed
2021/12/14 04:54:57 Pausing app: bucket_op_on_update
2021/12/14 04:54:57 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/14 04:55:02 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to paused
2021/12/14 04:55:02 Function: TestPauseResumeLoopDefaultSettings status changed to paused
2021/12/14 04:55:02 Resuming app: bucket_op_on_update
2021/12/14 04:55:03 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/14 04:55:03 Starting bucket ops to source bucket
2021/12/14 04:55:11 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/14 04:55:16 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/14 04:55:21 Waiting for dst bucket item count to get to: 20000 curr count: 15316
2021/12/14 04:55:26 Waiting for dst bucket item count to get to: 20000 curr count: 15316
2021/12/14 04:55:26 src & dst bucket item count matched up. src bucket count: 20000 dst bucket count: 20000
2021/12/14 04:55:26 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 04:55:31 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to deployed
2021/12/14 04:55:31 Function: TestPauseResumeLoopDefaultSettings status changed to deployed
2021/12/14 04:55:31 Pausing app: bucket_op_on_update
2021/12/14 04:55:31 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/14 04:55:36 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to paused
2021/12/14 04:55:36 Function: TestPauseResumeLoopDefaultSettings status changed to paused
2021/12/14 04:55:36 Resuming app: bucket_op_on_update
2021/12/14 04:55:37 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/14 04:55:37 Starting bucket ops to source bucket
2021/12/14 04:55:45 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/14 04:55:50 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/14 04:55:55 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/14 04:55:55 src & dst bucket item count matched up. src bucket count: 25000 dst bucket count: 25000
2021/12/14 04:55:55 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 04:56:00 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to deployed
2021/12/14 04:56:00 Function: TestPauseResumeLoopDefaultSettings status changed to deployed
2021/12/14 04:56:00 Pausing app: bucket_op_on_update
2021/12/14 04:56:00 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/14 04:56:05 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to paused
2021/12/14 04:56:05 Function: TestPauseResumeLoopDefaultSettings status changed to paused
2021/12/14 04:56:05 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 04:56:10 Waiting for app: TestPauseResumeLoopDefaultSettings to get un-deployed
2021/12/14 04:56:15 Waiting for app: TestPauseResumeLoopDefaultSettings to get un-deployed
2021/12/14 04:56:20 Waiting for app: TestPauseResumeLoopDefaultSettings to get un-deployed
2021/12/14 04:56:25 Waiting for app: TestPauseResumeLoopDefaultSettings to get un-deployed
2021/12/14 04:56:25 App: TestPauseResumeLoopDefaultSettings got un-deployed
2021/12/14 04:56:25 No eventing-con process running
2021/12/14 04:56:25 Function: TestPauseResumeLoopDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestPauseResumeLoopDefaultSettings deleting in the background"
}
2021/12/14 04:56:27 Context: Bucket flush, response code: 200 dump: 
2021/12/14 04:56:37 Waiting for dst bucket item count to get to: 0 curr count: 25000
2021/12/14 04:56:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 04:56:39 Context: Bucket flush, response code: 200 dump: 
2021/12/14 04:56:49 Waiting for dst bucket item count to get to: 0 curr count: 25000
2021/12/14 04:56:54 Waiting for dst bucket item count to get to: 0 curr count: 12500
2021/12/14 04:56:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestPauseResumeLoopDefaultSettings (233.84s)
=== RUN   TestPauseResumeLoopNonDefaultSettings
2021/12/14 04:57:04 Waiting for app: TestPauseResumeLoopNonDefaultSettings to get un-deployed
2021/12/14 04:57:04 App: TestPauseResumeLoopNonDefaultSettings got un-deployed
2021/12/14 04:57:04 No eventing-con process running
2021/12/14 04:57:04 Function: TestPauseResumeLoopNonDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestPauseResumeLoopNonDefaultSettings not found"
 }
}
2021/12/14 04:57:06 Context: Bucket flush, response code: 200 dump: 
2021/12/14 04:57:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 04:57:13 Context: Bucket flush, response code: 200 dump: 
2021/12/14 04:57:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 04:57:18 Deploying app: TestPauseResumeLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/14 04:57:19 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestPauseResumeLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/14 04:57:19 Starting bucket ops to source bucket
2021/12/14 04:57:27 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 04:57:32 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 04:57:37 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 04:57:42 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 04:57:47 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 04:57:52 Waiting for dst bucket item count to get to: 5000 curr count: 185
2021/12/14 04:57:57 Waiting for dst bucket item count to get to: 5000 curr count: 1868
2021/12/14 04:58:02 Waiting for dst bucket item count to get to: 5000 curr count: 4073
2021/12/14 04:58:07 Waiting for dst bucket item count to get to: 5000 curr count: 4890
2021/12/14 04:58:07 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/14 04:58:07 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 04:58:12 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to deployed
2021/12/14 04:58:12 Function: TestPauseResumeLoopNonDefaultSettings status changed to deployed
2021/12/14 04:58:12 Pausing app: bucket_op_with_timer
2021/12/14 04:58:12 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/14 04:58:17 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to paused
2021/12/14 04:58:17 Function: TestPauseResumeLoopNonDefaultSettings status changed to paused
2021/12/14 04:58:17 Resuming app: bucket_op_with_timer
2021/12/14 04:58:18 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/14 04:58:18 Starting bucket ops to source bucket
2021/12/14 04:58:26 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/14 04:58:31 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/14 04:58:36 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/14 04:58:41 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/14 04:58:46 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/14 04:58:51 Waiting for dst bucket item count to get to: 10000 curr count: 5218
2021/12/14 04:58:56 Waiting for dst bucket item count to get to: 10000 curr count: 7718
2021/12/14 04:58:56 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/14 04:58:56 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 04:59:01 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to deployed
2021/12/14 04:59:01 Function: TestPauseResumeLoopNonDefaultSettings status changed to deployed
2021/12/14 04:59:01 Pausing app: bucket_op_with_timer
2021/12/14 04:59:01 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/14 04:59:06 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to paused
2021/12/14 04:59:06 Function: TestPauseResumeLoopNonDefaultSettings status changed to paused
2021/12/14 04:59:06 Resuming app: bucket_op_with_timer
2021/12/14 04:59:08 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/14 04:59:08 Starting bucket ops to source bucket
2021/12/14 04:59:16 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/14 04:59:21 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/14 04:59:26 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/14 04:59:31 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/14 04:59:36 Waiting for dst bucket item count to get to: 15000 curr count: 10734
2021/12/14 04:59:41 Waiting for dst bucket item count to get to: 15000 curr count: 10734
2021/12/14 04:59:41 src & dst bucket item count matched up. src bucket count: 15000 dst bucket count: 15000
2021/12/14 04:59:41 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 04:59:46 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to deployed
2021/12/14 04:59:46 Function: TestPauseResumeLoopNonDefaultSettings status changed to deployed
2021/12/14 04:59:46 Pausing app: bucket_op_with_timer
2021/12/14 04:59:46 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/14 04:59:51 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to paused
2021/12/14 04:59:51 Function: TestPauseResumeLoopNonDefaultSettings status changed to paused
2021/12/14 04:59:51 Resuming app: bucket_op_with_timer
2021/12/14 04:59:52 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/14 04:59:52 Starting bucket ops to source bucket
2021/12/14 05:00:00 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/14 05:00:05 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/14 05:00:10 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/14 05:00:15 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/14 05:00:20 Waiting for dst bucket item count to get to: 20000 curr count: 17212
2021/12/14 05:00:25 Waiting for dst bucket item count to get to: 20000 curr count: 17212
2021/12/14 05:00:25 src & dst bucket item count matched up. src bucket count: 20000 dst bucket count: 20000
2021/12/14 05:00:25 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 05:00:30 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to deployed
2021/12/14 05:00:30 Function: TestPauseResumeLoopNonDefaultSettings status changed to deployed
2021/12/14 05:00:30 Pausing app: bucket_op_with_timer
2021/12/14 05:00:30 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/14 05:00:35 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to paused
2021/12/14 05:00:35 Function: TestPauseResumeLoopNonDefaultSettings status changed to paused
2021/12/14 05:00:35 Resuming app: bucket_op_with_timer
2021/12/14 05:00:36 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/14 05:00:36 Starting bucket ops to source bucket
2021/12/14 05:00:44 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/14 05:00:49 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/14 05:00:54 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/14 05:00:59 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/14 05:01:04 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/14 05:01:09 Waiting for dst bucket item count to get to: 25000 curr count: 22500
2021/12/14 05:01:14 Waiting for dst bucket item count to get to: 25000 curr count: 22500
2021/12/14 05:01:14 src & dst bucket item count matched up. src bucket count: 25000 dst bucket count: 25000
2021/12/14 05:01:14 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 05:01:19 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to deployed
2021/12/14 05:01:19 Function: TestPauseResumeLoopNonDefaultSettings status changed to deployed
2021/12/14 05:01:19 Pausing app: bucket_op_with_timer
2021/12/14 05:01:19 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/14 05:01:24 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to paused
2021/12/14 05:01:24 Function: TestPauseResumeLoopNonDefaultSettings status changed to paused
2021/12/14 05:01:24 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 05:01:29 Waiting for app: TestPauseResumeLoopNonDefaultSettings to get un-deployed
2021/12/14 05:01:34 Waiting for app: TestPauseResumeLoopNonDefaultSettings to get un-deployed
2021/12/14 05:01:39 Waiting for app: TestPauseResumeLoopNonDefaultSettings to get un-deployed
2021/12/14 05:01:44 Waiting for app: TestPauseResumeLoopNonDefaultSettings to get un-deployed
2021/12/14 05:01:44 App: TestPauseResumeLoopNonDefaultSettings got un-deployed
2021/12/14 05:01:45 No eventing-con process running
2021/12/14 05:01:45 Function: TestPauseResumeLoopNonDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestPauseResumeLoopNonDefaultSettings deleting in the background"
}
2021/12/14 05:01:46 Context: Bucket flush, response code: 200 dump: 
2021/12/14 05:01:56 Waiting for dst bucket item count to get to: 0 curr count: 25000
2021/12/14 05:01:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 05:01:58 Context: Bucket flush, response code: 200 dump: 
2021/12/14 05:02:08 Waiting for dst bucket item count to get to: 0 curr count: 18225
2021/12/14 05:02:13 Waiting for dst bucket item count to get to: 0 curr count: 5725
2021/12/14 05:02:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestPauseResumeLoopNonDefaultSettings (319.05s)
=== RUN   TestPauseAndResumeWithWorkerCountChange
2021/12/14 05:02:23 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get un-deployed
2021/12/14 05:02:23 App: TestPauseAndResumeWithWorkerCountChange got un-deployed
2021/12/14 05:02:23 No eventing-con process running
2021/12/14 05:02:23 Function: TestPauseAndResumeWithWorkerCountChange delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestPauseAndResumeWithWorkerCountChange not found"
 }
}
2021/12/14 05:02:25 Context: Bucket flush, response code: 200 dump: 
2021/12/14 05:02:30 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 05:02:32 Context: Bucket flush, response code: 200 dump: 
2021/12/14 05:02:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 05:02:37 Deploying app: TestPauseAndResumeWithWorkerCountChange. Item count src bucket: 0 dst bucket: 0
2021/12/14 05:02:38 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestPauseAndResumeWithWorkerCountChange' in metakv",
  "warnings": null
 }
}
2021/12/14 05:02:43 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get deployed
2021/12/14 05:02:48 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get deployed
2021/12/14 05:02:53 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get deployed
2021/12/14 05:02:58 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get deployed
2021/12/14 05:02:58 App: TestPauseAndResumeWithWorkerCountChange got deployed
2021/12/14 05:02:58 Starting bucket ops to source bucket
2021/12/14 05:03:07 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 05:03:12 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 05:03:17 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 05:03:22 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 05:03:27 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 05:03:32 Waiting for dst bucket item count to get to: 5000 curr count: 4620
2021/12/14 05:03:32 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/14 05:03:32 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 05:03:32 Pausing app: bucket_op_with_timer
2021/12/14 05:03:32 Function: TestPauseAndResumeWithWorkerCountChange update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/14 05:03:37 Waiting for function: TestPauseAndResumeWithWorkerCountChange status to change to paused
2021/12/14 05:03:37 Function: TestPauseAndResumeWithWorkerCountChange status changed to paused
2021/12/14 05:03:37 Resuming app: bucket_op_with_timer
2021/12/14 05:03:39 Function: TestPauseAndResumeWithWorkerCountChange update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:6] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/14 05:03:39 Starting bucket ops to source bucket
2021/12/14 05:03:50 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/14 05:03:55 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/14 05:04:00 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/14 05:04:05 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/14 05:04:10 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/14 05:04:15 Waiting for dst bucket item count to get to: 10000 curr count: 7500
2021/12/14 05:04:15 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/14 05:04:15 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 05:04:20 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get un-deployed
2021/12/14 05:04:25 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get un-deployed
2021/12/14 05:04:30 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get un-deployed
2021/12/14 05:04:35 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get un-deployed
2021/12/14 05:04:35 App: TestPauseAndResumeWithWorkerCountChange got un-deployed
2021/12/14 05:04:35 No eventing-con process running
2021/12/14 05:04:35 Function: TestPauseAndResumeWithWorkerCountChange delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestPauseAndResumeWithWorkerCountChange deleting in the background"
}
2021/12/14 05:04:37 Context: Bucket flush, response code: 200 dump: 
2021/12/14 05:04:47 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/14 05:04:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 05:04:49 Context: Bucket flush, response code: 200 dump: 
2021/12/14 05:04:59 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/14 05:05:04 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/14 05:05:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestPauseAndResumeWithWorkerCountChange (170.88s)
=== RUN   TestPauseResumeWithEventingReb
2021/12/14 05:05:14 Waiting for app: TestPauseResumeWithEventingReb to get un-deployed
2021/12/14 05:05:14 App: TestPauseResumeWithEventingReb got un-deployed
2021/12/14 05:05:14 No eventing-con process running
2021/12/14 05:05:14 Function: TestPauseResumeWithEventingReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestPauseResumeWithEventingReb not found"
 }
}
2021/12/14 05:05:16 Context: Bucket flush, response code: 200 dump: 
2021/12/14 05:05:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 05:05:23 Context: Bucket flush, response code: 200 dump: 
2021/12/14 05:05:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 05:05:28 Deploying app: TestPauseResumeWithEventingReb. Item count src bucket: 0 dst bucket: 0
2021/12/14 05:05:29 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestPauseResumeWithEventingReb' in metakv",
  "warnings": null
 }
}
2021/12/14 05:05:34 Waiting for app: TestPauseResumeWithEventingReb to get deployed
2021/12/14 05:05:39 Waiting for app: TestPauseResumeWithEventingReb to get deployed
2021/12/14 05:05:44 Waiting for app: TestPauseResumeWithEventingReb to get deployed
2021/12/14 05:05:49 Waiting for app: TestPauseResumeWithEventingReb to get deployed
2021/12/14 05:05:49 App: TestPauseResumeWithEventingReb got deployed
2021/12/14 05:05:49 Starting bucket ops to source bucket
2021/12/14 05:05:58 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 05:06:03 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 05:06:08 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 05:06:13 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 05:06:18 Waiting for dst bucket item count to get to: 5000 curr count: 322
2021/12/14 05:06:23 Waiting for dst bucket item count to get to: 5000 curr count: 954
2021/12/14 05:06:28 Waiting for dst bucket item count to get to: 5000 curr count: 3132
2021/12/14 05:06:28 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/14 05:06:28 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 05:06:28 Pausing app: bucket_op_with_timer
2021/12/14 05:06:28 Function: TestPauseResumeWithEventingReb update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/14 05:06:33 Waiting for function: TestPauseResumeWithEventingReb status to change to paused
2021/12/14 05:06:33 Function: TestPauseResumeWithEventingReb status changed to paused
2021/12/14 05:06:33 Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/14 05:06:34 Removing node(s): [] from the cluster
2021/12/14 05:06:39 Rebalance progress: 29.16666666666666
2021/12/14 05:06:44 Rebalance progress: 29.16666666666666
2021/12/14 05:06:49 Rebalance progress: 100
2021/12/14 05:06:49 Resuming app: bucket_op_with_timer
2021/12/14 05:06:50 Function: TestPauseResumeWithEventingReb update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:6] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/14 05:06:50 Starting bucket ops to source bucket
2021/12/14 05:07:02 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/14 05:07:07 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/14 05:07:12 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/14 05:07:17 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/14 05:07:22 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/14 05:07:27 Waiting for dst bucket item count to get to: 10000 curr count: 6047
2021/12/14 05:07:32 Waiting for dst bucket item count to get to: 10000 curr count: 7846
2021/12/14 05:07:32 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/14 05:07:32 Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/14 05:07:37 Rebalance progress: 29.16666666666666
2021/12/14 05:07:42 Rebalance progress: 29.16666666666666
2021/12/14 05:07:47 Rebalance progress: 100

VB distribution across eventing nodes and their workers::

2021/12/14 05:07:52 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:62} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/14 05:08:07 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:62} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/14 05:08:22 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:62} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/14 05:08:37 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:62} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/14 05:08:52 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:62} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/14 05:09:07 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:62} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/14 05:09:22 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:62} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/14 05:09:37 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:62} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/14 05:09:52 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:62} {Name:#primary Bucket:hello-world Completion:100}]

DCP Stream statuses:
	stream status: running
	len: 1024
	vb list dump: [0-1023]

vbucket curr owner:
Producer node: 172.31.2.134:9300	Node UUID: 3fa1fcb4ff18bc159369719de409a268
	workerID: worker_TestPauseResumeWithEventingReb_0
	len: 57
	v: [683-739]
	workerID: worker_TestPauseResumeWithEventingReb_1
	len: 57
	v: [740-796]
	workerID: worker_TestPauseResumeWithEventingReb_2
	len: 57
	v: [797-853]
	workerID: worker_TestPauseResumeWithEventingReb_3
	len: 57
	v: [854-910]
	workerID: worker_TestPauseResumeWithEventingReb_4
	len: 57
	v: [911-967]
	workerID: worker_TestPauseResumeWithEventingReb_5
	len: 56
	v: [968-1023]
Producer node: 127.0.0.1:9303	Node UUID: 8b53aa8d83193f40bdd4fac7e03faf52
	workerID: worker_TestPauseResumeWithEventingReb_2
	len: 57
	v: [456-512]
	workerID: worker_TestPauseResumeWithEventingReb_3
	len: 57
	v: [513-569]
	workerID: worker_TestPauseResumeWithEventingReb_4
	len: 57
	v: [570-626]
	workerID: worker_TestPauseResumeWithEventingReb_5
	len: 56
	v: [627-682]
	workerID: worker_TestPauseResumeWithEventingReb_0
	len: 57
	v: [342-398]
	workerID: worker_TestPauseResumeWithEventingReb_1
	len: 57
	v: [399-455]
Producer node: 127.0.0.1:9302	Node UUID: 98dad925ce8d8297216ed3d37d695f1e
	workerID: worker_TestPauseResumeWithEventingReb_0
	len: 57
	v: [0-56]
	workerID: worker_TestPauseResumeWithEventingReb_1
	len: 57
	v: [57-113]
	workerID: worker_TestPauseResumeWithEventingReb_2
	len: 57
	v: [114-170]
	workerID: worker_TestPauseResumeWithEventingReb_3
	len: 57
	v: [171-227]
	workerID: worker_TestPauseResumeWithEventingReb_4
	len: 57
	v: [228-284]
	workerID: worker_TestPauseResumeWithEventingReb_5
	len: 57
	v: [285-341]


2021/12/14 05:10:07 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 05:10:08 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 05:10:13 clusterSetup.go:397: Waiting for app: TestPauseResumeWithEventingReb to get un-deployed
2021/12/14 05:10:18 clusterSetup.go:397: Waiting for app: TestPauseResumeWithEventingReb to get un-deployed
2021/12/14 05:10:23 clusterSetup.go:397: Waiting for app: TestPauseResumeWithEventingReb to get un-deployed
2021/12/14 05:10:28 clusterSetup.go:397: Waiting for app: TestPauseResumeWithEventingReb to get un-deployed
2021/12/14 05:10:28 clusterSetup.go:405: App: TestPauseResumeWithEventingReb got un-deployed
2021/12/14 05:10:28 eventingSetup.go:46: No eventing-con process running
2021/12/14 05:10:28 eventingSetup.go:418: Function: TestPauseResumeWithEventingReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestPauseResumeWithEventingReb deleting in the background"
}
2021/12/14 05:10:29 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 05:10:39 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/14 05:10:44 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/14 05:10:44 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 05:10:46 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 05:10:56 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/14 05:10:56 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestPauseResumeWithEventingReb (352.33s)
=== RUN   TestBucketDeleteAfterPause
2021/12/14 05:11:06 clusterSetup.go:397: Waiting for app: TestBucketDeleteAfterPause to get un-deployed
2021/12/14 05:11:06 clusterSetup.go:405: App: TestBucketDeleteAfterPause got un-deployed
2021/12/14 05:11:06 eventingSetup.go:46: No eventing-con process running
2021/12/14 05:11:06 eventingSetup.go:418: Function: TestBucketDeleteAfterPause delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestBucketDeleteAfterPause not found"
 }
}
2021/12/14 05:11:08 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 05:11:13 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 05:11:15 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 05:11:20 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 05:11:20 eventingSetup.go:125: Deploying app: TestBucketDeleteAfterPause. Item count src bucket: 0 dst bucket: 0
2021/12/14 05:11:21 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestBucketDeleteAfterPause' in metakv",
  "warnings": null
 }
}
2021/12/14 05:11:26 clusterSetup.go:346: Waiting for app: TestBucketDeleteAfterPause to get deployed
2021/12/14 05:11:31 clusterSetup.go:346: Waiting for app: TestBucketDeleteAfterPause to get deployed
2021/12/14 05:11:36 clusterSetup.go:346: Waiting for app: TestBucketDeleteAfterPause to get deployed
2021/12/14 05:11:41 clusterSetup.go:346: Waiting for app: TestBucketDeleteAfterPause to get deployed
2021/12/14 05:11:41 clusterSetup.go:354: App: TestBucketDeleteAfterPause got deployed
2021/12/14 05:11:41 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/14 05:11:51 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 05:11:51 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/14 05:11:51 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 05:11:51 handler_test.go:712: Pausing app: bucket_op_on_update
2021/12/14 05:11:51 eventingSetup.go:270: Function: TestBucketDeleteAfterPause update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/14 05:11:56 eventingSetup.go:719: Waiting for function: TestBucketDeleteAfterPause status to change to paused
2021/12/14 05:11:56 eventingSetup.go:727: Function: TestBucketDeleteAfterPause status changed to paused
2021/12/14 05:11:56 handler_test.go:716: Deleting source bucket: default
2021/12/14 05:12:02 eventingSetup.go:719: Waiting for function: TestBucketDeleteAfterPause status to change to undeployed
2021/12/14 05:12:02 eventingSetup.go:727: Function: TestBucketDeleteAfterPause status changed to undeployed
2021/12/14 05:12:07 clusterSetup.go:397: Waiting for app: TestBucketDeleteAfterPause to get un-deployed
2021/12/14 05:12:07 clusterSetup.go:405: App: TestBucketDeleteAfterPause got un-deployed
2021/12/14 05:12:07 eventingSetup.go:46: No eventing-con process running
2021/12/14 05:12:07 eventingSetup.go:418: Function: TestBucketDeleteAfterPause delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestBucketDeleteAfterPause deleting in the background"
}
2021/12/14 05:12:09 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 05:12:19 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/14 05:12:19 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 05:12:20 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 05:12:30 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4186
2021/12/14 05:12:35 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4186
2021/12/14 05:12:35 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestBucketDeleteAfterPause (99.19s)
=== RUN   TestChangeFnCodeBetweenPauseResume
2021/12/14 05:12:46 clusterSetup.go:397: Waiting for app: TestChangeFnCodeBetweenPauseResume to get un-deployed
2021/12/14 05:12:46 clusterSetup.go:405: App: TestChangeFnCodeBetweenPauseResume got un-deployed
2021/12/14 05:12:46 eventingSetup.go:46: No eventing-con process running
2021/12/14 05:12:46 eventingSetup.go:418: Function: TestChangeFnCodeBetweenPauseResume delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestChangeFnCodeBetweenPauseResume not found"
 }
}
2021/12/14 05:12:47 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 05:12:52 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 05:12:54 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 05:12:59 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 05:12:59 eventingSetup.go:125: Deploying app: TestChangeFnCodeBetweenPauseResume. Item count src bucket: 0 dst bucket: 0
2021/12/14 05:13:00 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestChangeFnCodeBetweenPauseResume' in metakv",
  "warnings": null
 }
}
2021/12/14 05:13:05 clusterSetup.go:346: Waiting for app: TestChangeFnCodeBetweenPauseResume to get deployed
2021/12/14 05:13:10 clusterSetup.go:346: Waiting for app: TestChangeFnCodeBetweenPauseResume to get deployed
2021/12/14 05:13:15 clusterSetup.go:346: Waiting for app: TestChangeFnCodeBetweenPauseResume to get deployed
2021/12/14 05:13:20 clusterSetup.go:346: Waiting for app: TestChangeFnCodeBetweenPauseResume to get deployed
2021/12/14 05:13:20 clusterSetup.go:354: App: TestChangeFnCodeBetweenPauseResume got deployed
2021/12/14 05:13:20 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/14 05:13:55 handler_test.go:739: Pausing function: TestChangeFnCodeBetweenPauseResume
2021/12/14 05:13:55 eventingSetup.go:270: Function: TestChangeFnCodeBetweenPauseResume update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/14 05:14:00 eventingSetup.go:719: Waiting for function: TestChangeFnCodeBetweenPauseResume status to change to paused
2021/12/14 05:14:00 eventingSetup.go:727: Function: TestChangeFnCodeBetweenPauseResume status changed to paused
2021/12/14 05:17:00 handler_test.go:748: Resuming function: TestChangeFnCodeBetweenPauseResume
2021/12/14 05:17:00 eventingSetup.go:125: Deploying app: TestChangeFnCodeBetweenPauseResume. Item count src bucket: 5000 dst bucket: 0
2021/12/14 05:17:01 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestChangeFnCodeBetweenPauseResume' in metakv",
  "warnings": null
 }
}
2021/12/14 05:17:06 eventingSetup.go:795: Waiting for function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter to get to 5000
2021/12/14 05:17:06 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 05:17:06 eventingSetup.go:799: Function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter got to 0 expected 5000
2021/12/14 05:17:11 eventingSetup.go:795: Waiting for function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter to get to 5000
2021/12/14 05:17:11 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 05:17:11 eventingSetup.go:799: Function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter got to 3403 expected 5000
2021/12/14 05:17:16 eventingSetup.go:795: Waiting for function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter to get to 5000
2021/12/14 05:17:16 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 05:17:16 eventingSetup.go:799: Function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter got to 4492 expected 5000
2021/12/14 05:17:21 eventingSetup.go:795: Waiting for function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter to get to 5000
2021/12/14 05:17:21 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 05:17:21 eventingSetup.go:804: Function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter got to 5000
2021/12/14 05:17:21 handler_test.go:753: Undeploying function: TestChangeFnCodeBetweenPauseResume
2021/12/14 05:17:21 eventingSetup.go:270: Function: TestChangeFnCodeBetweenPauseResume update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/14 05:17:31 clusterSetup.go:397: Waiting for app: TestChangeFnCodeBetweenPauseResume to get un-deployed
2021/12/14 05:17:36 clusterSetup.go:397: Waiting for app: TestChangeFnCodeBetweenPauseResume to get un-deployed
2021/12/14 05:17:41 clusterSetup.go:397: Waiting for app: TestChangeFnCodeBetweenPauseResume to get un-deployed
2021/12/14 05:17:41 clusterSetup.go:405: App: TestChangeFnCodeBetweenPauseResume got un-deployed
2021/12/14 05:17:41 eventingSetup.go:46: No eventing-con process running
2021/12/14 05:17:41 eventingSetup.go:418: Function: TestChangeFnCodeBetweenPauseResume delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestChangeFnCodeBetweenPauseResume deleting in the background"
}
2021/12/14 05:17:43 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 05:17:53 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/14 05:17:58 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/14 05:17:58 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 05:17:59 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 05:18:04 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestChangeFnCodeBetweenPauseResume (328.93s)
=== RUN   TestCommentUnCommentOnDelete
2021/12/14 05:18:14 clusterSetup.go:397: Waiting for app: TestCommentUnCommentOnDelete to get un-deployed
2021/12/14 05:18:14 clusterSetup.go:405: App: TestCommentUnCommentOnDelete got un-deployed
2021/12/14 05:18:14 eventingSetup.go:46: No eventing-con process running
2021/12/14 05:18:14 eventingSetup.go:418: Function: TestCommentUnCommentOnDelete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestCommentUnCommentOnDelete not found"
 }
}
2021/12/14 05:18:16 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 05:18:21 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 05:18:23 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 05:18:28 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 05:18:28 eventingSetup.go:125: Deploying app: TestCommentUnCommentOnDelete. Item count src bucket: 0 dst bucket: 0
2021/12/14 05:18:29 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCommentUnCommentOnDelete' in metakv",
  "warnings": null
 }
}
2021/12/14 05:18:34 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/14 05:18:39 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/14 05:18:44 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/14 05:18:49 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/14 05:18:49 clusterSetup.go:354: App: TestCommentUnCommentOnDelete got deployed
2021/12/14 05:18:49 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/14 05:18:58 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 05:19:03 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 222
2021/12/14 05:19:08 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 2722
2021/12/14 05:19:08 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/14 05:19:08 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 05:19:08 handler_test.go:780: Undeploying app: TestCommentUnCommentOnDelete
2021/12/14 05:19:08 eventingSetup.go:270: Function: TestCommentUnCommentOnDelete update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/14 05:19:38 eventingSetup.go:125: Deploying app: TestCommentUnCommentOnDelete. Item count src bucket: 5000 dst bucket: 5000
2021/12/14 05:19:39 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCommentUnCommentOnDelete' in metakv",
  "warnings": null
 }
}
2021/12/14 05:19:44 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/14 05:19:49 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/14 05:19:54 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/14 05:19:59 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/14 05:19:59 clusterSetup.go:354: App: TestCommentUnCommentOnDelete got deployed
2021/12/14 05:19:59 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/14 05:20:12 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/14 05:20:17 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2504
2021/12/14 05:20:22 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4
2021/12/14 05:20:22 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 05:20:22 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 05:20:22 handler_test.go:799: Undeploying app: TestCommentUnCommentOnDelete
2021/12/14 05:20:22 eventingSetup.go:270: Function: TestCommentUnCommentOnDelete update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/14 05:20:32 clusterSetup.go:397: Waiting for app: TestCommentUnCommentOnDelete to get un-deployed
2021/12/14 05:20:37 clusterSetup.go:397: Waiting for app: TestCommentUnCommentOnDelete to get un-deployed
2021/12/14 05:20:42 clusterSetup.go:397: Waiting for app: TestCommentUnCommentOnDelete to get un-deployed
2021/12/14 05:20:42 clusterSetup.go:405: App: TestCommentUnCommentOnDelete got un-deployed
2021/12/14 05:20:42 eventingSetup.go:46: No eventing-con process running
2021/12/14 05:20:42 eventingSetup.go:418: Function: TestCommentUnCommentOnDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCommentUnCommentOnDelete deleting in the background"
}
2021/12/14 05:20:44 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 05:20:49 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 05:20:50 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 05:20:55 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCommentUnCommentOnDelete (170.98s)
=== RUN   TestCPPWorkerCleanup
2021/12/14 05:21:05 clusterSetup.go:397: Waiting for app: TestCPPWorkerCleanup to get un-deployed
2021/12/14 05:21:05 clusterSetup.go:405: App: TestCPPWorkerCleanup got un-deployed
2021/12/14 05:21:05 eventingSetup.go:46: No eventing-con process running
2021/12/14 05:21:05 eventingSetup.go:418: Function: TestCPPWorkerCleanup delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestCPPWorkerCleanup not found"
 }
}
2021/12/14 05:21:07 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 05:21:12 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 05:21:14 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 05:21:19 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 05:21:19 eventingSetup.go:125: Deploying app: TestCPPWorkerCleanup. Item count src bucket: 0 dst bucket: 0
2021/12/14 05:21:20 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCPPWorkerCleanup' in metakv",
  "warnings": null
 }
}
2021/12/14 05:21:20 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/14 05:21:27 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 05:21:32 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 05:21:37 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 05:21:42 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 05:21:47 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 2403
2021/12/14 05:21:52 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 4903
2021/12/14 05:21:52 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/14 05:21:52 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 05:21:52 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 05:21:57 clusterSetup.go:397: Waiting for app: TestCPPWorkerCleanup to get un-deployed
2021/12/14 05:22:02 clusterSetup.go:397: Waiting for app: TestCPPWorkerCleanup to get un-deployed
2021/12/14 05:22:07 clusterSetup.go:397: Waiting for app: TestCPPWorkerCleanup to get un-deployed
2021/12/14 05:22:12 clusterSetup.go:397: Waiting for app: TestCPPWorkerCleanup to get un-deployed
2021/12/14 05:22:12 clusterSetup.go:405: App: TestCPPWorkerCleanup got un-deployed
2021/12/14 05:22:12 eventingSetup.go:46: No eventing-con process running
2021/12/14 05:22:12 eventingSetup.go:418: Function: TestCPPWorkerCleanup delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCPPWorkerCleanup deleting in the background"
}
2021/12/14 05:22:13 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 05:22:23 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/14 05:22:28 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/14 05:22:28 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 05:22:30 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 05:22:40 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3590
2021/12/14 05:22:45 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3590
2021/12/14 05:22:45 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCPPWorkerCleanup (139.80s)
=== RUN   TestWithUserXattrs
2021/12/14 05:23:25 clusterSetup.go:397: Waiting for app: TestWithUserXattrs to get un-deployed
2021/12/14 05:23:25 clusterSetup.go:405: App: TestWithUserXattrs got un-deployed
2021/12/14 05:23:25 eventingSetup.go:46: No eventing-con process running
2021/12/14 05:23:25 eventingSetup.go:418: Function: TestWithUserXattrs delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestWithUserXattrs not found"
 }
}
2021/12/14 05:23:27 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 05:23:32 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 05:23:33 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 05:23:38 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 05:23:38 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/14 05:23:40 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/14 05:23:42 eventingSetup.go:125: Deploying app: TestWithUserXattrs. Item count src bucket: 0 dst bucket: 0
2021/12/14 05:23:44 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestWithUserXattrs' in metakv",
  "warnings": null
 }
}
2021/12/14 05:23:49 clusterSetup.go:346: Waiting for app: TestWithUserXattrs to get deployed
2021/12/14 05:23:54 clusterSetup.go:346: Waiting for app: TestWithUserXattrs to get deployed
2021/12/14 05:23:59 clusterSetup.go:346: Waiting for app: TestWithUserXattrs to get deployed
2021/12/14 05:24:04 clusterSetup.go:346: Waiting for app: TestWithUserXattrs to get deployed
2021/12/14 05:24:04 clusterSetup.go:354: App: TestWithUserXattrs got deployed
2021/12/14 05:24:04 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/14 05:24:13 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 05:24:18 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 2083
2021/12/14 05:24:23 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 4583
2021/12/14 05:24:23 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/14 05:24:23 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 05:24:23 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 05:24:28 clusterSetup.go:397: Waiting for app: TestWithUserXattrs to get un-deployed
2021/12/14 05:24:33 clusterSetup.go:397: Waiting for app: TestWithUserXattrs to get un-deployed
2021/12/14 05:24:38 clusterSetup.go:397: Waiting for app: TestWithUserXattrs to get un-deployed
2021/12/14 05:24:43 clusterSetup.go:397: Waiting for app: TestWithUserXattrs to get un-deployed
2021/12/14 05:24:43 clusterSetup.go:405: App: TestWithUserXattrs got un-deployed
2021/12/14 05:24:43 eventingSetup.go:46: No eventing-con process running
2021/12/14 05:24:43 eventingSetup.go:418: Function: TestWithUserXattrs delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestWithUserXattrs deleting in the background"
}
2021/12/14 05:24:44 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 05:24:54 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/14 05:24:54 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 05:24:56 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 05:25:06 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/14 05:25:06 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestWithUserXattrs (110.97s)
=== RUN   TestEventProcessingPostBucketFlush
2021/12/14 05:25:16 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/14 05:25:16 clusterSetup.go:405: App: TestEventProcessingPostBucketFlush got un-deployed
2021/12/14 05:25:16 eventingSetup.go:46: No eventing-con process running
2021/12/14 05:25:16 eventingSetup.go:418: Function: TestEventProcessingPostBucketFlush delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventProcessingPostBucketFlush not found"
 }
}
2021/12/14 05:25:18 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 05:25:23 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 05:25:24 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 05:25:29 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 05:25:29 eventingSetup.go:125: Deploying app: TestEventProcessingPostBucketFlush. Item count src bucket: 0 dst bucket: 0
2021/12/14 05:25:30 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventProcessingPostBucketFlush' in metakv",
  "warnings": null
 }
}
2021/12/14 05:25:35 clusterSetup.go:346: Waiting for app: TestEventProcessingPostBucketFlush to get deployed
2021/12/14 05:25:40 clusterSetup.go:346: Waiting for app: TestEventProcessingPostBucketFlush to get deployed
2021/12/14 05:25:45 clusterSetup.go:346: Waiting for app: TestEventProcessingPostBucketFlush to get deployed
2021/12/14 05:25:50 clusterSetup.go:346: Waiting for app: TestEventProcessingPostBucketFlush to get deployed
2021/12/14 05:25:50 clusterSetup.go:354: App: TestEventProcessingPostBucketFlush got deployed
2021/12/14 05:25:50 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/14 05:26:00 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 05:26:05 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 05:26:05 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/14 05:26:05 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 05:26:08 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 05:26:18 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2573
2021/12/14 05:26:23 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 73
2021/12/14 05:26:23 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 05:26:25 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 05:26:35 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/14 05:26:35 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 05:26:35 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/14 05:27:01 eventingSetup.go:441: Waiting for dst bucket item count to get to: 25000 curr count: 15615
2021/12/14 05:27:06 eventingSetup.go:441: Waiting for dst bucket item count to get to: 25000 curr count: 15615
2021/12/14 05:27:06 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 25000 dst bucket count: 25000
2021/12/14 05:27:06 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 05:27:06 eventingSetup.go:270: Function: TestEventProcessingPostBucketFlush update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/14 05:27:11 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/14 05:27:16 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/14 05:27:21 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/14 05:27:26 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/14 05:27:26 clusterSetup.go:405: App: TestEventProcessingPostBucketFlush got un-deployed
2021/12/14 05:27:26 eventingSetup.go:46: No eventing-con process running
2021/12/14 05:27:31 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/14 05:27:31 clusterSetup.go:405: App: TestEventProcessingPostBucketFlush got un-deployed
2021/12/14 05:27:31 eventingSetup.go:46: No eventing-con process running
2021/12/14 05:27:31 eventingSetup.go:418: Function: TestEventProcessingPostBucketFlush delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventProcessingPostBucketFlush deleting in the background"
}
2021/12/14 05:27:32 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 05:27:42 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 25000
2021/12/14 05:27:47 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 25000
2021/12/14 05:27:47 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 05:27:49 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 05:27:59 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 25000
2021/12/14 05:28:04 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12500
2021/12/14 05:28:04 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventProcessingPostBucketFlush (177.81s)
=== RUN   TestMetaBucketDelete
2021/12/14 05:28:14 clusterSetup.go:397: Waiting for app: TestMetaBucketDelete to get un-deployed
2021/12/14 05:28:14 clusterSetup.go:405: App: TestMetaBucketDelete got un-deployed
2021/12/14 05:28:14 eventingSetup.go:46: No eventing-con process running
2021/12/14 05:28:14 eventingSetup.go:418: Function: TestMetaBucketDelete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMetaBucketDelete not found"
 }
}
2021/12/14 05:28:15 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 05:28:20 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 05:28:22 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 05:28:27 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 05:28:27 eventingSetup.go:125: Deploying app: TestMetaBucketDelete. Item count src bucket: 0 dst bucket: 0
2021/12/14 05:28:28 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMetaBucketDelete' in metakv",
  "warnings": null
 }
}
2021/12/14 05:28:33 clusterSetup.go:346: Waiting for app: TestMetaBucketDelete to get deployed
2021/12/14 05:28:38 clusterSetup.go:346: Waiting for app: TestMetaBucketDelete to get deployed
2021/12/14 05:28:43 clusterSetup.go:346: Waiting for app: TestMetaBucketDelete to get deployed
2021/12/14 05:28:48 clusterSetup.go:346: Waiting for app: TestMetaBucketDelete to get deployed
2021/12/14 05:28:48 clusterSetup.go:354: App: TestMetaBucketDelete got deployed
2021/12/14 05:28:48 handler_test.go:902: Deleting metadata bucket: eventing
2021/12/14 05:28:50 handler_test.go:904: Deleted metadata bucket: eventing
2021/12/14 05:28:55 clusterSetup.go:397: Waiting for app: TestMetaBucketDelete to get un-deployed
2021/12/14 05:28:55 clusterSetup.go:405: App: TestMetaBucketDelete got un-deployed
2021/12/14 05:29:10 clusterSetup.go:397: Waiting for app: TestMetaBucketDelete to get un-deployed
2021/12/14 05:29:10 clusterSetup.go:405: App: TestMetaBucketDelete got un-deployed
2021/12/14 05:29:10 eventingSetup.go:46: No eventing-con process running
2021/12/14 05:29:10 eventingSetup.go:418: Function: TestMetaBucketDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMetaBucketDelete deleting in the background"
}
2021/12/14 05:29:11 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 05:29:16 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 05:29:18 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 05:29:23 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestMetaBucketDelete (78.95s)
=== RUN   TestMetaBucketDeleteWithBootstrap
2021/12/14 05:29:33 clusterSetup.go:397: Waiting for app: TestMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/14 05:29:33 clusterSetup.go:405: App: TestMetaBucketDeleteWithBootstrap got un-deployed
2021/12/14 05:29:33 eventingSetup.go:46: No eventing-con process running
2021/12/14 05:29:33 eventingSetup.go:418: Function: TestMetaBucketDeleteWithBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMetaBucketDeleteWithBootstrap not found"
 }
}
2021/12/14 05:29:34 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 05:29:39 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 05:29:41 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 05:29:46 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 05:29:46 eventingSetup.go:125: Deploying app: TestMetaBucketDeleteWithBootstrap. Item count src bucket: 0 dst bucket: 0
2021/12/14 05:29:47 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMetaBucketDeleteWithBootstrap' in metakv",
  "warnings": null
 }
}
2021/12/14 05:30:06 handler_test.go:923: Deleting metadata bucket: eventing
2021/12/14 05:30:07 handler_test.go:925: Deleted metadata bucket: eventing
2021/12/14 05:30:22 clusterSetup.go:397: Waiting for app: TestMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/14 05:30:22 clusterSetup.go:405: App: TestMetaBucketDeleteWithBootstrap got un-deployed
2021/12/14 05:30:37 clusterSetup.go:397: Waiting for app: TestMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/14 05:30:37 clusterSetup.go:405: App: TestMetaBucketDeleteWithBootstrap got un-deployed
2021/12/14 05:30:37 eventingSetup.go:46: No eventing-con process running
2021/12/14 05:30:37 eventingSetup.go:418: Function: TestMetaBucketDeleteWithBootstrap delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMetaBucketDeleteWithBootstrap deleting in the background"
}
2021/12/14 05:30:38 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 05:30:43 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 05:30:45 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 05:30:50 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestMetaBucketDeleteWithBootstrap (86.81s)
=== RUN   TestSourceBucketDelete
2021/12/14 05:31:00 clusterSetup.go:397: Waiting for app: TestSourceBucketDelete to get un-deployed
2021/12/14 05:31:00 clusterSetup.go:405: App: TestSourceBucketDelete got un-deployed
2021/12/14 05:31:00 eventingSetup.go:46: No eventing-con process running
2021/12/14 05:31:00 eventingSetup.go:418: Function: TestSourceBucketDelete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestSourceBucketDelete not found"
 }
}
2021/12/14 05:31:01 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 05:31:06 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 05:31:08 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 05:31:13 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 05:31:13 eventingSetup.go:125: Deploying app: TestSourceBucketDelete. Item count src bucket: 0 dst bucket: 0
2021/12/14 05:31:14 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestSourceBucketDelete' in metakv",
  "warnings": null
 }
}
2021/12/14 05:31:19 clusterSetup.go:346: Waiting for app: TestSourceBucketDelete to get deployed
2021/12/14 05:31:24 clusterSetup.go:346: Waiting for app: TestSourceBucketDelete to get deployed
2021/12/14 05:31:29 clusterSetup.go:346: Waiting for app: TestSourceBucketDelete to get deployed
2021/12/14 05:31:34 clusterSetup.go:346: Waiting for app: TestSourceBucketDelete to get deployed
2021/12/14 05:31:34 clusterSetup.go:354: App: TestSourceBucketDelete got deployed
2021/12/14 05:31:34 handler_test.go:945: Deleting source bucket: default
2021/12/14 05:31:36 handler_test.go:947: Deleted source bucket: default
2021/12/14 05:31:41 clusterSetup.go:397: Waiting for app: TestSourceBucketDelete to get un-deployed
2021/12/14 05:31:41 clusterSetup.go:405: App: TestSourceBucketDelete got un-deployed
2021/12/14 05:31:56 clusterSetup.go:397: Waiting for app: TestSourceBucketDelete to get un-deployed
2021/12/14 05:31:56 clusterSetup.go:405: App: TestSourceBucketDelete got un-deployed
2021/12/14 05:31:56 eventingSetup.go:46: No eventing-con process running
2021/12/14 05:31:56 eventingSetup.go:418: Function: TestSourceBucketDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestSourceBucketDelete deleting in the background"
}
2021/12/14 05:31:57 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 05:32:02 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 05:32:04 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 05:32:09 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestSourceBucketDelete (79.26s)
=== RUN   TestSourceBucketDeleteWithBootstrap
2021/12/14 05:32:19 clusterSetup.go:397: Waiting for app: TestSourceBucketDeleteWithBootstrap to get un-deployed
2021/12/14 05:32:19 clusterSetup.go:405: App: TestSourceBucketDeleteWithBootstrap got un-deployed
2021/12/14 05:32:19 eventingSetup.go:46: No eventing-con process running
2021/12/14 05:32:19 eventingSetup.go:418: Function: TestSourceBucketDeleteWithBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestSourceBucketDeleteWithBootstrap not found"
 }
}
2021/12/14 05:32:20 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 05:32:25 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 05:32:27 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 05:32:32 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 05:32:32 eventingSetup.go:125: Deploying app: TestSourceBucketDeleteWithBootstrap. Item count src bucket: 0 dst bucket: 0
2021/12/14 05:32:33 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestSourceBucketDeleteWithBootstrap' in metakv",
  "warnings": null
 }
}
2021/12/14 05:32:52 handler_test.go:966: Deleting source bucket: default
2021/12/14 05:32:53 handler_test.go:968: Deleted source bucket: default
2021/12/14 05:33:08 clusterSetup.go:397: Waiting for app: TestSourceBucketDeleteWithBootstrap to get un-deployed
2021/12/14 05:33:08 clusterSetup.go:405: App: TestSourceBucketDeleteWithBootstrap got un-deployed
2021/12/14 05:33:24 clusterSetup.go:397: Waiting for app: TestSourceBucketDeleteWithBootstrap to get un-deployed
2021/12/14 05:33:24 clusterSetup.go:405: App: TestSourceBucketDeleteWithBootstrap got un-deployed
2021/12/14 05:33:24 eventingSetup.go:46: No eventing-con process running
2021/12/14 05:33:24 eventingSetup.go:418: Function: TestSourceBucketDeleteWithBootstrap delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestSourceBucketDeleteWithBootstrap deleting in the background"
}
2021/12/14 05:33:25 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 05:33:30 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 05:33:32 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 05:33:37 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestSourceBucketDeleteWithBootstrap (87.69s)
=== RUN   TestSourceAndMetaBucketDeleteWithBootstrap
2021/12/14 05:33:47 clusterSetup.go:397: Waiting for app: TestSourceAndMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/14 05:33:47 clusterSetup.go:405: App: TestSourceAndMetaBucketDeleteWithBootstrap got un-deployed
2021/12/14 05:33:47 eventingSetup.go:46: No eventing-con process running
2021/12/14 05:33:47 eventingSetup.go:418: Function: TestSourceAndMetaBucketDeleteWithBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestSourceAndMetaBucketDeleteWithBootstrap not found"
 }
}
2021/12/14 05:33:48 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 05:33:53 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 05:33:55 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 05:34:00 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 05:34:00 eventingSetup.go:125: Deploying app: TestSourceAndMetaBucketDeleteWithBootstrap. Item count src bucket: 0 dst bucket: 0
2021/12/14 05:34:01 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestSourceAndMetaBucketDeleteWithBootstrap' in metakv",
  "warnings": null
 }
}
2021/12/14 05:34:20 handler_test.go:988: Deleting source bucket: default
2021/12/14 05:34:21 handler_test.go:990: Deleted source bucket: default
2021/12/14 05:34:21 handler_test.go:992: Deleting metadata bucket: eventing
2021/12/14 05:34:22 handler_test.go:994: Deleted metadata bucket: eventing
2021/12/14 05:34:37 clusterSetup.go:397: Waiting for app: TestSourceAndMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/14 05:34:37 clusterSetup.go:405: App: TestSourceAndMetaBucketDeleteWithBootstrap got un-deployed
2021/12/14 05:34:52 clusterSetup.go:397: Waiting for app: TestSourceAndMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/14 05:34:52 clusterSetup.go:405: App: TestSourceAndMetaBucketDeleteWithBootstrap got un-deployed
2021/12/14 05:34:52 eventingSetup.go:46: No eventing-con process running
2021/12/14 05:34:52 eventingSetup.go:418: Function: TestSourceAndMetaBucketDeleteWithBootstrap delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestSourceAndMetaBucketDeleteWithBootstrap deleting in the background"
}
2021/12/14 05:34:53 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 05:34:58 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 05:35:00 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 05:35:05 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestSourceAndMetaBucketDeleteWithBootstrap (88.28s)
=== RUN   TestUndeployDuringBootstrap
2021/12/14 05:35:15 clusterSetup.go:397: Waiting for app: TestUndeployDuringBootstrap to get un-deployed
2021/12/14 05:35:15 clusterSetup.go:405: App: TestUndeployDuringBootstrap got un-deployed
2021/12/14 05:35:15 eventingSetup.go:46: No eventing-con process running
2021/12/14 05:35:15 eventingSetup.go:418: Function: TestUndeployDuringBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestUndeployDuringBootstrap not found"
 }
}
2021/12/14 05:35:16 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 05:35:21 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 05:35:23 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 05:35:28 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 05:35:28 eventingSetup.go:125: Deploying app: TestUndeployDuringBootstrap. Item count src bucket: 0 dst bucket: 0
2021/12/14 05:35:29 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestUndeployDuringBootstrap' in metakv",
  "warnings": null
 }
}
2021/12/14 05:35:29 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/14 05:35:41 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 05:35:41 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 05:35:41 clusterSetup.go:375: Apps undergoing bootstrap: true
2021/12/14 05:35:41 clusterSetup.go:378: App: TestUndeployDuringBootstrap started bootstrap
2021/12/14 05:35:41 clusterSetup.go:375: Apps undergoing bootstrap: true
2021/12/14 05:35:46 clusterSetup.go:375: Apps undergoing bootstrap: true
2021/12/14 05:35:51 clusterSetup.go:367: No apps undergoing bootstrap
2021/12/14 05:35:56 clusterSetup.go:397: Waiting for app: TestUndeployDuringBootstrap to get un-deployed
2021/12/14 05:36:01 clusterSetup.go:397: Waiting for app: TestUndeployDuringBootstrap to get un-deployed
2021/12/14 05:36:06 clusterSetup.go:397: Waiting for app: TestUndeployDuringBootstrap to get un-deployed
2021/12/14 05:36:06 clusterSetup.go:405: App: TestUndeployDuringBootstrap got un-deployed
2021/12/14 05:36:06 eventingSetup.go:46: No eventing-con process running
2021/12/14 05:36:31 clusterSetup.go:397: Waiting for app: TestUndeployDuringBootstrap to get un-deployed
2021/12/14 05:36:31 clusterSetup.go:405: App: TestUndeployDuringBootstrap got un-deployed
2021/12/14 05:36:31 eventingSetup.go:46: No eventing-con process running
2021/12/14 05:36:31 eventingSetup.go:418: Function: TestUndeployDuringBootstrap delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestUndeployDuringBootstrap deleting in the background"
}
2021/12/14 05:36:33 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 05:36:43 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/14 05:36:48 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/14 05:36:48 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 05:36:49 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 05:36:59 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 954
2021/12/14 05:37:04 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 954
2021/12/14 05:37:04 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestUndeployDuringBootstrap (119.50s)
=== RUN   TestDeleteBeforeUndeploy
2021/12/14 05:37:09 eventingSetup.go:125: Deploying app: TestDeleteBeforeUndeploy. Item count src bucket: 0 dst bucket: 0
2021/12/14 05:37:11 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeleteBeforeUndeploy' in metakv",
  "warnings": null
 }
}
2021/12/14 05:37:16 clusterSetup.go:346: Waiting for app: TestDeleteBeforeUndeploy to get deployed
2021/12/14 05:37:21 clusterSetup.go:346: Waiting for app: TestDeleteBeforeUndeploy to get deployed
2021/12/14 05:37:26 clusterSetup.go:346: Waiting for app: TestDeleteBeforeUndeploy to get deployed
2021/12/14 05:37:31 clusterSetup.go:346: Waiting for app: TestDeleteBeforeUndeploy to get deployed
2021/12/14 05:37:31 clusterSetup.go:354: App: TestDeleteBeforeUndeploy got deployed
2021/12/14 05:37:31 eventingSetup.go:418: Function: TestDeleteBeforeUndeploy delete from main store request response code: 422 dump: {
 "name": "ERR_APP_NOT_UNDEPLOYED",
 "code": 22,
 "description": "Function hasn't undeployed",
 "attributes": null,
 "runtime_info": {
  "code": 22,
  "info": "Function: TestDeleteBeforeUndeploy skipping delete request from primary store, as it hasn't been undeployed"
 }
}
2021/12/14 05:37:31 eventingSetup.go:270: Function: TestDeleteBeforeUndeploy update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/14 05:37:31 eventingSetup.go:418: Function: TestDeleteBeforeUndeploy delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDeleteBeforeUndeploy deleting in the background"
}
2021/12/14 05:37:36 clusterSetup.go:397: Waiting for app: TestDeleteBeforeUndeploy to get un-deployed
2021/12/14 05:37:36 clusterSetup.go:405: App: TestDeleteBeforeUndeploy got un-deployed
2021/12/14 05:37:36 eventingSetup.go:46: No eventing-con process running
2021/12/14 05:37:36 eventingSetup.go:418: Function: TestDeleteBeforeUndeploy delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDeleteBeforeUndeploy not found"
 }
}
2021/12/14 05:37:37 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 05:37:42 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 05:37:44 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 05:37:49 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDeleteBeforeUndeploy (44.64s)
=== RUN   TestUndeployWhenTimersAreFired
2021/12/14 05:37:59 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/14 05:37:59 clusterSetup.go:405: App: TestUndeployWhenTimersAreFired got un-deployed
2021/12/14 05:37:59 eventingSetup.go:46: No eventing-con process running
2021/12/14 05:37:59 eventingSetup.go:418: Function: TestUndeployWhenTimersAreFired delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestUndeployWhenTimersAreFired not found"
 }
}
2021/12/14 05:38:00 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 05:38:05 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 05:38:07 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 05:38:12 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 05:38:12 eventingSetup.go:125: Deploying app: TestUndeployWhenTimersAreFired. Item count src bucket: 0 dst bucket: 0
2021/12/14 05:38:14 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestUndeployWhenTimersAreFired' in metakv",
  "warnings": null
 }
}
2021/12/14 05:38:19 clusterSetup.go:346: Waiting for app: TestUndeployWhenTimersAreFired to get deployed
2021/12/14 05:38:24 clusterSetup.go:346: Waiting for app: TestUndeployWhenTimersAreFired to get deployed
2021/12/14 05:38:29 clusterSetup.go:346: Waiting for app: TestUndeployWhenTimersAreFired to get deployed
2021/12/14 05:38:34 clusterSetup.go:346: Waiting for app: TestUndeployWhenTimersAreFired to get deployed
2021/12/14 05:38:34 clusterSetup.go:354: App: TestUndeployWhenTimersAreFired got deployed
2021/12/14 05:38:34 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/14 05:39:04 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 05:39:09 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/14 05:39:14 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/14 05:39:19 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/14 05:39:24 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/14 05:39:24 clusterSetup.go:405: App: TestUndeployWhenTimersAreFired got un-deployed
2021/12/14 05:39:24 eventingSetup.go:46: No eventing-con process running
2021/12/14 05:41:09 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/14 05:41:09 clusterSetup.go:405: App: TestUndeployWhenTimersAreFired got un-deployed
2021/12/14 05:41:09 eventingSetup.go:46: No eventing-con process running
2021/12/14 05:41:09 eventingSetup.go:418: Function: TestUndeployWhenTimersAreFired delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestUndeployWhenTimersAreFired deleting in the background"
}
2021/12/14 05:41:10 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 05:41:20 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4419
2021/12/14 05:41:25 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4419
2021/12/14 05:41:25 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 05:41:27 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 05:41:37 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/14 05:41:37 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestUndeployWhenTimersAreFired (228.17s)
=== RUN   TestUndeployWithKVFailover
2021/12/14 05:41:47 clusterSetup.go:397: Waiting for app: TestUndeployWithKVFailover to get un-deployed
2021/12/14 05:41:47 clusterSetup.go:405: App: TestUndeployWithKVFailover got un-deployed
2021/12/14 05:41:47 eventingSetup.go:46: No eventing-con process running
2021/12/14 05:41:47 eventingSetup.go:418: Function: TestUndeployWithKVFailover delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestUndeployWithKVFailover not found"
 }
}
2021/12/14 05:41:49 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 05:41:54 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 05:41:55 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 05:42:00 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 05:42:00 eventingSetup.go:125: Deploying app: TestUndeployWithKVFailover. Item count src bucket: 0 dst bucket: 0
2021/12/14 05:42:02 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestUndeployWithKVFailover' in metakv",
  "warnings": null
 }
}
2021/12/14 05:42:02 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/14 05:42:09 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 05:42:14 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 05:42:19 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 05:42:24 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 05:42:29 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 05:42:34 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 05:42:39 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 2109
2021/12/14 05:42:44 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 2109
2021/12/14 05:42:44 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/14 05:42:44 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: kv to the cluster
2021/12/14 05:42:44 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/14 05:42:49 Rebalance progress: 29.16666666666666
2021/12/14 05:42:54 Rebalance progress: 29.16666666666666
2021/12/14 05:42:59 Rebalance progress: 100
2021/12/14 05:42:59 Failing over: [127.0.0.1:19003]
2021/12/14 05:43:09 Function: TestUndeployWithKVFailover update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/14 05:44:09 Removing node(s): [] from the cluster
2021/12/14 05:44:14 Rebalance progress: 29.16666666666666
2021/12/14 05:44:19 Rebalance progress: 29.16666666666666
2021/12/14 05:44:24 Rebalance progress: 100
2021/12/14 05:44:29 Waiting for app: TestUndeployWithKVFailover to get un-deployed
2021/12/14 05:44:29 App: TestUndeployWithKVFailover got un-deployed
2021/12/14 05:44:29 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 05:44:34 Waiting for app: TestUndeployWithKVFailover to get un-deployed
2021/12/14 05:44:34 App: TestUndeployWithKVFailover got un-deployed
2021/12/14 05:44:34 No eventing-con process running
2021/12/14 05:44:34 Function: TestUndeployWithKVFailover delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestUndeployWithKVFailover deleting in the background"
}
2021/12/14 05:44:35 Context: Bucket flush, response code: 200 dump: 
2021/12/14 05:44:45 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/14 05:44:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 05:44:47 Context: Bucket flush, response code: 200 dump: 
2021/12/14 05:44:57 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/14 05:44:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestUndeployWithKVFailover (199.82s)
=== RUN   TestBucketFlushWhileFnDeployed
2021/12/14 05:45:07 Waiting for app: TestBucketFlushWhileFnDeployed to get un-deployed
2021/12/14 05:45:07 App: TestBucketFlushWhileFnDeployed got un-deployed
2021/12/14 05:45:07 No eventing-con process running
2021/12/14 05:45:07 Function: TestBucketFlushWhileFnDeployed delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestBucketFlushWhileFnDeployed not found"
 }
}
2021/12/14 05:45:08 Context: Bucket flush, response code: 200 dump: 
2021/12/14 05:45:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 05:45:15 Context: Bucket flush, response code: 200 dump: 
2021/12/14 05:45:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 05:45:20 Deploying app: TestBucketFlushWhileFnDeployed. Item count src bucket: 0 dst bucket: 0
2021/12/14 05:45:21 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestBucketFlushWhileFnDeployed' in metakv",
  "warnings": null
 }
}
2021/12/14 05:45:26 Waiting for app: TestBucketFlushWhileFnDeployed to get deployed
2021/12/14 05:45:31 Waiting for app: TestBucketFlushWhileFnDeployed to get deployed
2021/12/14 05:45:36 Waiting for app: TestBucketFlushWhileFnDeployed to get deployed
2021/12/14 05:45:41 Waiting for app: TestBucketFlushWhileFnDeployed to get deployed
2021/12/14 05:45:41 App: TestBucketFlushWhileFnDeployed got deployed
2021/12/14 05:45:41 Starting bucket ops to source bucket
2021/12/14 05:46:02 Waiting for dst bucket item count to get to: 20000 curr count: 8765
2021/12/14 05:46:02 src & dst bucket item count matched up. src bucket count: 20000 dst bucket count: 20000
2021/12/14 05:46:02 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 05:46:04 Context: Bucket flush, response code: 200 dump: 
2021/12/14 05:46:14 Waiting for dst bucket item count to get to: 0 curr count: 20000
2021/12/14 05:46:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 05:46:16 Context: Bucket flush, response code: 200 dump: 
2021/12/14 05:46:26 Waiting for dst bucket item count to get to: 0 curr count: 20000
2021/12/14 05:46:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 05:46:26 Starting bucket ops to source bucket
2021/12/14 05:46:40 Waiting for dst bucket item count to get to: 10000 curr count: 2159
2021/12/14 05:46:45 Waiting for dst bucket item count to get to: 10000 curr count: 2159
2021/12/14 05:46:45 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/14 05:46:45 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 05:46:45 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 05:46:50 Waiting for app: TestBucketFlushWhileFnDeployed to get un-deployed
2021/12/14 05:46:55 Waiting for app: TestBucketFlushWhileFnDeployed to get un-deployed
2021/12/14 05:47:00 Waiting for app: TestBucketFlushWhileFnDeployed to get un-deployed
2021/12/14 05:47:05 Waiting for app: TestBucketFlushWhileFnDeployed to get un-deployed
2021/12/14 05:47:05 App: TestBucketFlushWhileFnDeployed got un-deployed
2021/12/14 05:47:05 No eventing-con process running
2021/12/14 05:47:05 Function: TestBucketFlushWhileFnDeployed delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestBucketFlushWhileFnDeployed deleting in the background"
}
2021/12/14 05:47:06 Context: Bucket flush, response code: 200 dump: 
2021/12/14 05:47:16 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/14 05:47:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 05:47:18 Context: Bucket flush, response code: 200 dump: 
2021/12/14 05:47:28 Waiting for dst bucket item count to get to: 0 curr count: 7022
2021/12/14 05:47:33 Waiting for dst bucket item count to get to: 0 curr count: 2022
2021/12/14 05:47:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestBucketFlushWhileFnDeployed (155.90s)
=== RUN   TestTimerOverwrite
2021/12/14 05:47:43 Waiting for app: TestTimerOverwrite to get un-deployed
2021/12/14 05:47:43 App: TestTimerOverwrite got un-deployed
2021/12/14 05:47:43 No eventing-con process running
2021/12/14 05:47:43 Function: TestTimerOverwrite delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestTimerOverwrite not found"
 }
}
2021/12/14 05:47:44 Context: Bucket flush, response code: 200 dump: 
2021/12/14 05:47:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 05:47:51 Context: Bucket flush, response code: 200 dump: 
2021/12/14 05:47:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 05:47:56 Deploying app: TestTimerOverwrite. Item count src bucket: 0 dst bucket: 0
2021/12/14 05:47:57 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestTimerOverwrite' in metakv",
  "warnings": null
 }
}
2021/12/14 05:48:02 Waiting for app: TestTimerOverwrite to get deployed
2021/12/14 05:48:07 Waiting for app: TestTimerOverwrite to get deployed
2021/12/14 05:48:12 Waiting for app: TestTimerOverwrite to get deployed
2021/12/14 05:48:17 Waiting for app: TestTimerOverwrite to get deployed
2021/12/14 05:48:17 App: TestTimerOverwrite got deployed
2021/12/14 05:48:27 Starting bucket ops to source bucket
2021/12/14 05:48:37 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 05:48:42 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 05:48:47 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 05:48:52 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 05:48:57 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 05:49:02 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 05:49:07 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 05:49:12 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 05:49:17 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 05:49:22 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 05:49:27 Waiting for dst bucket item count to get to: 5000 curr count: 555
2021/12/14 05:49:32 Waiting for dst bucket item count to get to: 5000 curr count: 1124
2021/12/14 05:49:37 Waiting for dst bucket item count to get to: 5000 curr count: 3072
2021/12/14 05:49:37 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/14 05:49:47 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 05:49:47 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 05:49:52 Waiting for app: TestTimerOverwrite to get un-deployed
2021/12/14 05:49:57 Waiting for app: TestTimerOverwrite to get un-deployed
2021/12/14 05:50:02 Waiting for app: TestTimerOverwrite to get un-deployed
2021/12/14 05:50:07 Waiting for app: TestTimerOverwrite to get un-deployed
2021/12/14 05:50:07 App: TestTimerOverwrite got un-deployed
2021/12/14 05:50:07 No eventing-con process running
2021/12/14 05:50:07 Function: TestTimerOverwrite delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestTimerOverwrite deleting in the background"
}
2021/12/14 05:50:08 Context: Bucket flush, response code: 200 dump: 
2021/12/14 05:50:18 Waiting for dst bucket item count to get to: 0 curr count: 4204
2021/12/14 05:50:23 Waiting for dst bucket item count to get to: 0 curr count: 1704
2021/12/14 05:50:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 05:50:25 Context: Bucket flush, response code: 200 dump: 
2021/12/14 05:50:35 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/14 05:50:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestTimerOverwrite (181.87s)
=== RUN   TestUndeployBackdoorDuringBootstrap
2021/12/14 05:50:40 Adding node: https://127.0.0.1:19003 with role: kv to the cluster
2021/12/14 05:50:40 Removing node(s): [] from the cluster
2021/12/14 05:50:45 Rebalance progress: 100
2021/12/14 05:50:45 Failing over: [127.0.0.1:19003]
2021/12/14 05:50:50 Waiting for app: TestUndeployBackdoorDuringBootstrap to get un-deployed
2021/12/14 05:50:50 App: TestUndeployBackdoorDuringBootstrap got un-deployed
2021/12/14 05:50:50 No eventing-con process running
2021/12/14 05:50:50 Function: TestUndeployBackdoorDuringBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestUndeployBackdoorDuringBootstrap not found"
 }
}
2021/12/14 05:50:52 Context: Bucket flush, response code: 200 dump: 
2021/12/14 05:50:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 05:50:58 Context: Bucket flush, response code: 200 dump: 
2021/12/14 05:51:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 05:51:03 Deploying app: TestUndeployBackdoorDuringBootstrap. Item count src bucket: 0 dst bucket: 0
2021/12/14 05:51:05 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestUndeployBackdoorDuringBootstrap' in metakv",
  "warnings": null
 }
}
2021/12/14 05:51:05 Starting bucket ops to source bucket
2021/12/14 05:51:15 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 05:51:15 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 05:51:15 Context: Post to set retry counter, response code: 200 dump: 
2021/12/14 05:51:15 set retry counter, response &{[] }
2021/12/14 05:52:20 Waiting for app: TestUndeployBackdoorDuringBootstrap to get un-deployed
2021/12/14 05:52:20 App: TestUndeployBackdoorDuringBootstrap got un-deployed
2021/12/14 05:52:20 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 05:52:20 Function: TestUndeployBackdoorDuringBootstrap delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestUndeployBackdoorDuringBootstrap deleting in the background"
}
2021/12/14 05:52:25 Waiting for app: TestUndeployBackdoorDuringBootstrap to get un-deployed
2021/12/14 05:52:25 App: TestUndeployBackdoorDuringBootstrap got un-deployed
2021/12/14 05:52:25 No eventing-con process running
2021/12/14 05:52:25 Function: TestUndeployBackdoorDuringBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestUndeployBackdoorDuringBootstrap not found"
 }
}
2021/12/14 05:52:26 Context: Bucket flush, response code: 200 dump: 
2021/12/14 05:52:36 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/14 05:52:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 05:52:38 Context: Bucket flush, response code: 200 dump: 
2021/12/14 05:52:48 Waiting for dst bucket item count to get to: 0 curr count: 2756
2021/12/14 05:52:53 Waiting for dst bucket item count to get to: 0 curr count: 256
2021/12/14 05:52:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 05:52:53 Removing node(s): [] from the cluster
2021/12/14 05:52:58 Rebalance progress: 100
--- PASS: TestUndeployBackdoorDuringBootstrap (143.01s)
=== RUN   TestOnUpdateSrcMutation
2021/12/14 05:53:08 Waiting for app: TestOnUpdateSrcMutation to get un-deployed
2021/12/14 05:53:08 App: TestOnUpdateSrcMutation got un-deployed
2021/12/14 05:53:08 No eventing-con process running
2021/12/14 05:53:08 Function: TestOnUpdateSrcMutation delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateSrcMutation not found"
 }
}
2021/12/14 05:53:09 Context: Bucket flush, response code: 200 dump: 
2021/12/14 05:53:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 05:53:16 Context: Bucket flush, response code: 200 dump: 
2021/12/14 05:53:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 05:53:21 Deploying app: TestOnUpdateSrcMutation. Item count src bucket: 0 dst bucket: 0
2021/12/14 05:53:22 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateSrcMutation' in metakv",
  "warnings": null
 }
}
2021/12/14 05:53:22 Starting bucket ops to source bucket
2021/12/14 05:53:44 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/14 05:53:44 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 05:53:44 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 05:53:49 Waiting for app: TestOnUpdateSrcMutation to get un-deployed
2021/12/14 05:53:54 Waiting for app: TestOnUpdateSrcMutation to get un-deployed
2021/12/14 05:53:59 Waiting for app: TestOnUpdateSrcMutation to get un-deployed
2021/12/14 05:54:04 Waiting for app: TestOnUpdateSrcMutation to get un-deployed
2021/12/14 05:54:04 App: TestOnUpdateSrcMutation got un-deployed
2021/12/14 05:54:05 No eventing-con process running
2021/12/14 05:54:05 Function: TestOnUpdateSrcMutation delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateSrcMutation deleting in the background"
}
2021/12/14 05:54:06 Context: Bucket flush, response code: 200 dump: 
2021/12/14 05:54:16 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/14 05:54:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 05:54:18 Context: Bucket flush, response code: 200 dump: 
2021/12/14 05:54:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateSrcMutation (84.69s)
=== RUN   TestOnDeleteSrcMutation
2021/12/14 05:54:33 Waiting for app: TestOnDeleteSrcMutation to get un-deployed
2021/12/14 05:54:33 App: TestOnDeleteSrcMutation got un-deployed
2021/12/14 05:54:33 No eventing-con process running
2021/12/14 05:54:33 Function: TestOnDeleteSrcMutation delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnDeleteSrcMutation not found"
 }
}
2021/12/14 05:54:34 Context: Bucket flush, response code: 200 dump: 
2021/12/14 05:54:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 05:54:41 Context: Bucket flush, response code: 200 dump: 
2021/12/14 05:54:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 05:54:46 Deploying app: TestOnDeleteSrcMutation. Item count src bucket: 0 dst bucket: 0
2021/12/14 05:54:47 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnDeleteSrcMutation' in metakv",
  "warnings": null
 }
}
2021/12/14 05:54:47 Starting bucket ops to source bucket
2021/12/14 05:55:16 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/14 05:55:16 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 05:55:16 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 05:55:21 Waiting for app: TestOnDeleteSrcMutation to get un-deployed
2021/12/14 05:55:26 Waiting for app: TestOnDeleteSrcMutation to get un-deployed
2021/12/14 05:55:31 Waiting for app: TestOnDeleteSrcMutation to get un-deployed
2021/12/14 05:55:36 Waiting for app: TestOnDeleteSrcMutation to get un-deployed
2021/12/14 05:55:36 App: TestOnDeleteSrcMutation got un-deployed
2021/12/14 05:55:36 No eventing-con process running
2021/12/14 05:55:36 Function: TestOnDeleteSrcMutation delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnDeleteSrcMutation deleting in the background"
}
2021/12/14 05:55:37 Context: Bucket flush, response code: 200 dump: 
2021/12/14 05:55:47 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/14 05:55:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 05:55:49 Context: Bucket flush, response code: 200 dump: 
2021/12/14 05:55:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnDeleteSrcMutation (91.25s)
=== RUN   TestOnUpdateSrcMutationWithTimer
2021/12/14 05:56:04 Waiting for app: TestOnUpdateSrcMutationWithTimer to get un-deployed
2021/12/14 05:56:04 App: TestOnUpdateSrcMutationWithTimer got un-deployed
2021/12/14 05:56:04 No eventing-con process running
2021/12/14 05:56:04 Function: TestOnUpdateSrcMutationWithTimer delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateSrcMutationWithTimer not found"
 }
}
2021/12/14 05:56:05 Context: Bucket flush, response code: 200 dump: 
2021/12/14 05:56:10 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 05:56:12 Context: Bucket flush, response code: 200 dump: 
2021/12/14 05:56:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 05:56:17 Deploying app: TestOnUpdateSrcMutationWithTimer. Item count src bucket: 0 dst bucket: 0
2021/12/14 05:56:18 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateSrcMutationWithTimer' in metakv",
  "warnings": null
 }
}
2021/12/14 05:56:18 Starting bucket ops to source bucket
2021/12/14 05:56:55 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/14 05:56:55 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 05:56:55 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 05:57:00 Waiting for app: TestOnUpdateSrcMutationWithTimer to get un-deployed
2021/12/14 05:57:05 Waiting for app: TestOnUpdateSrcMutationWithTimer to get un-deployed
2021/12/14 05:57:10 Waiting for app: TestOnUpdateSrcMutationWithTimer to get un-deployed
2021/12/14 05:57:15 Waiting for app: TestOnUpdateSrcMutationWithTimer to get un-deployed
2021/12/14 05:57:15 App: TestOnUpdateSrcMutationWithTimer got un-deployed
2021/12/14 05:57:15 No eventing-con process running
2021/12/14 05:57:15 Function: TestOnUpdateSrcMutationWithTimer delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateSrcMutationWithTimer deleting in the background"
}
2021/12/14 05:57:17 Context: Bucket flush, response code: 200 dump: 
2021/12/14 05:57:27 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/14 05:57:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 05:57:28 Context: Bucket flush, response code: 200 dump: 
2021/12/14 05:57:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateSrcMutationWithTimer (99.66s)
=== RUN   TestOnDeleteSrcMutationsWithTimer
2021/12/14 05:57:43 Waiting for app: TestOnDeleteSrcMutationsWithTimer to get un-deployed
2021/12/14 05:57:43 App: TestOnDeleteSrcMutationsWithTimer got un-deployed
2021/12/14 05:57:44 No eventing-con process running
2021/12/14 05:57:44 Function: TestOnDeleteSrcMutationsWithTimer delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnDeleteSrcMutationsWithTimer not found"
 }
}
2021/12/14 05:57:45 Context: Bucket flush, response code: 200 dump: 
2021/12/14 05:57:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 05:57:52 Context: Bucket flush, response code: 200 dump: 
2021/12/14 05:57:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 05:57:57 Deploying app: TestOnDeleteSrcMutationsWithTimer. Item count src bucket: 0 dst bucket: 0
2021/12/14 05:57:58 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnDeleteSrcMutationsWithTimer' in metakv",
  "warnings": null
 }
}
2021/12/14 05:57:58 Starting bucket ops to source bucket
2021/12/14 05:58:37 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/14 05:58:37 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 05:58:37 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 05:58:42 Waiting for app: TestOnDeleteSrcMutationsWithTimer to get un-deployed
2021/12/14 05:58:47 Waiting for app: TestOnDeleteSrcMutationsWithTimer to get un-deployed
2021/12/14 05:58:52 Waiting for app: TestOnDeleteSrcMutationsWithTimer to get un-deployed
2021/12/14 05:58:57 Waiting for app: TestOnDeleteSrcMutationsWithTimer to get un-deployed
2021/12/14 05:58:57 App: TestOnDeleteSrcMutationsWithTimer got un-deployed
2021/12/14 05:58:57 No eventing-con process running
2021/12/14 05:58:57 Function: TestOnDeleteSrcMutationsWithTimer delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnDeleteSrcMutationsWithTimer deleting in the background"
}
2021/12/14 05:58:58 Context: Bucket flush, response code: 200 dump: 
2021/12/14 05:59:08 Waiting for dst bucket item count to get to: 0 curr count: 4104
2021/12/14 05:59:13 Waiting for dst bucket item count to get to: 0 curr count: 1604
2021/12/14 05:59:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 05:59:15 Context: Bucket flush, response code: 200 dump: 
2021/12/14 05:59:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnDeleteSrcMutationsWithTimer (106.47s)
=== RUN   TestInterHandlerRecursion
2021/12/14 05:59:30 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/14 05:59:30 App: TestInterHandlerRecursion_source_bucket_update_op got un-deployed
2021/12/14 05:59:30 No eventing-con process running
2021/12/14 05:59:30 Function: TestInterHandlerRecursion_source_bucket_update_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestInterHandlerRecursion_source_bucket_update_op not found"
 }
}
2021/12/14 05:59:31 Context: Bucket flush, response code: 200 dump: 
2021/12/14 05:59:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 05:59:38 Context: Bucket flush, response code: 200 dump: 
2021/12/14 05:59:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 05:59:48 Waiting for app: TestInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/14 05:59:48 App: TestInterHandlerRecursion_src_bucket_op_on_delete got un-deployed
2021/12/14 05:59:48 No eventing-con process running
2021/12/14 05:59:48 Function: TestInterHandlerRecursion_src_bucket_op_on_delete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestInterHandlerRecursion_src_bucket_op_on_delete not found"
 }
}
2021/12/14 05:59:49 Context: Bucket flush, response code: 200 dump: 
2021/12/14 05:59:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 05:59:56 Context: Bucket flush, response code: 200 dump: 
2021/12/14 06:00:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 06:00:01 Deploying app: TestInterHandlerRecursion_source_bucket_update_op. Item count src bucket: 0 dst bucket: 0
2021/12/14 06:00:02 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterHandlerRecursion_source_bucket_update_op' in metakv",
  "warnings": null
 }
}
2021/12/14 06:00:02 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterHandlerRecursion_source_bucket_update_op' in metakv",
  "warnings": null
 }
} err 
2021/12/14 06:00:07 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/14 06:00:12 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/14 06:00:17 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/14 06:00:22 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/14 06:00:22 App: TestInterHandlerRecursion_source_bucket_update_op got deployed
2021/12/14 06:00:22 Deploying app: TestInterHandlerRecursion_src_bucket_op_on_delete. Item count src bucket: 0 dst bucket: 0
2021/12/14 06:00:22 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_INTER_FUNCTION_RECURSION",
 "code": 50,
 "description": "Inter function recursion error, only one function is allowed to do source bucket mutation/delete on a bucket",
 "attributes": null,
 "runtime_info": {
  "code": 50,
  "info": "Inter handler recursion error"
 }
}
2021/12/14 06:00:22 response body {
 "name": "ERR_INTER_FUNCTION_RECURSION",
 "code": 50,
 "description": "Inter function recursion error, only one function is allowed to do source bucket mutation/delete on a bucket",
 "attributes": null,
 "runtime_info": {
  "code": 50,
  "info": "Inter handler recursion error"
 }
} err 
2021/12/14 06:00:23 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 06:00:28 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/14 06:00:33 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/14 06:00:38 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/14 06:00:43 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/14 06:00:43 App: TestInterHandlerRecursion_source_bucket_update_op got un-deployed
2021/12/14 06:00:43 No eventing-con process running
2021/12/14 06:00:43 Function: TestInterHandlerRecursion_source_bucket_update_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestInterHandlerRecursion_source_bucket_update_op deleting in the background"
}
2021/12/14 06:00:44 Context: Bucket flush, response code: 200 dump: 
2021/12/14 06:00:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 06:00:51 Context: Bucket flush, response code: 200 dump: 
2021/12/14 06:00:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 06:01:01 Waiting for app: TestInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/14 06:01:01 App: TestInterHandlerRecursion_src_bucket_op_on_delete got un-deployed
2021/12/14 06:01:01 No eventing-con process running
2021/12/14 06:01:01 Function: TestInterHandlerRecursion_src_bucket_op_on_delete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestInterHandlerRecursion_src_bucket_op_on_delete not found"
 }
}
2021/12/14 06:01:02 Context: Bucket flush, response code: 200 dump: 
2021/12/14 06:01:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 06:01:09 Context: Bucket flush, response code: 200 dump: 
2021/12/14 06:01:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestInterHandlerRecursion (113.90s)
=== RUN   TestInterBucketRecursion
2021/12/14 06:01:19 Deploying app: TestInterBucketRecursion_function1. Item count src bucket: 0 dst bucket: 0
2021/12/14 06:01:20 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterBucketRecursion_function1' in metakv",
  "warnings": null
 }
}
2021/12/14 06:01:20 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterBucketRecursion_function1' in metakv",
  "warnings": null
 }
} err 
2021/12/14 06:01:20 Deploying app: TestInterBucketRecursion_function2. Item count src bucket: 0 dst bucket: 0
2021/12/14 06:01:20 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_INTER_BUCKET_RECURSION",
 "code": 51,
 "description": "Inter bucket recursion error, deployment of current handler will cause inter bucket recursion",
 "attributes": null,
 "runtime_info": {
  "code": 51,
  "info": "Inter bucket recursion error; function: TestInterBucketRecursion_function2 causes a cycle involving functions: [TestInterBucketRecursion_function1], hence deployment is disallowed"
 }
}
2021/12/14 06:01:25 Waiting for app: TestInterBucketRecursion_function1 to get deployed
2021/12/14 06:01:30 Waiting for app: TestInterBucketRecursion_function1 to get deployed
2021/12/14 06:01:35 Waiting for app: TestInterBucketRecursion_function1 to get deployed
2021/12/14 06:01:40 Waiting for app: TestInterBucketRecursion_function1 to get deployed
2021/12/14 06:01:40 App: TestInterBucketRecursion_function1 got deployed
2021/12/14 06:01:40 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 06:01:45 Waiting for app: TestInterBucketRecursion_function1 to get un-deployed
2021/12/14 06:01:50 Waiting for app: TestInterBucketRecursion_function1 to get un-deployed
2021/12/14 06:01:55 Waiting for app: TestInterBucketRecursion_function1 to get un-deployed
2021/12/14 06:02:00 Waiting for app: TestInterBucketRecursion_function1 to get un-deployed
2021/12/14 06:02:00 App: TestInterBucketRecursion_function1 got un-deployed
2021/12/14 06:02:00 No eventing-con process running
2021/12/14 06:02:00 Function: TestInterBucketRecursion_function1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestInterBucketRecursion_function1 deleting in the background"
}
2021/12/14 06:02:01 Context: Bucket flush, response code: 200 dump: 
2021/12/14 06:02:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 06:02:08 Context: Bucket flush, response code: 200 dump: 
2021/12/14 06:02:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 06:02:18 Waiting for app: TestInterBucketRecursion_function2 to get un-deployed
2021/12/14 06:02:18 App: TestInterBucketRecursion_function2 got un-deployed
2021/12/14 06:02:18 No eventing-con process running
2021/12/14 06:02:18 Function: TestInterBucketRecursion_function2 delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestInterBucketRecursion_function2 not found"
 }
}
2021/12/14 06:02:20 Context: Bucket flush, response code: 200 dump: 
2021/12/14 06:02:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 06:02:26 Context: Bucket flush, response code: 200 dump: 
2021/12/14 06:02:31 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestInterBucketRecursion (77.58s)
=== RUN   TestLargeHandler
2021/12/14 06:02:31 Changed force_compress value to false
2021/12/14 06:02:31 Deploying app: TestLargeHandler_function. Item count src bucket: 0 dst bucket: 0
2021/12/14 06:02:32 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_APPCODE_SIZE",
 "code": 39,
 "description": "Function Code size is more than the configured limit",
 "attributes": null,
 "runtime_info": {
  "code": 39,
  "info": "Function: TestLargeHandler_function handler Code size is more than 131072. Code Size: 131768"
 }
}
2021/12/14 06:02:32 Changed force_compress value to true
2021/12/14 06:02:32 Deploying app: TestLargeHandler_function. Item count src bucket: 0 dst bucket: 0
2021/12/14 06:02:33 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestLargeHandler_function' in metakv",
  "warnings": null
 }
}
2021/12/14 06:02:38 Waiting for app: TestLargeHandler_function to get deployed
2021/12/14 06:02:43 Waiting for app: TestLargeHandler_function to get deployed
2021/12/14 06:02:48 Waiting for app: TestLargeHandler_function to get deployed
2021/12/14 06:02:53 Waiting for app: TestLargeHandler_function to get deployed
2021/12/14 06:02:53 App: TestLargeHandler_function got deployed
2021/12/14 06:02:54 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 06:02:59 Waiting for app: TestLargeHandler_function to get un-deployed
2021/12/14 06:03:04 Waiting for app: TestLargeHandler_function to get un-deployed
2021/12/14 06:03:09 Waiting for app: TestLargeHandler_function to get un-deployed
2021/12/14 06:03:14 Waiting for app: TestLargeHandler_function to get un-deployed
2021/12/14 06:03:14 App: TestLargeHandler_function got un-deployed
2021/12/14 06:03:14 No eventing-con process running
2021/12/14 06:03:14 Function: TestLargeHandler_function delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestLargeHandler_function deleting in the background"
}
2021/12/14 06:03:15 Context: Bucket flush, response code: 200 dump: 
2021/12/14 06:03:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 06:03:22 Context: Bucket flush, response code: 200 dump: 
2021/12/14 06:03:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestLargeHandler (55.37s)
=== RUN   TestAllowInterHandlerRecursion
2021/12/14 06:03:37 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/14 06:03:37 App: TestAllowInterHandlerRecursion_source_bucket_update_op got un-deployed
2021/12/14 06:03:37 No eventing-con process running
2021/12/14 06:03:37 Function: TestAllowInterHandlerRecursion_source_bucket_update_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestAllowInterHandlerRecursion_source_bucket_update_op not found"
 }
}
2021/12/14 06:03:38 Context: Bucket flush, response code: 200 dump: 
2021/12/14 06:03:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 06:03:45 Context: Bucket flush, response code: 200 dump: 
2021/12/14 06:03:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 06:03:55 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/14 06:03:55 App: TestAllowInterHandlerRecursion_src_bucket_op_on_delete got un-deployed
2021/12/14 06:03:55 No eventing-con process running
2021/12/14 06:03:55 Function: TestAllowInterHandlerRecursion_src_bucket_op_on_delete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestAllowInterHandlerRecursion_src_bucket_op_on_delete not found"
 }
}
2021/12/14 06:03:56 Context: Bucket flush, response code: 200 dump: 
2021/12/14 06:04:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 06:04:03 Context: Bucket flush, response code: 200 dump: 
2021/12/14 06:04:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 06:04:08 Deploying app: TestAllowInterHandlerRecursion_source_bucket_update_op. Item count src bucket: 0 dst bucket: 0
2021/12/14 06:04:09 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterHandlerRecursion_source_bucket_update_op' in metakv",
  "warnings": null
 }
}
2021/12/14 06:04:09 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterHandlerRecursion_source_bucket_update_op' in metakv",
  "warnings": null
 }
} err 
2021/12/14 06:04:14 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/14 06:04:19 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/14 06:04:24 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/14 06:04:29 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/14 06:04:29 App: TestAllowInterHandlerRecursion_source_bucket_update_op got deployed
2021/12/14 06:04:29 Deploying app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete. Item count src bucket: 0 dst bucket: 0
2021/12/14 06:04:30 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterHandlerRecursion_src_bucket_op_on_delete' in metakv",
  "warnings": null
 }
}
2021/12/14 06:04:30 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterHandlerRecursion_src_bucket_op_on_delete' in metakv",
  "warnings": null
 }
} err 
2021/12/14 06:04:35 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get deployed
2021/12/14 06:04:40 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get deployed
2021/12/14 06:04:45 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get deployed
2021/12/14 06:04:50 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get deployed
2021/12/14 06:04:50 App: TestAllowInterHandlerRecursion_src_bucket_op_on_delete got deployed
2021/12/14 06:04:50 TestAllowInterHandlerRecursion_src_bucket_op_on_delete is deployed
2021/12/14 06:04:51 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 06:04:56 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/14 06:05:01 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/14 06:05:06 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/14 06:05:11 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/14 06:05:11 App: TestAllowInterHandlerRecursion_source_bucket_update_op got un-deployed
2021/12/14 06:05:11 No eventing-con process running
2021/12/14 06:05:11 Function: TestAllowInterHandlerRecursion_source_bucket_update_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAllowInterHandlerRecursion_source_bucket_update_op deleting in the background"
}
2021/12/14 06:05:13 Context: Bucket flush, response code: 200 dump: 
2021/12/14 06:05:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 06:05:20 Context: Bucket flush, response code: 200 dump: 
2021/12/14 06:05:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 06:05:25 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 06:05:30 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/14 06:05:35 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/14 06:05:40 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/14 06:05:45 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/14 06:05:45 App: TestAllowInterHandlerRecursion_src_bucket_op_on_delete got un-deployed
2021/12/14 06:05:45 No eventing-con process running
2021/12/14 06:05:45 Function: TestAllowInterHandlerRecursion_src_bucket_op_on_delete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAllowInterHandlerRecursion_src_bucket_op_on_delete deleting in the background"
}
2021/12/14 06:05:46 Context: Bucket flush, response code: 200 dump: 
2021/12/14 06:05:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 06:05:53 Context: Bucket flush, response code: 200 dump: 
2021/12/14 06:05:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAllowInterHandlerRecursion (151.33s)
=== RUN   TestAllowInterBucketRecursion
2021/12/14 06:06:03 Deploying app: TestAllowInterBucketRecursion_function1. Item count src bucket: 0 dst bucket: 0
2021/12/14 06:06:04 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterBucketRecursion_function1' in metakv",
  "warnings": null
 }
}
2021/12/14 06:06:04 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterBucketRecursion_function1' in metakv",
  "warnings": null
 }
} err 
2021/12/14 06:06:09 Waiting for app: TestAllowInterBucketRecursion_function1 to get deployed
2021/12/14 06:06:14 Waiting for app: TestAllowInterBucketRecursion_function1 to get deployed
2021/12/14 06:06:19 Waiting for app: TestAllowInterBucketRecursion_function1 to get deployed
2021/12/14 06:06:24 Waiting for app: TestAllowInterBucketRecursion_function1 to get deployed
2021/12/14 06:06:24 App: TestAllowInterBucketRecursion_function1 got deployed
2021/12/14 06:06:24 Deploying app: TestAllowInterBucketRecursion_function2. Item count src bucket: 0 dst bucket: 0
2021/12/14 06:06:25 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterBucketRecursion_function2' in metakv",
  "warnings": null
 }
}
2021/12/14 06:06:25 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterBucketRecursion_function2' in metakv",
  "warnings": null
 }
} err 
2021/12/14 06:06:30 Waiting for app: TestAllowInterBucketRecursion_function2 to get deployed
2021/12/14 06:06:36 Waiting for app: TestAllowInterBucketRecursion_function2 to get deployed
2021/12/14 06:06:41 Waiting for app: TestAllowInterBucketRecursion_function2 to get deployed
2021/12/14 06:06:46 Waiting for app: TestAllowInterBucketRecursion_function2 to get deployed
2021/12/14 06:06:46 App: TestAllowInterBucketRecursion_function2 got deployed
2021/12/14 06:06:46 TestAllowInterBucketRecursion_function2 is deployed
2021/12/14 06:06:46 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 06:06:51 Waiting for app: TestAllowInterBucketRecursion_function1 to get un-deployed
2021/12/14 06:06:56 Waiting for app: TestAllowInterBucketRecursion_function1 to get un-deployed
2021/12/14 06:07:01 Waiting for app: TestAllowInterBucketRecursion_function1 to get un-deployed
2021/12/14 06:07:06 Waiting for app: TestAllowInterBucketRecursion_function1 to get un-deployed
2021/12/14 06:07:06 App: TestAllowInterBucketRecursion_function1 got un-deployed
2021/12/14 06:07:06 No eventing-con process running
2021/12/14 06:07:06 Function: TestAllowInterBucketRecursion_function1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAllowInterBucketRecursion_function1 deleting in the background"
}
2021/12/14 06:07:07 Context: Bucket flush, response code: 200 dump: 
2021/12/14 06:07:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 06:07:15 Context: Bucket flush, response code: 200 dump: 
2021/12/14 06:07:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 06:07:20 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 06:07:25 Waiting for app: TestAllowInterBucketRecursion_function2 to get un-deployed
2021/12/14 06:07:30 Waiting for app: TestAllowInterBucketRecursion_function2 to get un-deployed
2021/12/14 06:07:35 Waiting for app: TestAllowInterBucketRecursion_function2 to get un-deployed
2021/12/14 06:07:40 Waiting for app: TestAllowInterBucketRecursion_function2 to get un-deployed
2021/12/14 06:07:40 App: TestAllowInterBucketRecursion_function2 got un-deployed
2021/12/14 06:07:40 No eventing-con process running
2021/12/14 06:07:40 Function: TestAllowInterBucketRecursion_function2 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAllowInterBucketRecursion_function2 deleting in the background"
}
2021/12/14 06:07:41 Context: Bucket flush, response code: 200 dump: 
2021/12/14 06:07:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 06:07:48 Context: Bucket flush, response code: 200 dump: 
2021/12/14 06:07:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAllowInterBucketRecursion (115.00s)
=== RUN   TestInterBucketRecursion2
2021/12/14 06:07:59 Context: Bucket flush, response code: 200 dump: 
2021/12/14 06:08:05 Context: Bucket flush, response code: 200 dump: 
2021/12/14 06:08:11 Context: Bucket flush, response code: 200 dump: 
2021/12/14 06:08:18 Context: Bucket flush, response code: 200 dump: 
2021/12/14 06:08:23 Deploying app: TestInterBucketRecursion2_function1. Item count src bucket: 0 dst bucket: 0
2021/12/14 06:08:24 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterBucketRecursion2_function1' in metakv",
  "warnings": null
 }
}
2021/12/14 06:08:29 Waiting for app: TestInterBucketRecursion2_function1 to get deployed
2021/12/14 06:08:34 Waiting for app: TestInterBucketRecursion2_function1 to get deployed
2021/12/14 06:08:39 Waiting for app: TestInterBucketRecursion2_function1 to get deployed
2021/12/14 06:08:44 Waiting for app: TestInterBucketRecursion2_function1 to get deployed
2021/12/14 06:08:44 App: TestInterBucketRecursion2_function1 got deployed
2021/12/14 06:08:44 Deploying app: TestInterBucketRecursion2_function2. Item count src bucket: 0 dst bucket: 0
2021/12/14 06:08:45 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterBucketRecursion2_function2' in metakv",
  "warnings": null
 }
}
2021/12/14 06:08:50 Waiting for app: TestInterBucketRecursion2_function2 to get deployed
2021/12/14 06:08:55 Waiting for app: TestInterBucketRecursion2_function2 to get deployed
2021/12/14 06:09:00 Waiting for app: TestInterBucketRecursion2_function2 to get deployed
2021/12/14 06:09:00 App: TestInterBucketRecursion2_function2 got deployed
2021/12/14 06:09:00 Deploying app: TestInterBucketRecursion2_function3. Item count src bucket: 0 dst bucket: 0
2021/12/14 06:09:01 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterBucketRecursion2_function3' in metakv",
  "warnings": null
 }
}
2021/12/14 06:09:06 Waiting for app: TestInterBucketRecursion2_function3 to get deployed
2021/12/14 06:09:11 Waiting for app: TestInterBucketRecursion2_function3 to get deployed
2021/12/14 06:09:16 Waiting for app: TestInterBucketRecursion2_function3 to get deployed
2021/12/14 06:09:21 Waiting for app: TestInterBucketRecursion2_function3 to get deployed
2021/12/14 06:09:21 App: TestInterBucketRecursion2_function3 got deployed
2021/12/14 06:09:21 Deploying app: TestInterBucketRecursion2_function4. Item count src bucket: 0 dst bucket: 0
2021/12/14 06:09:21 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_INTER_BUCKET_RECURSION",
 "code": 51,
 "description": "Inter bucket recursion error, deployment of current handler will cause inter bucket recursion",
 "attributes": null,
 "runtime_info": {
  "code": 51,
  "info": "Inter bucket recursion error; function: TestInterBucketRecursion2_function4 causes a cycle involving functions: [TestInterBucketRecursion2_function1 TestInterBucketRecursion2_function2 TestInterBucketRecursion2_function3], hence deployment is disallowed"
 }
}
2021/12/14 06:09:21 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 06:09:26 Waiting for app: TestInterBucketRecursion2_function1 to get un-deployed
2021/12/14 06:09:31 Waiting for app: TestInterBucketRecursion2_function1 to get un-deployed
2021/12/14 06:09:36 Waiting for app: TestInterBucketRecursion2_function1 to get un-deployed
2021/12/14 06:09:41 Waiting for app: TestInterBucketRecursion2_function1 to get un-deployed
2021/12/14 06:09:41 App: TestInterBucketRecursion2_function1 got un-deployed
2021/12/14 06:09:41 No eventing-con process running
2021/12/14 06:09:41 Function: TestInterBucketRecursion2_function1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestInterBucketRecursion2_function1 deleting in the background"
}
2021/12/14 06:09:43 Context: Bucket flush, response code: 200 dump: 
2021/12/14 06:09:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 06:09:50 Context: Bucket flush, response code: 200 dump: 
2021/12/14 06:09:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 06:09:55 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 06:10:00 Waiting for app: TestInterBucketRecursion2_function2 to get un-deployed
2021/12/14 06:10:05 Waiting for app: TestInterBucketRecursion2_function2 to get un-deployed
2021/12/14 06:10:10 Waiting for app: TestInterBucketRecursion2_function2 to get un-deployed
2021/12/14 06:10:15 Waiting for app: TestInterBucketRecursion2_function2 to get un-deployed
2021/12/14 06:10:15 App: TestInterBucketRecursion2_function2 got un-deployed
2021/12/14 06:10:15 No eventing-con process running
2021/12/14 06:10:15 Function: TestInterBucketRecursion2_function2 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestInterBucketRecursion2_function2 deleting in the background"
}
2021/12/14 06:10:16 Context: Bucket flush, response code: 200 dump: 
2021/12/14 06:10:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 06:10:23 Context: Bucket flush, response code: 200 dump: 
2021/12/14 06:10:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 06:10:28 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 06:10:33 Waiting for app: TestInterBucketRecursion2_function3 to get un-deployed
2021/12/14 06:10:38 Waiting for app: TestInterBucketRecursion2_function3 to get un-deployed
2021/12/14 06:10:43 Waiting for app: TestInterBucketRecursion2_function3 to get un-deployed
2021/12/14 06:10:48 Waiting for app: TestInterBucketRecursion2_function3 to get un-deployed
2021/12/14 06:10:48 App: TestInterBucketRecursion2_function3 got un-deployed
2021/12/14 06:10:48 No eventing-con process running
2021/12/14 06:10:48 Function: TestInterBucketRecursion2_function3 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestInterBucketRecursion2_function3 deleting in the background"
}
2021/12/14 06:10:50 Context: Bucket flush, response code: 200 dump: 
2021/12/14 06:10:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 06:10:56 Context: Bucket flush, response code: 200 dump: 
2021/12/14 06:11:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 06:11:06 Waiting for app: TestInterBucketRecursion2_function4 to get un-deployed
2021/12/14 06:11:06 App: TestInterBucketRecursion2_function4 got un-deployed
2021/12/14 06:11:06 No eventing-con process running
2021/12/14 06:11:06 Function: TestInterBucketRecursion2_function4 delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestInterBucketRecursion2_function4 not found"
 }
}
2021/12/14 06:11:08 Context: Bucket flush, response code: 200 dump: 
2021/12/14 06:11:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 06:11:15 Context: Bucket flush, response code: 200 dump: 
2021/12/14 06:11:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestInterBucketRecursion2 (208.38s)
=== RUN   TestBucketDeleteWithRebOut
2021/12/14 06:11:27 Removing node(s): [127.0.0.1:19001 127.0.0.1:19002 127.0.0.1:19003] from the cluster
2021/12/14 06:11:32 Rebalance progress: 100

VB distribution across eventing nodes and their workers::

2021/12/14 06:11:35 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:57} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/14 06:11:50 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:57} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/14 06:12:05 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:57} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/14 06:12:20 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:57} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/14 06:12:35 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:57} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/14 06:12:50 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:57} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/14 06:13:05 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:57} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/14 06:13:20 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:57} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/14 06:13:35 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:57} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/14 06:13:50 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:57} {Name:#primary Bucket:hello-world Completion:100}]

DCP Stream statuses:

vbucket curr owner:


2021/12/14 06:14:05 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 06:14:05 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/14 06:14:05 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/14 06:14:10 Rebalance progress: 100

VB distribution across eventing nodes and their workers::


DCP Stream statuses:

vbucket curr owner:


2021/12/14 06:14:10 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 06:14:10 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/14 06:14:10 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/14 06:14:15 Rebalance progress: 100

VB distribution across eventing nodes and their workers::


DCP Stream statuses:

vbucket curr owner:


2021/12/14 06:14:15 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 06:14:20 clusterSetup.go:397: Waiting for app: TestBucketDeleteWithRebOut to get un-deployed
2021/12/14 06:14:20 clusterSetup.go:405: App: TestBucketDeleteWithRebOut got un-deployed
2021/12/14 06:14:20 eventingSetup.go:46: No eventing-con process running
2021/12/14 06:14:20 eventingSetup.go:418: Function: TestBucketDeleteWithRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestBucketDeleteWithRebOut not found"
 }
}
2021/12/14 06:14:22 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 06:14:27 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 06:14:29 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 06:14:34 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 06:14:39 eventingSetup.go:125: Deploying app: TestBucketDeleteWithRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/14 06:14:40 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestBucketDeleteWithRebOut' in metakv",
  "warnings": null
 }
}
2021/12/14 06:14:45 clusterSetup.go:346: Waiting for app: TestBucketDeleteWithRebOut to get deployed
2021/12/14 06:14:50 clusterSetup.go:346: Waiting for app: TestBucketDeleteWithRebOut to get deployed
2021/12/14 06:14:55 clusterSetup.go:346: Waiting for app: TestBucketDeleteWithRebOut to get deployed
2021/12/14 06:15:00 clusterSetup.go:346: Waiting for app: TestBucketDeleteWithRebOut to get deployed
2021/12/14 06:15:00 clusterSetup.go:354: App: TestBucketDeleteWithRebOut got deployed

VB distribution across eventing nodes and their workers::


DCP Stream statuses:
	stream status: running
	len: 1024
	vb list dump: [0-1023]

vbucket curr owner:
Producer node: 127.0.0.1:9303	Node UUID: 8b53aa8d83193f40bdd4fac7e03faf52
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [342-455]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [456-569]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 113
	v: [570-682]
Producer node: 127.0.0.1:9302	Node UUID: 98dad925ce8d8297216ed3d37d695f1e
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [0-113]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [114-227]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 114
	v: [228-341]
Producer node: 172.31.2.134:9300	Node UUID: 3fa1fcb4ff18bc159369719de409a268
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [797-910]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 113
	v: [911-1023]
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [683-796]


2021/12/14 06:15:00 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 06:15:00 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/14 06:15:05 Rebalance progress: 29.16666666666666
2021/12/14 06:15:10 Rebalance progress: 29.16666666666666
2021/12/14 06:15:15 Rebalance progress: 100

VB distribution across eventing nodes and their workers::


DCP Stream statuses:
	stream status: running
	len: 1024
	vb list dump: [0-1023]

vbucket curr owner:
Producer node: 127.0.0.1:9303	Node UUID: 8b53aa8d83193f40bdd4fac7e03faf52
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [456-569]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 113
	v: [570-682]
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [342-455]
Producer node: 127.0.0.1:9302	Node UUID: 98dad925ce8d8297216ed3d37d695f1e
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [0-113]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [114-227]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 114
	v: [228-341]
Producer node: 172.31.2.134:9300	Node UUID: 3fa1fcb4ff18bc159369719de409a268
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 113
	v: [911-1023]
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [683-796]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [797-910]


2021/12/14 06:15:15 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 06:15:15 handler_test.go:1674: Deleting source bucket: default
2021/12/14 06:15:17 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/14 06:15:17 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/14 06:15:22 Rebalance progress: 29.16666666666666
2021/12/14 06:15:27 Rebalance progress: 29.16666666666666
2021/12/14 06:15:32 Rebalance progress: 100

VB distribution across eventing nodes and their workers::


DCP Stream statuses:
	stream status: running
	len: 894
	vb list dump: [0-7, 9-14, 16-18, 20-23, 25-31, 33-45, 48-51, 53-56, 58-64, 66-69, 71, 73-76, 78-87, 89-103, 105-107, 109-133, 135-138, 140-141, 143, 145-150, 152-156, 159, 161-177, 179-191, 193-195, 197-221, 223-225, 227-231, 233-235, 237-239, 241-250, 252-256, 258, 260-267, 269, 271-278, 280-299, 301-308, 310-323, 325, 328, 330-331, 333, 335-346, 348, 351-374, 377-384, 387-392, 394-403, 405-412, 415-417, 419-434, 436-440, 442-456, 458-461, 463-465, 467-470, 472-480, 482-493, 496-497, 500-503, 505-506, 508, 511-576, 578-586, 588-595, 597-610, 613-615, 617-623, 625-633, 635-651, 653-658, 660-661, 663, 665-682, 684, 686-691, 693-694, 696-700, 702-705, 707-713, 715-731, 733-744, 746-747, 749-753, 755-758, 760-769, 772-802, 804-813, 815, 817-819, 821-839, 841, 843-844, 846-848, 850-862, 864, 866, 868-875, 877-908, 910-913, 915-921, 923-924, 926-934, 936-938, 940-948, 950-954, 956-959, 961-964, 966-967, 969-976, 978-980, 982-994, 996-998, 1000-1007, 1010-1011, 1014-1023]
	stream status: stopped
	len: 130
	vb list dump: [8, 15, 19, 24, 32, 46-47, 52, 57, 65, 70, 72, 77, 88, 104, 108, 134, 139, 142, 144, 151, 157-158, 160, 178, 192, 196, 222, 226, 232, 236, 240, 251, 257, 259, 268, 270, 279, 300, 309, 324, 326-327, 329, 332, 334, 347, 349-350, 375-376, 385-386, 393, 404, 413-414, 418, 435, 441, 457, 462, 466, 471, 481, 494-495, 498-499, 504, 507, 509-510, 577, 587, 596, 611-612, 616, 624, 634, 652, 659, 662, 664, 683, 685, 692, 695, 701, 706, 714, 732, 745, 748, 754, 759, 770-771, 803, 814, 816, 820, 840, 842, 845, 849, 863, 865, 867, 876, 909, 914, 922, 925, 935, 939, 949, 955, 960, 965, 968, 977, 981, 995, 999, 1008-1009, 1012-1013]

vbucket curr owner:
Producer node: 127.0.0.1:9302	Node UUID: 98dad925ce8d8297216ed3d37d695f1e
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 98
	v: [0-7, 9-14, 16-18, 20-23, 25-31, 33-45, 48-51, 53-56, 58-64, 66-69, 71, 73-76, 78-87, 89-103, 105-107, 109-113]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 101
	v: [114-133, 135-138, 140-141, 143, 145-150, 152-156, 159, 161-177, 179-191, 193-195, 197-221, 223-225, 227]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 97
	v: [228-231, 233-235, 237-239, 241-250, 252-256, 258, 260-267, 269, 271-278, 280-299, 301-308, 310-323, 325, 328, 330-331, 333, 335-341]
Producer node: 172.31.2.134:9300	Node UUID: 3fa1fcb4ff18bc159369719de409a268
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 101
	v: [797-802, 804-813, 815, 817-819, 821-839, 841, 843-844, 846-848, 850-862, 864, 866, 868-875, 877-908, 910]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 95
	v: [911-913, 915-921, 923-924, 926-934, 936-938, 940-948, 950-954, 956-959, 961-964, 966-967, 969-976, 978-980, 982-994, 996-998, 1000-1007, 1010-1011, 1014-1023]
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 100
	v: [684, 686-691, 693-694, 696-700, 702-705, 707-713, 715-731, 733-744, 746-747, 749-753, 755-758, 760-769, 772-796]
Producer node: 127.0.0.1:9303	Node UUID: 8b53aa8d83193f40bdd4fac7e03faf52
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 101
	v: [456, 458-461, 463-465, 467-470, 472-480, 482-493, 496-497, 500-503, 505-506, 508, 511-569]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 101
	v: [570-576, 578-586, 588-595, 597-610, 613-615, 617-623, 625-633, 635-651, 653-658, 660-661, 663, 665-682]
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 100
	v: [342-346, 348, 351-374, 377-384, 387-392, 394-403, 405-412, 415-417, 419-434, 436-440, 442-455]


2021/12/14 06:15:32 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 06:15:37 eventingSetup.go:719: Waiting for function: TestBucketDeleteWithRebOut status to change to undeployed
2021/12/14 06:15:37 eventingSetup.go:727: Function: TestBucketDeleteWithRebOut status changed to undeployed
2021/12/14 06:15:42 clusterSetup.go:397: Waiting for app: TestBucketDeleteWithRebOut to get un-deployed
2021/12/14 06:15:43 clusterSetup.go:405: App: TestBucketDeleteWithRebOut got un-deployed
2021/12/14 06:15:43 eventingSetup.go:46: No eventing-con process running
2021/12/14 06:15:43 eventingSetup.go:418: Function: TestBucketDeleteWithRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestBucketDeleteWithRebOut deleting in the background"
}
2021/12/14 06:15:44 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 06:15:49 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 06:15:51 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 06:15:56 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 06:15:56 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv,index,n1ql to the cluster
2021/12/14 06:15:56 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/14 06:16:01 Rebalance progress: 100
--- PASS: TestBucketDeleteWithRebOut (282.24s)
=== RUN   TestN1QLRecursion
2021/12/14 06:16:04 Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/14 06:16:04 Error in adding nodes : , response: ["Prepare join failed. Node is already part of cluster."]
2021/12/14 06:16:04 Removing node(s): [] from the cluster
2021/12/14 06:16:09 Rebalance progress: 100
2021/12/14 06:16:15 Context: Bucket flush, response code: 200 dump: 
2021/12/14 06:16:21 Context: Bucket flush, response code: 200 dump: 
2021/12/14 06:16:27 Context: Bucket flush, response code: 200 dump: 
2021/12/14 06:16:32 Deploying app: TestN1QLRecursion_function1. Item count src bucket: 0 dst bucket: 0
2021/12/14 06:16:33 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLRecursion_function1' in metakv",
  "warnings": null
 }
}
2021/12/14 06:16:33 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLRecursion_function1' in metakv",
  "warnings": null
 }
} err 
2021/12/14 06:16:38 Waiting for app: TestN1QLRecursion_function1 to get deployed
2021/12/14 06:16:43 Waiting for app: TestN1QLRecursion_function1 to get deployed
2021/12/14 06:16:48 Waiting for app: TestN1QLRecursion_function1 to get deployed
2021/12/14 06:16:53 Waiting for app: TestN1QLRecursion_function1 to get deployed
2021/12/14 06:16:53 App: TestN1QLRecursion_function1 got deployed
2021/12/14 06:16:53 TestN1QLRecursion_function1 is deployed
2021/12/14 06:16:53 Deploying app: TestN1QLRecursion_function3. Item count src bucket: 0 dst bucket: 0
2021/12/14 06:16:54 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLRecursion_function3' in metakv",
  "warnings": null
 }
}
2021/12/14 06:16:54 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLRecursion_function3' in metakv",
  "warnings": null
 }
} err 
2021/12/14 06:16:59 Waiting for app: TestN1QLRecursion_function3 to get deployed
2021/12/14 06:17:04 Waiting for app: TestN1QLRecursion_function3 to get deployed
2021/12/14 06:17:09 Waiting for app: TestN1QLRecursion_function3 to get deployed
2021/12/14 06:17:14 Waiting for app: TestN1QLRecursion_function3 to get deployed
2021/12/14 06:17:15 App: TestN1QLRecursion_function3 got deployed
2021/12/14 06:17:15 TestN1QLRecursion_function3 is deployed
2021/12/14 06:17:15 Deploying app: TestN1QLRecursion_function2. Item count src bucket: 0 dst bucket: 0
2021/12/14 06:17:15 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_INTER_BUCKET_RECURSION",
 "code": 51,
 "description": "Inter bucket recursion error, deployment of current handler will cause inter bucket recursion",
 "attributes": null,
 "runtime_info": {
  "code": 51,
  "info": "Inter bucket recursion error; function: TestN1QLRecursion_function2 causes a cycle involving functions: [TestN1QLRecursion_function3 TestN1QLRecursion_function1], hence deployment is disallowed"
 }
}
2021/12/14 06:17:15 response body {
 "name": "ERR_INTER_BUCKET_RECURSION",
 "code": 51,
 "description": "Inter bucket recursion error, deployment of current handler will cause inter bucket recursion",
 "attributes": null,
 "runtime_info": {
  "code": 51,
  "info": "Inter bucket recursion error; function: TestN1QLRecursion_function2 causes a cycle involving functions: [TestN1QLRecursion_function3 TestN1QLRecursion_function1], hence deployment is disallowed"
 }
} err 
2021/12/14 06:17:15 Success: TestN1QLRecursion_function2 is not deployed
2021/12/14 06:17:15 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 06:17:20 Waiting for app: TestN1QLRecursion_function1 to get un-deployed
2021/12/14 06:17:25 Waiting for app: TestN1QLRecursion_function1 to get un-deployed
2021/12/14 06:17:30 Waiting for app: TestN1QLRecursion_function1 to get un-deployed
2021/12/14 06:17:35 Waiting for app: TestN1QLRecursion_function1 to get un-deployed
2021/12/14 06:17:35 App: TestN1QLRecursion_function1 got un-deployed
2021/12/14 06:17:35 No eventing-con process running
2021/12/14 06:17:35 Function: TestN1QLRecursion_function1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLRecursion_function1 deleting in the background"
}
2021/12/14 06:17:37 Context: Bucket flush, response code: 200 dump: 
2021/12/14 06:17:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 06:17:43 Context: Bucket flush, response code: 200 dump: 
2021/12/14 06:17:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 06:17:49 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 06:17:54 Waiting for app: TestN1QLRecursion_function3 to get un-deployed
2021/12/14 06:17:59 Waiting for app: TestN1QLRecursion_function3 to get un-deployed
2021/12/14 06:18:04 Waiting for app: TestN1QLRecursion_function3 to get un-deployed
2021/12/14 06:18:09 Waiting for app: TestN1QLRecursion_function3 to get un-deployed
2021/12/14 06:18:09 App: TestN1QLRecursion_function3 got un-deployed
2021/12/14 06:18:09 No eventing-con process running
2021/12/14 06:18:09 Function: TestN1QLRecursion_function3 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLRecursion_function3 deleting in the background"
}
2021/12/14 06:18:11 Context: Bucket flush, response code: 200 dump: 
2021/12/14 06:18:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 06:18:17 Context: Bucket flush, response code: 200 dump: 
2021/12/14 06:18:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 06:18:24 Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/14 06:18:29 Rebalance progress: 100
--- PASS: TestN1QLRecursion (145.12s)
=== RUN   TestN1QLAllowRecursion
2021/12/14 06:18:29 Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/14 06:18:29 Error in adding nodes : , response: ["Prepare join failed. Node is already part of cluster."]
2021/12/14 06:18:29 Removing node(s): [] from the cluster
2021/12/14 06:18:34 Rebalance progress: 100
2021/12/14 06:18:41 Context: Bucket flush, response code: 200 dump: 
2021/12/14 06:18:46 Context: Bucket flush, response code: 200 dump: 
2021/12/14 06:18:52 Context: Bucket flush, response code: 200 dump: 
2021/12/14 06:18:57 Deploying app: TestN1QLAllowRecursion_function1. Item count src bucket: 0 dst bucket: 0
2021/12/14 06:18:58 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function1' in metakv",
  "warnings": null
 }
}
2021/12/14 06:18:58 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function1' in metakv",
  "warnings": null
 }
} err 
2021/12/14 06:19:03 Waiting for app: TestN1QLAllowRecursion_function1 to get deployed
2021/12/14 06:19:08 Waiting for app: TestN1QLAllowRecursion_function1 to get deployed
2021/12/14 06:19:13 Waiting for app: TestN1QLAllowRecursion_function1 to get deployed
2021/12/14 06:19:18 Waiting for app: TestN1QLAllowRecursion_function1 to get deployed
2021/12/14 06:19:18 App: TestN1QLAllowRecursion_function1 got deployed
2021/12/14 06:19:18 TestN1QLAllowRecursion_function1 is deployed
2021/12/14 06:19:18 Deploying app: TestN1QLAllowRecursion_function3. Item count src bucket: 0 dst bucket: 0
2021/12/14 06:19:20 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function3' in metakv",
  "warnings": null
 }
}
2021/12/14 06:19:20 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function3' in metakv",
  "warnings": null
 }
} err 
2021/12/14 06:19:25 Waiting for app: TestN1QLAllowRecursion_function3 to get deployed
2021/12/14 06:19:30 Waiting for app: TestN1QLAllowRecursion_function3 to get deployed
2021/12/14 06:19:35 Waiting for app: TestN1QLAllowRecursion_function3 to get deployed
2021/12/14 06:19:40 Waiting for app: TestN1QLAllowRecursion_function3 to get deployed
2021/12/14 06:19:40 App: TestN1QLAllowRecursion_function3 got deployed
2021/12/14 06:19:40 TestN1QLAllowRecursion_function3 is deployed
2021/12/14 06:19:40 Deploying app: TestN1QLAllowRecursion_function2. Item count src bucket: 0 dst bucket: 0
2021/12/14 06:19:41 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function2' in metakv",
  "warnings": null
 }
}
2021/12/14 06:19:41 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function2' in metakv",
  "warnings": null
 }
} err 
2021/12/14 06:19:46 Waiting for app: TestN1QLAllowRecursion_function2 to get deployed
2021/12/14 06:19:51 Waiting for app: TestN1QLAllowRecursion_function2 to get deployed
2021/12/14 06:19:56 Waiting for app: TestN1QLAllowRecursion_function2 to get deployed
2021/12/14 06:20:01 Waiting for app: TestN1QLAllowRecursion_function2 to get deployed
2021/12/14 06:20:01 App: TestN1QLAllowRecursion_function2 got deployed
2021/12/14 06:20:01 Success: TestN1QLAllowRecursion_function2 is deployed after allow_interbucket_recursion
2021/12/14 06:20:01 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 06:20:06 Waiting for app: TestN1QLAllowRecursion_function1 to get un-deployed
2021/12/14 06:20:11 Waiting for app: TestN1QLAllowRecursion_function1 to get un-deployed
2021/12/14 06:20:16 Waiting for app: TestN1QLAllowRecursion_function1 to get un-deployed
2021/12/14 06:20:21 Waiting for app: TestN1QLAllowRecursion_function1 to get un-deployed
2021/12/14 06:20:21 App: TestN1QLAllowRecursion_function1 got un-deployed
2021/12/14 06:20:21 No eventing-con process running
2021/12/14 06:20:21 Function: TestN1QLAllowRecursion_function1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLAllowRecursion_function1 deleting in the background"
}
2021/12/14 06:20:23 Context: Bucket flush, response code: 200 dump: 
2021/12/14 06:20:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 06:20:30 Context: Bucket flush, response code: 200 dump: 
2021/12/14 06:20:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 06:20:35 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 06:20:40 Waiting for app: TestN1QLAllowRecursion_function2 to get un-deployed
2021/12/14 06:20:45 Waiting for app: TestN1QLAllowRecursion_function2 to get un-deployed
2021/12/14 06:20:50 Waiting for app: TestN1QLAllowRecursion_function2 to get un-deployed
2021/12/14 06:20:55 Waiting for app: TestN1QLAllowRecursion_function2 to get un-deployed
2021/12/14 06:20:55 App: TestN1QLAllowRecursion_function2 got un-deployed
2021/12/14 06:20:55 No eventing-con process running
2021/12/14 06:20:55 Function: TestN1QLAllowRecursion_function2 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLAllowRecursion_function2 deleting in the background"
}
2021/12/14 06:20:57 Context: Bucket flush, response code: 200 dump: 
2021/12/14 06:21:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 06:21:04 Context: Bucket flush, response code: 200 dump: 
2021/12/14 06:21:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 06:21:09 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 06:21:14 Waiting for app: TestN1QLAllowRecursion_function3 to get un-deployed
2021/12/14 06:21:19 Waiting for app: TestN1QLAllowRecursion_function3 to get un-deployed
2021/12/14 06:21:24 Waiting for app: TestN1QLAllowRecursion_function3 to get un-deployed
2021/12/14 06:21:29 Waiting for app: TestN1QLAllowRecursion_function3 to get un-deployed
2021/12/14 06:21:29 App: TestN1QLAllowRecursion_function3 got un-deployed
2021/12/14 06:21:29 No eventing-con process running
2021/12/14 06:21:29 Function: TestN1QLAllowRecursion_function3 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLAllowRecursion_function3 deleting in the background"
}
2021/12/14 06:21:31 Context: Bucket flush, response code: 200 dump: 
2021/12/14 06:21:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 06:21:38 Context: Bucket flush, response code: 200 dump: 
2021/12/14 06:21:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 06:21:44 Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/14 06:21:49 Rebalance progress: 100
--- PASS: TestN1QLAllowRecursion (200.53s)
=== RUN   TestOnDeleteExpiryBucketOp
2021/12/14 06:21:49 Sleeping for some time
2021/12/14 06:21:59 Waiting for app: TestOnDeleteExpiryBucketOp to get un-deployed
2021/12/14 06:21:59 App: TestOnDeleteExpiryBucketOp got un-deployed
2021/12/14 06:21:59 No eventing-con process running
2021/12/14 06:21:59 Function: TestOnDeleteExpiryBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnDeleteExpiryBucketOp not found"
 }
}
2021/12/14 06:22:01 Context: Bucket flush, response code: 200 dump: 
2021/12/14 06:22:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 06:22:08 Context: Bucket flush, response code: 200 dump: 
2021/12/14 06:22:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 06:22:13 Deploying app: TestOnDeleteExpiryBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/14 06:22:14 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnDeleteExpiryBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/14 06:22:14 Starting bucket ops to source bucket
2021/12/14 06:23:04 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 06:23:09 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 06:23:14 Waiting for dst bucket item count to get to: 5000 curr count: 2500
2021/12/14 06:23:14 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/14 06:23:14 Starting bucket ops to source bucket
2021/12/14 06:23:15 Starting bucket ops to source bucket
2021/12/14 06:24:21 Waiting for dst bucket item count to get to: 7000 curr count: 5000
2021/12/14 06:24:21 src & dst bucket item count matched up. src bucket count: 7000 dst bucket count: 7000
2021/12/14 06:24:21 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 06:24:21 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 06:24:26 Waiting for app: TestOnDeleteExpiryBucketOp to get un-deployed
2021/12/14 06:24:31 Waiting for app: TestOnDeleteExpiryBucketOp to get un-deployed
2021/12/14 06:24:36 Waiting for app: TestOnDeleteExpiryBucketOp to get un-deployed
2021/12/14 06:24:41 Waiting for app: TestOnDeleteExpiryBucketOp to get un-deployed
2021/12/14 06:24:41 App: TestOnDeleteExpiryBucketOp got un-deployed
2021/12/14 06:24:41 No eventing-con process running
2021/12/14 06:24:41 Function: TestOnDeleteExpiryBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnDeleteExpiryBucketOp deleting in the background"
}
2021/12/14 06:24:43 Context: Bucket flush, response code: 200 dump: 
2021/12/14 06:24:53 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/14 06:24:58 Waiting for dst bucket item count to get to: 0 curr count: 501
2021/12/14 06:24:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 06:24:59 Context: Bucket flush, response code: 200 dump: 
2021/12/14 06:25:09 Waiting for dst bucket item count to get to: 0 curr count: 3500
2021/12/14 06:25:14 Waiting for dst bucket item count to get to: 0 curr count: 3500
2021/12/14 06:25:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnDeleteExpiryBucketOp (205.05s)
=== RUN   TestTimerOverWriteSameReference
2021/12/14 06:25:19 Waiting for app: TestTimerOverWriteSameReference to get un-deployed
2021/12/14 06:25:19 App: TestTimerOverWriteSameReference got un-deployed
2021/12/14 06:25:19 No eventing-con process running
2021/12/14 06:25:19 Function: TestTimerOverWriteSameReference delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestTimerOverWriteSameReference not found"
 }
}
2021/12/14 06:25:21 Context: Bucket flush, response code: 200 dump: 
2021/12/14 06:25:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 06:25:28 Context: Bucket flush, response code: 200 dump: 
2021/12/14 06:25:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 06:25:33 Deploying app: TestTimerOverWriteSameReference. Item count src bucket: 0 dst bucket: 0
2021/12/14 06:25:34 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestTimerOverWriteSameReference' in metakv",
  "warnings": null
 }
}
2021/12/14 06:25:39 Waiting for app: TestTimerOverWriteSameReference to get deployed
2021/12/14 06:25:44 Waiting for app: TestTimerOverWriteSameReference to get deployed
2021/12/14 06:25:49 Waiting for app: TestTimerOverWriteSameReference to get deployed
2021/12/14 06:25:54 Waiting for app: TestTimerOverWriteSameReference to get deployed
2021/12/14 06:25:54 App: TestTimerOverWriteSameReference got deployed
2021/12/14 06:26:04 Starting bucket ops to source bucket
2021/12/14 06:27:06 src & dst bucket item count matched up. src bucket count: 1 dst bucket count: 1
2021/12/14 06:27:16 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 06:27:16 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 06:27:21 Waiting for app: TestTimerOverWriteSameReference to get un-deployed
2021/12/14 06:27:26 Waiting for app: TestTimerOverWriteSameReference to get un-deployed
2021/12/14 06:27:31 Waiting for app: TestTimerOverWriteSameReference to get un-deployed
2021/12/14 06:27:36 Waiting for app: TestTimerOverWriteSameReference to get un-deployed
2021/12/14 06:27:36 App: TestTimerOverWriteSameReference got un-deployed
2021/12/14 06:27:36 No eventing-con process running
2021/12/14 06:27:36 Function: TestTimerOverWriteSameReference delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestTimerOverWriteSameReference deleting in the background"
}
2021/12/14 06:27:38 Context: Bucket flush, response code: 200 dump: 
2021/12/14 06:27:48 Waiting for dst bucket item count to get to: 0 curr count: 500
2021/12/14 06:27:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 06:27:49 Context: Bucket flush, response code: 200 dump: 
2021/12/14 06:27:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestTimerOverWriteSameReference (159.93s)
=== RUN   TestCancelTimerBucketop
2021/12/14 06:27:59 Waiting for app: TestCancelTimerBucketop to get un-deployed
2021/12/14 06:27:59 App: TestCancelTimerBucketop got un-deployed
2021/12/14 06:27:59 No eventing-con process running
2021/12/14 06:27:59 Function: TestCancelTimerBucketop delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestCancelTimerBucketop not found"
 }
}
2021/12/14 06:28:01 Context: Bucket flush, response code: 200 dump: 
2021/12/14 06:28:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 06:28:08 Context: Bucket flush, response code: 200 dump: 
2021/12/14 06:28:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 06:28:13 Deploying app: TestCancelTimerBucketop. Item count src bucket: 0 dst bucket: 0
2021/12/14 06:28:14 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCancelTimerBucketop' in metakv",
  "warnings": null
 }
}
2021/12/14 06:28:19 Waiting for app: TestCancelTimerBucketop to get deployed
2021/12/14 06:28:24 Waiting for app: TestCancelTimerBucketop to get deployed
2021/12/14 06:28:29 Waiting for app: TestCancelTimerBucketop to get deployed
2021/12/14 06:28:34 Waiting for app: TestCancelTimerBucketop to get deployed
2021/12/14 06:28:34 App: TestCancelTimerBucketop got deployed
2021/12/14 06:28:34 Starting bucket ops to source bucket
2021/12/14 06:28:41 Starting bucket ops to source bucket
2021/12/14 06:29:44 src & dst bucket item count matched up. src bucket count: 1000 dst bucket count: 1000
2021/12/14 06:29:44 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 06:29:44 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 06:29:49 Waiting for app: TestCancelTimerBucketop to get un-deployed
2021/12/14 06:29:54 Waiting for app: TestCancelTimerBucketop to get un-deployed
2021/12/14 06:29:59 Waiting for app: TestCancelTimerBucketop to get un-deployed
2021/12/14 06:30:04 Waiting for app: TestCancelTimerBucketop to get un-deployed
2021/12/14 06:30:04 App: TestCancelTimerBucketop got un-deployed
2021/12/14 06:30:04 No eventing-con process running
2021/12/14 06:30:04 Function: TestCancelTimerBucketop delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCancelTimerBucketop deleting in the background"
}
2021/12/14 06:30:06 Context: Bucket flush, response code: 200 dump: 
2021/12/14 06:30:16 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/14 06:30:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 06:30:18 Context: Bucket flush, response code: 200 dump: 
2021/12/14 06:30:28 Waiting for dst bucket item count to get to: 0 curr count: 514
2021/12/14 06:30:33 Waiting for dst bucket item count to get to: 0 curr count: 14
2021/12/14 06:30:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCancelTimerBucketop (158.32s)
=== RUN   TestJSExpiryDate
2021/12/14 06:30:38 Starting bucket ops to source bucket
2021/12/14 06:30:56 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/14 06:30:56 Deploying app: TestJSExpiryDate. Item count src bucket: 5000 dst bucket: 0
2021/12/14 06:30:57 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestJSExpiryDate' in metakv",
  "warnings": null
 }
}
2021/12/14 06:31:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 06:31:27 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 06:31:27 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 06:31:32 Waiting for app: TestJSExpiryDate to get un-deployed
2021/12/14 06:31:37 Waiting for app: TestJSExpiryDate to get un-deployed
2021/12/14 06:31:42 Waiting for app: TestJSExpiryDate to get un-deployed
2021/12/14 06:31:47 Waiting for app: TestJSExpiryDate to get un-deployed
2021/12/14 06:31:47 App: TestJSExpiryDate got un-deployed
2021/12/14 06:31:47 No eventing-con process running
2021/12/14 06:31:47 Function: TestJSExpiryDate delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestJSExpiryDate deleting in the background"
}
2021/12/14 06:31:49 Context: Bucket flush, response code: 200 dump: 
2021/12/14 06:31:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 06:31:56 Context: Bucket flush, response code: 200 dump: 
2021/12/14 06:32:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestJSExpiryDate (88.70s)
=== RUN   TestBinaryDoc
2021/12/14 06:32:06 Starting bucket ops to source bucket
2021/12/14 06:32:07 Deploying app: TestBinaryDoc. Item count src bucket: 0 dst bucket: 0
2021/12/14 06:32:08 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestBinaryDoc' in metakv",
  "warnings": null
 }
}
2021/12/14 06:32:08 Starting bucket ops to source bucket
2021/12/14 06:32:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 06:32:08 Deleting source bucket docs now
2021/12/14 06:32:08 Starting bucket ops to source bucket
2021/12/14 06:32:45 Waiting for dst bucket item count to get to: 32 curr count: 20
2021/12/14 06:32:50 Waiting for dst bucket item count to get to: 32 curr count: 31
2021/12/14 06:32:55 Waiting for dst bucket item count to get to: 32 curr count: 31
2021/12/14 06:32:55 src & dst bucket item count matched up. src bucket count: 32 dst bucket count: 32
2021/12/14 06:32:55 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 06:32:55 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 06:33:00 Waiting for app: TestBinaryDoc to get un-deployed
2021/12/14 06:33:05 Waiting for app: TestBinaryDoc to get un-deployed
2021/12/14 06:33:10 Waiting for app: TestBinaryDoc to get un-deployed
2021/12/14 06:33:15 Waiting for app: TestBinaryDoc to get un-deployed
2021/12/14 06:33:15 App: TestBinaryDoc got un-deployed
2021/12/14 06:33:15 No eventing-con process running
2021/12/14 06:33:15 Function: TestBinaryDoc delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestBinaryDoc deleting in the background"
}
2021/12/14 06:33:17 Context: Bucket flush, response code: 200 dump: 
2021/12/14 06:33:27 Waiting for dst bucket item count to get to: 0 curr count: 32
2021/12/14 06:33:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 06:33:29 Context: Bucket flush, response code: 200 dump: 
2021/12/14 06:33:39 Waiting for dst bucket item count to get to: 0 curr count: 30
2021/12/14 06:33:44 Waiting for dst bucket item count to get to: 0 curr count: 14
2021/12/14 06:33:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestBinaryDoc (102.43s)
=== RUN   TestConstantBindings
2021/12/14 06:33:44 Deploying app: TestConstantBindings. Item count src bucket: 0 dst bucket: 0
2021/12/14 06:33:45 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestConstantBindings' in metakv",
  "warnings": null
 }
}
2021/12/14 06:33:45 Starting bucket ops to source bucket
2021/12/14 06:33:51 Waiting for dst bucket item count to get to: 6000 curr count: 0
2021/12/14 06:33:56 Waiting for dst bucket item count to get to: 6000 curr count: 0
2021/12/14 06:34:01 Waiting for dst bucket item count to get to: 6000 curr count: 0
2021/12/14 06:34:06 Waiting for dst bucket item count to get to: 6000 curr count: 967
2021/12/14 06:34:11 Waiting for dst bucket item count to get to: 6000 curr count: 3967
2021/12/14 06:34:11 src & dst bucket item count matched up. src bucket count: 6000 dst bucket count: 6000
2021/12/14 06:34:11 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 06:34:11 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 06:34:16 Waiting for app: TestConstantBindings to get un-deployed
2021/12/14 06:34:21 Waiting for app: TestConstantBindings to get un-deployed
2021/12/14 06:34:26 Waiting for app: TestConstantBindings to get un-deployed
2021/12/14 06:34:31 Waiting for app: TestConstantBindings to get un-deployed
2021/12/14 06:34:31 App: TestConstantBindings got un-deployed
2021/12/14 06:34:31 No eventing-con process running
2021/12/14 06:34:31 Function: TestConstantBindings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestConstantBindings deleting in the background"
}
2021/12/14 06:34:33 Context: Bucket flush, response code: 200 dump: 
2021/12/14 06:34:43 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/14 06:34:48 Waiting for dst bucket item count to get to: 0 curr count: 500
2021/12/14 06:34:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 06:34:50 Context: Bucket flush, response code: 200 dump: 
2021/12/14 06:35:00 Waiting for dst bucket item count to get to: 0 curr count: 3000
2021/12/14 06:35:05 Waiting for dst bucket item count to get to: 0 curr count: 3000
2021/12/14 06:35:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestConstantBindings (80.75s)
=== RUN   TestCodeInsight
2021/12/14 06:35:10 Waiting for app: TestCodeInsight to get un-deployed
2021/12/14 06:35:10 App: TestCodeInsight got un-deployed
2021/12/14 06:35:10 No eventing-con process running
2021/12/14 06:35:10 Function: TestCodeInsight delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestCodeInsight not found"
 }
}
2021/12/14 06:35:11 Context: Bucket flush, response code: 200 dump: 
2021/12/14 06:35:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 06:35:18 Context: Bucket flush, response code: 200 dump: 
2021/12/14 06:35:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 06:35:23 Starting bucket ops to source bucket
2021/12/14 06:35:24 Deploying app: TestCodeInsight. Item count src bucket: 0 dst bucket: 0
2021/12/14 06:35:25 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCodeInsight' in metakv",
  "warnings": null
 }
}
2021/12/14 06:35:30 Waiting for app: TestCodeInsight to get deployed
2021/12/14 06:35:35 Waiting for app: TestCodeInsight to get deployed
2021/12/14 06:35:40 Waiting for app: TestCodeInsight to get deployed
2021/12/14 06:35:45 Waiting for app: TestCodeInsight to get deployed
2021/12/14 06:35:45 App: TestCodeInsight got deployed
2021/12/14 06:35:50 Waiting for dst bucket item count to get to: 1000 curr count: 562
2021/12/14 06:35:50 src & dst bucket item count matched up. src bucket count: 1000 dst bucket count: 1000
2021/12/14 06:35:50 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 06:35:55 Waiting for app: TestCodeInsight to get un-deployed
2021/12/14 06:36:00 Waiting for app: TestCodeInsight to get un-deployed
2021/12/14 06:36:05 Waiting for app: TestCodeInsight to get un-deployed
2021/12/14 06:36:10 Waiting for app: TestCodeInsight to get un-deployed
2021/12/14 06:36:10 App: TestCodeInsight got un-deployed
2021/12/14 06:36:10 No eventing-con process running
2021/12/14 06:36:10 Function: TestCodeInsight delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCodeInsight deleting in the background"
}
2021/12/14 06:36:12 Context: Bucket flush, response code: 200 dump: 
2021/12/14 06:36:22 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/14 06:36:27 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/14 06:36:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 06:36:29 Context: Bucket flush, response code: 200 dump: 
2021/12/14 06:36:39 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/14 06:36:44 Waiting for dst bucket item count to get to: 0 curr count: 500
2021/12/14 06:36:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCodeInsight (99.39s)
=== RUN   TestParserTransform
--- PASS: TestParserTransform (0.05s)
=== RUN   TestParserDetect
--- PASS: TestParserDetect (0.00s)
PASS
ok  	github.com/couchbase/eventing/tests/functional_tests	12695.872s

Functional tests took 3 hours, 31 minutes and 39 seconds.

N1ql tests

2021/12/14 06:36:53 Started n1ql
Post "http://127.0.0.1:9000/pools/default": EOF
Quota setup Post "http://127.0.0.1:9000/pools/default": EOF
2021/12/14 06:37:03 Adding node: https://127.0.0.1:19001 with role: kv,index,n1ql to the cluster
2021/12/14 06:37:04 Error in adding nodes : , response: ["Prepare join failed. Node is already part of cluster."]
2021/12/14 06:37:04 Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/14 06:37:05 Removing node(s): [] from the cluster
2021/12/14 06:37:10 Rebalance progress: 100
=== RUN   TestRecursiveMutationN1QL
2021/12/14 06:37:15 Waiting for app: TestRecursiveMutationN1QL to get un-deployed
2021/12/14 06:37:15 App: TestRecursiveMutationN1QL got un-deployed
2021/12/14 06:37:15 No eventing-con process running
2021/12/14 06:37:15 Function: TestRecursiveMutationN1QL delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestRecursiveMutationN1QL not found"
 }
}
2021/12/14 06:37:16 Context: Bucket flush, response code: 200 dump: 
2021/12/14 06:37:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 06:37:23 Context: Bucket flush, response code: 200 dump: 
2021/12/14 06:37:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 06:37:28 Deploying app: TestRecursiveMutationN1QL. Item count src bucket: 0 dst bucket: 0
2021/12/14 06:37:28 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_INTER_BUCKET_RECURSION",
 "code": 51,
 "description": "Inter bucket recursion error, deployment of current handler will cause inter bucket recursion",
 "attributes": null,
 "runtime_info": {
  "code": 51,
  "info": "Function: TestRecursiveMutationN1QL N1QL DML to source keyspace default:default"
 }
}
--- PASS: TestRecursiveMutationN1QL (18.64s)
=== RUN   TestFlexReset1
2021/12/14 06:37:33 Waiting for app: TestFlexReset1 to get un-deployed
2021/12/14 06:37:33 App: TestFlexReset1 got un-deployed
2021/12/14 06:37:33 No eventing-con process running
2021/12/14 06:37:33 Function: TestFlexReset1 delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestFlexReset1 not found"
 }
}
2021/12/14 06:37:35 Context: Bucket flush, response code: 200 dump: 
2021/12/14 06:37:40 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 06:37:42 Context: Bucket flush, response code: 200 dump: 
2021/12/14 06:37:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 06:37:47 Starting bucket ops to source bucket
2021/12/14 06:37:48 Deploying app: TestFlexReset1. Item count src bucket: 0 dst bucket: 0
2021/12/14 06:37:49 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestFlexReset1' in metakv",
  "warnings": null
 }
}
2021/12/14 06:37:54 Waiting for app: TestFlexReset1 to get deployed
2021/12/14 06:37:59 Waiting for app: TestFlexReset1 to get deployed
2021/12/14 06:38:04 Waiting for app: TestFlexReset1 to get deployed
2021/12/14 06:38:09 Waiting for app: TestFlexReset1 to get deployed
2021/12/14 06:38:09 App: TestFlexReset1 got deployed
2021/12/14 06:38:14 Waiting for dst bucket item count to get to: 1000 curr count: 500
2021/12/14 06:38:14 src & dst bucket item count matched up. src bucket count: 1000 dst bucket count: 1000
2021/12/14 06:38:14 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 06:38:14 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 06:38:19 Waiting for app: TestFlexReset1 to get un-deployed
2021/12/14 06:38:24 Waiting for app: TestFlexReset1 to get un-deployed
2021/12/14 06:38:29 Waiting for app: TestFlexReset1 to get un-deployed
2021/12/14 06:38:34 Waiting for app: TestFlexReset1 to get un-deployed
2021/12/14 06:38:34 App: TestFlexReset1 got un-deployed
2021/12/14 06:38:34 No eventing-con process running
2021/12/14 06:38:34 Function: TestFlexReset1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestFlexReset1 deleting in the background"
}
2021/12/14 06:38:36 Context: Bucket flush, response code: 200 dump: 
2021/12/14 06:38:46 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/14 06:38:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 06:38:47 Context: Bucket flush, response code: 200 dump: 
2021/12/14 06:38:57 Waiting for dst bucket item count to get to: 0 curr count: 500
2021/12/14 06:38:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestFlexReset1 (89.13s)
=== RUN   TestFlexReset2
2021/12/14 06:39:03 Waiting for app: TestFlexReset2 to get un-deployed
2021/12/14 06:39:03 App: TestFlexReset2 got un-deployed
2021/12/14 06:39:03 No eventing-con process running
2021/12/14 06:39:03 Function: TestFlexReset2 delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestFlexReset2 not found"
 }
}
2021/12/14 06:39:04 Context: Bucket flush, response code: 200 dump: 
2021/12/14 06:39:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 06:39:11 Context: Bucket flush, response code: 200 dump: 
2021/12/14 06:39:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 06:39:16 Starting bucket ops to source bucket
2021/12/14 06:39:17 Deploying app: TestFlexReset2. Item count src bucket: 0 dst bucket: 0
2021/12/14 06:39:18 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestFlexReset2' in metakv",
  "warnings": null
 }
}
2021/12/14 06:39:23 Waiting for app: TestFlexReset2 to get deployed
2021/12/14 06:39:28 Waiting for app: TestFlexReset2 to get deployed
2021/12/14 06:39:33 Waiting for app: TestFlexReset2 to get deployed
2021/12/14 06:39:38 Waiting for app: TestFlexReset2 to get deployed
2021/12/14 06:39:38 App: TestFlexReset2 got deployed
2021/12/14 06:39:43 Waiting for dst bucket item count to get to: 1000 curr count: 1
2021/12/14 06:39:43 src & dst bucket item count matched up. src bucket count: 1000 dst bucket count: 1000
2021/12/14 06:39:43 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 06:39:43 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 06:39:48 Waiting for app: TestFlexReset2 to get un-deployed
2021/12/14 06:39:53 Waiting for app: TestFlexReset2 to get un-deployed
2021/12/14 06:39:58 Waiting for app: TestFlexReset2 to get un-deployed
2021/12/14 06:40:03 Waiting for app: TestFlexReset2 to get un-deployed
2021/12/14 06:40:03 App: TestFlexReset2 got un-deployed
2021/12/14 06:40:03 No eventing-con process running
2021/12/14 06:40:03 Function: TestFlexReset2 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestFlexReset2 deleting in the background"
}
2021/12/14 06:40:05 Context: Bucket flush, response code: 200 dump: 
2021/12/14 06:40:15 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/14 06:40:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 06:40:17 Context: Bucket flush, response code: 200 dump: 
2021/12/14 06:40:27 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/14 06:40:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestFlexReset2 (89.14s)
=== RUN   TestN1QLLabelledBreak
2021/12/14 06:40:32 Waiting for app: TestN1QLLabelledBreak to get un-deployed
2021/12/14 06:40:32 App: TestN1QLLabelledBreak got un-deployed
2021/12/14 06:40:32 No eventing-con process running
2021/12/14 06:40:32 Function: TestN1QLLabelledBreak delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLLabelledBreak not found"
 }
}
2021/12/14 06:40:33 Context: Bucket flush, response code: 200 dump: 
2021/12/14 06:40:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 06:40:40 Context: Bucket flush, response code: 200 dump: 
2021/12/14 06:40:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 06:40:45 Deploying app: TestN1QLLabelledBreak. Item count src bucket: 0 dst bucket: 0
2021/12/14 06:40:46 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLLabelledBreak' in metakv",
  "warnings": null
 }
}
2021/12/14 06:40:51 Waiting for app: TestN1QLLabelledBreak to get deployed
2021/12/14 06:40:56 Waiting for app: TestN1QLLabelledBreak to get deployed
2021/12/14 06:41:01 Waiting for app: TestN1QLLabelledBreak to get deployed
2021/12/14 06:41:06 Waiting for app: TestN1QLLabelledBreak to get deployed
2021/12/14 06:41:06 App: TestN1QLLabelledBreak got deployed
2021/12/14 06:41:06 Starting bucket ops to source bucket
2021/12/14 06:41:16 Waiting for dst bucket item count to get to: 10000 curr count: 217
2021/12/14 06:41:21 Waiting for dst bucket item count to get to: 10000 curr count: 217
2021/12/14 06:41:26 Waiting for dst bucket item count to get to: 10000 curr count: 217
2021/12/14 06:41:31 Waiting for dst bucket item count to get to: 10000 curr count: 3533
2021/12/14 06:41:36 Waiting for dst bucket item count to get to: 10000 curr count: 3533
2021/12/14 06:41:41 Waiting for dst bucket item count to get to: 10000 curr count: 7559
2021/12/14 06:41:46 Waiting for dst bucket item count to get to: 10000 curr count: 7559
2021/12/14 06:41:46 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/14 06:41:46 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 06:41:46 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 06:41:51 Waiting for app: TestN1QLLabelledBreak to get un-deployed
2021/12/14 06:41:56 Waiting for app: TestN1QLLabelledBreak to get un-deployed
2021/12/14 06:42:01 Waiting for app: TestN1QLLabelledBreak to get un-deployed
2021/12/14 06:42:06 Waiting for app: TestN1QLLabelledBreak to get un-deployed
2021/12/14 06:42:06 App: TestN1QLLabelledBreak got un-deployed
2021/12/14 06:42:06 No eventing-con process running
2021/12/14 06:42:06 Function: TestN1QLLabelledBreak delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLLabelledBreak deleting in the background"
}
2021/12/14 06:42:08 Context: Bucket flush, response code: 200 dump: 
2021/12/14 06:42:18 Waiting for dst bucket item count to get to: 0 curr count: 3607
2021/12/14 06:42:23 Waiting for dst bucket item count to get to: 0 curr count: 1107
2021/12/14 06:42:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 06:42:25 Context: Bucket flush, response code: 200 dump: 
2021/12/14 06:42:35 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/14 06:42:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLLabelledBreak (128.25s)
=== RUN   TestN1QLUnlabelledBreak
2021/12/14 06:42:40 Waiting for app: TestN1QLUnlabelledBreak to get un-deployed
2021/12/14 06:42:40 App: TestN1QLUnlabelledBreak got un-deployed
2021/12/14 06:42:40 No eventing-con process running
2021/12/14 06:42:40 Function: TestN1QLUnlabelledBreak delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLUnlabelledBreak not found"
 }
}
2021/12/14 06:42:42 Context: Bucket flush, response code: 200 dump: 
2021/12/14 06:42:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 06:42:48 Context: Bucket flush, response code: 200 dump: 
2021/12/14 06:42:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 06:42:53 Deploying app: TestN1QLUnlabelledBreak. Item count src bucket: 0 dst bucket: 0
2021/12/14 06:42:55 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLUnlabelledBreak' in metakv",
  "warnings": null
 }
}
2021/12/14 06:43:00 Waiting for app: TestN1QLUnlabelledBreak to get deployed
2021/12/14 06:43:05 Waiting for app: TestN1QLUnlabelledBreak to get deployed
2021/12/14 06:43:10 Waiting for app: TestN1QLUnlabelledBreak to get deployed
2021/12/14 06:43:15 Waiting for app: TestN1QLUnlabelledBreak to get deployed
2021/12/14 06:43:15 App: TestN1QLUnlabelledBreak got deployed
2021/12/14 06:43:15 Starting bucket ops to source bucket
2021/12/14 06:43:24 Waiting for dst bucket item count to get to: 10000 curr count: 0
2021/12/14 06:43:29 Waiting for dst bucket item count to get to: 10000 curr count: 279
2021/12/14 06:43:34 Waiting for dst bucket item count to get to: 10000 curr count: 809
2021/12/14 06:43:39 Waiting for dst bucket item count to get to: 10000 curr count: 3297
2021/12/14 06:43:44 Waiting for dst bucket item count to get to: 10000 curr count: 6129
2021/12/14 06:43:49 Waiting for dst bucket item count to get to: 10000 curr count: 8362
2021/12/14 06:43:49 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/14 06:43:49 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 06:43:49 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 06:43:54 Waiting for app: TestN1QLUnlabelledBreak to get un-deployed
2021/12/14 06:43:59 Waiting for app: TestN1QLUnlabelledBreak to get un-deployed
2021/12/14 06:44:04 Waiting for app: TestN1QLUnlabelledBreak to get un-deployed
2021/12/14 06:44:09 Waiting for app: TestN1QLUnlabelledBreak to get un-deployed
2021/12/14 06:44:09 App: TestN1QLUnlabelledBreak got un-deployed
2021/12/14 06:44:09 No eventing-con process running
2021/12/14 06:44:09 Function: TestN1QLUnlabelledBreak delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLUnlabelledBreak deleting in the background"
}
2021/12/14 06:44:11 Context: Bucket flush, response code: 200 dump: 
2021/12/14 06:44:21 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/14 06:44:26 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/14 06:44:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 06:44:28 Context: Bucket flush, response code: 200 dump: 
2021/12/14 06:44:38 Waiting for dst bucket item count to get to: 0 curr count: 7134
2021/12/14 06:44:43 Waiting for dst bucket item count to get to: 0 curr count: 2134
2021/12/14 06:44:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLUnlabelledBreak (128.16s)
=== RUN   TestN1QLThrowStatement
2021/12/14 06:44:48 Waiting for app: TestN1QLThrowStatement to get un-deployed
2021/12/14 06:44:48 App: TestN1QLThrowStatement got un-deployed
2021/12/14 06:44:48 No eventing-con process running
2021/12/14 06:44:48 Function: TestN1QLThrowStatement delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLThrowStatement not found"
 }
}
2021/12/14 06:44:50 Context: Bucket flush, response code: 200 dump: 
2021/12/14 06:44:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 06:44:57 Context: Bucket flush, response code: 200 dump: 
2021/12/14 06:45:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 06:45:02 Deploying app: TestN1QLThrowStatement. Item count src bucket: 0 dst bucket: 0
2021/12/14 06:45:03 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLThrowStatement' in metakv",
  "warnings": null
 }
}
2021/12/14 06:45:08 Waiting for app: TestN1QLThrowStatement to get deployed
2021/12/14 06:45:13 Waiting for app: TestN1QLThrowStatement to get deployed
2021/12/14 06:45:18 Waiting for app: TestN1QLThrowStatement to get deployed
2021/12/14 06:45:23 Waiting for app: TestN1QLThrowStatement to get deployed
2021/12/14 06:45:23 App: TestN1QLThrowStatement got deployed
2021/12/14 06:45:23 Starting bucket ops to source bucket
2021/12/14 06:45:32 Waiting for dst bucket item count to get to: 10000 curr count: 0
2021/12/14 06:45:37 Waiting for dst bucket item count to get to: 10000 curr count: 0
2021/12/14 06:45:42 Waiting for dst bucket item count to get to: 10000 curr count: 966
2021/12/14 06:45:47 Waiting for dst bucket item count to get to: 10000 curr count: 966
2021/12/14 06:45:52 Waiting for dst bucket item count to get to: 10000 curr count: 4947
2021/12/14 06:45:57 Waiting for dst bucket item count to get to: 10000 curr count: 6973
2021/12/14 06:46:02 Waiting for dst bucket item count to get to: 10000 curr count: 8975
2021/12/14 06:46:07 Waiting for dst bucket item count to get to: 10000 curr count: 9695
2021/12/14 06:46:07 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/14 06:46:07 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 06:46:08 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 06:46:13 Waiting for app: TestN1QLThrowStatement to get un-deployed
2021/12/14 06:46:18 Waiting for app: TestN1QLThrowStatement to get un-deployed
2021/12/14 06:46:23 Waiting for app: TestN1QLThrowStatement to get un-deployed
2021/12/14 06:46:28 Waiting for app: TestN1QLThrowStatement to get un-deployed
2021/12/14 06:46:28 App: TestN1QLThrowStatement got un-deployed
2021/12/14 06:46:28 No eventing-con process running
2021/12/14 06:46:28 Function: TestN1QLThrowStatement delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLThrowStatement deleting in the background"
}
2021/12/14 06:46:29 Context: Bucket flush, response code: 200 dump: 
2021/12/14 06:46:39 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/14 06:46:44 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/14 06:46:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 06:46:46 Context: Bucket flush, response code: 200 dump: 
2021/12/14 06:46:56 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/14 06:46:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLThrowStatement (133.19s)
=== RUN   TestN1QLNestedForLoop
2021/12/14 06:47:01 Waiting for app: TestN1QLNestedForLoop to get un-deployed
2021/12/14 06:47:01 App: TestN1QLNestedForLoop got un-deployed
2021/12/14 06:47:01 No eventing-con process running
2021/12/14 06:47:01 Function: TestN1QLNestedForLoop delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLNestedForLoop not found"
 }
}
2021/12/14 06:47:03 Context: Bucket flush, response code: 200 dump: 
2021/12/14 06:47:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 06:47:10 Context: Bucket flush, response code: 200 dump: 
2021/12/14 06:47:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 06:47:15 Deploying app: TestN1QLNestedForLoop. Item count src bucket: 0 dst bucket: 0
2021/12/14 06:47:16 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLNestedForLoop' in metakv",
  "warnings": null
 }
}
2021/12/14 06:47:21 Waiting for app: TestN1QLNestedForLoop to get deployed
2021/12/14 06:47:26 Waiting for app: TestN1QLNestedForLoop to get deployed
2021/12/14 06:47:31 Waiting for app: TestN1QLNestedForLoop to get deployed
2021/12/14 06:47:36 Waiting for app: TestN1QLNestedForLoop to get deployed
2021/12/14 06:47:36 App: TestN1QLNestedForLoop got deployed
2021/12/14 06:47:36 Starting bucket ops to source bucket
2021/12/14 06:47:45 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 06:47:50 Waiting for dst bucket item count to get to: 5000 curr count: 152
2021/12/14 06:47:55 Waiting for dst bucket item count to get to: 5000 curr count: 152
2021/12/14 06:48:00 Waiting for dst bucket item count to get to: 5000 curr count: 2216
2021/12/14 06:48:05 Waiting for dst bucket item count to get to: 5000 curr count: 2216
2021/12/14 06:48:10 Waiting for dst bucket item count to get to: 5000 curr count: 4616
2021/12/14 06:48:15 Waiting for dst bucket item count to get to: 5000 curr count: 4616
2021/12/14 06:48:15 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/14 06:48:15 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 06:48:15 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 06:48:20 Waiting for app: TestN1QLNestedForLoop to get un-deployed
2021/12/14 06:48:25 Waiting for app: TestN1QLNestedForLoop to get un-deployed
2021/12/14 06:48:30 Waiting for app: TestN1QLNestedForLoop to get un-deployed
2021/12/14 06:48:35 Waiting for app: TestN1QLNestedForLoop to get un-deployed
2021/12/14 06:48:35 App: TestN1QLNestedForLoop got un-deployed
2021/12/14 06:48:35 No eventing-con process running
2021/12/14 06:48:36 Function: TestN1QLNestedForLoop delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLNestedForLoop deleting in the background"
}
2021/12/14 06:48:37 Context: Bucket flush, response code: 200 dump: 
2021/12/14 06:48:47 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/14 06:48:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 06:48:49 Context: Bucket flush, response code: 200 dump: 
2021/12/14 06:48:59 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/14 06:49:04 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/14 06:49:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLNestedForLoop (127.89s)
=== RUN   TestN1QLAutoClose
2021/12/14 06:49:09 Waiting for app: TestN1QLAutoClose to get un-deployed
2021/12/14 06:49:09 App: TestN1QLAutoClose got un-deployed
2021/12/14 06:49:09 No eventing-con process running
2021/12/14 06:49:09 Function: TestN1QLAutoClose delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLAutoClose not found"
 }
}
2021/12/14 06:49:11 Context: Bucket flush, response code: 200 dump: 
2021/12/14 06:49:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 06:49:18 Context: Bucket flush, response code: 200 dump: 
2021/12/14 06:49:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 06:49:23 Deploying app: TestN1QLAutoClose. Item count src bucket: 0 dst bucket: 0
2021/12/14 06:49:24 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAutoClose' in metakv",
  "warnings": null
 }
}
2021/12/14 06:49:29 Waiting for app: TestN1QLAutoClose to get deployed
2021/12/14 06:49:34 Waiting for app: TestN1QLAutoClose to get deployed
2021/12/14 06:49:39 Waiting for app: TestN1QLAutoClose to get deployed
2021/12/14 06:49:44 Waiting for app: TestN1QLAutoClose to get deployed
2021/12/14 06:49:44 App: TestN1QLAutoClose got deployed
2021/12/14 06:49:44 Starting bucket ops to source bucket
2021/12/14 06:49:49 Waiting for dst bucket item count to get to: 500 curr count: 0
2021/12/14 06:49:54 Waiting for dst bucket item count to get to: 500 curr count: 4
2021/12/14 06:49:59 Waiting for dst bucket item count to get to: 500 curr count: 8
2021/12/14 06:50:05 Waiting for dst bucket item count to get to: 500 curr count: 17
2021/12/14 06:50:10 Waiting for dst bucket item count to get to: 500 curr count: 29
2021/12/14 06:50:15 Waiting for dst bucket item count to get to: 500 curr count: 41
2021/12/14 06:50:20 Waiting for dst bucket item count to get to: 500 curr count: 53
2021/12/14 06:50:25 Waiting for dst bucket item count to get to: 500 curr count: 63
2021/12/14 06:50:30 Waiting for dst bucket item count to get to: 500 curr count: 74
2021/12/14 06:50:35 Waiting for dst bucket item count to get to: 500 curr count: 84
2021/12/14 06:50:40 Waiting for dst bucket item count to get to: 500 curr count: 95
2021/12/14 06:50:45 Waiting for dst bucket item count to get to: 500 curr count: 104
2021/12/14 06:50:50 Waiting for dst bucket item count to get to: 500 curr count: 117
2021/12/14 06:50:55 Waiting for dst bucket item count to get to: 500 curr count: 117
2021/12/14 06:51:00 Waiting for dst bucket item count to get to: 500 curr count: 138
2021/12/14 06:51:05 Waiting for dst bucket item count to get to: 500 curr count: 138
2021/12/14 06:51:10 Waiting for dst bucket item count to get to: 500 curr count: 159
2021/12/14 06:51:15 Waiting for dst bucket item count to get to: 500 curr count: 159
2021/12/14 06:51:20 Waiting for dst bucket item count to get to: 500 curr count: 181
2021/12/14 06:51:25 Waiting for dst bucket item count to get to: 500 curr count: 181
2021/12/14 06:51:30 Waiting for dst bucket item count to get to: 500 curr count: 204
2021/12/14 06:51:35 Waiting for dst bucket item count to get to: 500 curr count: 204
2021/12/14 06:51:40 Waiting for dst bucket item count to get to: 500 curr count: 226
2021/12/14 06:51:45 Waiting for dst bucket item count to get to: 500 curr count: 226
2021/12/14 06:51:50 Waiting for dst bucket item count to get to: 500 curr count: 244
2021/12/14 06:51:55 Waiting for dst bucket item count to get to: 500 curr count: 244
2021/12/14 06:52:00 Waiting for dst bucket item count to get to: 500 curr count: 265
2021/12/14 06:52:05 Waiting for dst bucket item count to get to: 500 curr count: 265
2021/12/14 06:52:10 Waiting for dst bucket item count to get to: 500 curr count: 287
2021/12/14 06:52:15 Waiting for dst bucket item count to get to: 500 curr count: 287
2021/12/14 06:52:20 Waiting for dst bucket item count to get to: 500 curr count: 309
2021/12/14 06:52:25 Waiting for dst bucket item count to get to: 500 curr count: 309
2021/12/14 06:52:30 Waiting for dst bucket item count to get to: 500 curr count: 332
2021/12/14 06:52:35 Waiting for dst bucket item count to get to: 500 curr count: 332
2021/12/14 06:52:40 Waiting for dst bucket item count to get to: 500 curr count: 354
2021/12/14 06:52:45 Waiting for dst bucket item count to get to: 500 curr count: 354
2021/12/14 06:52:50 Waiting for dst bucket item count to get to: 500 curr count: 373
2021/12/14 06:52:55 Waiting for dst bucket item count to get to: 500 curr count: 373
2021/12/14 06:53:00 Waiting for dst bucket item count to get to: 500 curr count: 395
2021/12/14 06:53:05 Waiting for dst bucket item count to get to: 500 curr count: 395
2021/12/14 06:53:10 Waiting for dst bucket item count to get to: 500 curr count: 415
2021/12/14 06:53:15 Waiting for dst bucket item count to get to: 500 curr count: 429
2021/12/14 06:53:20 Waiting for dst bucket item count to get to: 500 curr count: 438
2021/12/14 06:53:25 Waiting for dst bucket item count to get to: 500 curr count: 449
2021/12/14 06:53:30 Waiting for dst bucket item count to get to: 500 curr count: 459
2021/12/14 06:53:35 Waiting for dst bucket item count to get to: 500 curr count: 471
2021/12/14 06:53:40 Waiting for dst bucket item count to get to: 500 curr count: 481
2021/12/14 06:53:45 Waiting for dst bucket item count to get to: 500 curr count: 489
2021/12/14 06:53:50 Waiting for dst bucket item count to get to: 500 curr count: 498
2021/12/14 06:53:50 src & dst bucket item count matched up. src bucket count: 500 dst bucket count: 500
2021/12/14 06:53:50 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 06:53:50 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 06:53:55 Waiting for app: TestN1QLAutoClose to get un-deployed
2021/12/14 06:54:00 Waiting for app: TestN1QLAutoClose to get un-deployed
2021/12/14 06:54:05 Waiting for app: TestN1QLAutoClose to get un-deployed
2021/12/14 06:54:10 Waiting for app: TestN1QLAutoClose to get un-deployed
2021/12/14 06:54:10 App: TestN1QLAutoClose got un-deployed
2021/12/14 06:54:10 No eventing-con process running
2021/12/14 06:54:10 Function: TestN1QLAutoClose delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLAutoClose deleting in the background"
}
2021/12/14 06:54:12 Context: Bucket flush, response code: 200 dump: 
2021/12/14 06:54:22 Waiting for dst bucket item count to get to: 0 curr count: 500
2021/12/14 06:54:27 Waiting for dst bucket item count to get to: 0 curr count: 500
2021/12/14 06:54:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 06:54:29 Context: Bucket flush, response code: 200 dump: 
2021/12/14 06:54:39 Waiting for dst bucket item count to get to: 0 curr count: 452
2021/12/14 06:54:44 Waiting for dst bucket item count to get to: 0 curr count: 202
2021/12/14 06:54:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLAutoClose (339.54s)
=== RUN   TestN1QLPosParams
2021/12/14 06:54:49 Waiting for app: TestN1QLPosParams to get un-deployed
2021/12/14 06:54:49 App: TestN1QLPosParams got un-deployed
2021/12/14 06:54:49 No eventing-con process running
2021/12/14 06:54:49 Function: TestN1QLPosParams delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLPosParams not found"
 }
}
2021/12/14 06:54:50 Context: Bucket flush, response code: 200 dump: 
2021/12/14 06:54:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 06:54:57 Context: Bucket flush, response code: 200 dump: 
2021/12/14 06:55:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 06:55:02 Deploying app: TestN1QLPosParams. Item count src bucket: 0 dst bucket: 0
2021/12/14 06:55:03 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLPosParams' in metakv",
  "warnings": null
 }
}
2021/12/14 06:55:08 Waiting for app: TestN1QLPosParams to get deployed
2021/12/14 06:55:13 Waiting for app: TestN1QLPosParams to get deployed
2021/12/14 06:55:18 Waiting for app: TestN1QLPosParams to get deployed
2021/12/14 06:55:23 Waiting for app: TestN1QLPosParams to get deployed
2021/12/14 06:55:23 App: TestN1QLPosParams got deployed
2021/12/14 06:55:23 Starting bucket ops to source bucket
2021/12/14 06:55:33 Waiting for dst bucket item count to get to: 5000 curr count: 596
2021/12/14 06:55:38 Waiting for dst bucket item count to get to: 5000 curr count: 1788
2021/12/14 06:55:43 Waiting for dst bucket item count to get to: 5000 curr count: 3692
2021/12/14 06:55:43 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/14 06:55:43 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 06:55:43 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 06:55:48 Waiting for app: TestN1QLPosParams to get un-deployed
2021/12/14 06:55:53 Waiting for app: TestN1QLPosParams to get un-deployed
2021/12/14 06:55:58 Waiting for app: TestN1QLPosParams to get un-deployed
2021/12/14 06:56:03 Waiting for app: TestN1QLPosParams to get un-deployed
2021/12/14 06:56:03 App: TestN1QLPosParams got un-deployed
2021/12/14 06:56:03 No eventing-con process running
2021/12/14 06:56:03 Function: TestN1QLPosParams delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLPosParams deleting in the background"
}
2021/12/14 06:56:05 Context: Bucket flush, response code: 200 dump: 
2021/12/14 06:56:15 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/14 06:56:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 06:56:17 Context: Bucket flush, response code: 200 dump: 
2021/12/14 06:56:27 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/14 06:56:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLPosParams (103.36s)
=== RUN   TestN1QLExhaustConnPool
2021/12/14 06:56:32 Waiting for app: TestN1QLExhaustConnPool to get un-deployed
2021/12/14 06:56:32 App: TestN1QLExhaustConnPool got un-deployed
2021/12/14 06:56:32 No eventing-con process running
2021/12/14 06:56:32 Function: TestN1QLExhaustConnPool delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLExhaustConnPool not found"
 }
}
2021/12/14 06:56:34 Context: Bucket flush, response code: 200 dump: 
2021/12/14 06:56:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 06:56:41 Context: Bucket flush, response code: 200 dump: 
2021/12/14 06:56:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 06:56:46 Deploying app: TestN1QLExhaustConnPool. Item count src bucket: 0 dst bucket: 0
2021/12/14 06:56:47 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLExhaustConnPool' in metakv",
  "warnings": null
 }
}
2021/12/14 06:56:52 Waiting for app: TestN1QLExhaustConnPool to get deployed
2021/12/14 06:56:57 Waiting for app: TestN1QLExhaustConnPool to get deployed
2021/12/14 06:57:02 Waiting for app: TestN1QLExhaustConnPool to get deployed
2021/12/14 06:57:07 Waiting for app: TestN1QLExhaustConnPool to get deployed
2021/12/14 06:57:07 App: TestN1QLExhaustConnPool got deployed
2021/12/14 06:57:07 Starting bucket ops to source bucket
2021/12/14 06:57:12 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/14 06:57:17 Waiting for dst bucket item count to get to: 100 curr count: 41
2021/12/14 06:57:22 Waiting for dst bucket item count to get to: 100 curr count: 41
2021/12/14 06:57:22 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/14 06:57:22 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 06:57:22 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 06:57:27 Waiting for app: TestN1QLExhaustConnPool to get un-deployed
2021/12/14 06:57:32 Waiting for app: TestN1QLExhaustConnPool to get un-deployed
2021/12/14 06:57:37 Waiting for app: TestN1QLExhaustConnPool to get un-deployed
2021/12/14 06:57:42 Waiting for app: TestN1QLExhaustConnPool to get un-deployed
2021/12/14 06:57:42 App: TestN1QLExhaustConnPool got un-deployed
2021/12/14 06:57:42 No eventing-con process running
2021/12/14 06:57:42 Function: TestN1QLExhaustConnPool delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLExhaustConnPool deleting in the background"
}
2021/12/14 06:57:44 Context: Bucket flush, response code: 200 dump: 
2021/12/14 06:57:54 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/14 06:57:59 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/14 06:57:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 06:58:01 Context: Bucket flush, response code: 200 dump: 
2021/12/14 06:58:11 Waiting for dst bucket item count to get to: 0 curr count: 99
2021/12/14 06:58:16 Waiting for dst bucket item count to get to: 0 curr count: 99
2021/12/14 06:58:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLExhaustConnPool (108.97s)
=== RUN   TestN1QLGraceSufficientTimeOut
2021/12/14 06:58:21 Waiting for app: TestN1QLGraceSufficientTimeOut to get un-deployed
2021/12/14 06:58:21 App: TestN1QLGraceSufficientTimeOut got un-deployed
2021/12/14 06:58:21 No eventing-con process running
2021/12/14 06:58:21 Function: TestN1QLGraceSufficientTimeOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLGraceSufficientTimeOut not found"
 }
}
2021/12/14 06:58:23 Context: Bucket flush, response code: 200 dump: 
2021/12/14 06:58:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 06:58:30 Context: Bucket flush, response code: 200 dump: 
2021/12/14 06:58:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 06:58:35 Starting bucket ops to source bucket
2021/12/14 06:58:52 Starting bucket ops to source bucket
2021/12/14 06:59:10 Deploying app: TestN1QLGraceSufficientTimeOut. Item count src bucket: 30000 dst bucket: 10412
2021/12/14 06:59:11 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLGraceSufficientTimeOut' in metakv",
  "warnings": null
 }
}
2021/12/14 06:59:16 Waiting for app: TestN1QLGraceSufficientTimeOut to get deployed
2021/12/14 06:59:21 Waiting for app: TestN1QLGraceSufficientTimeOut to get deployed
2021/12/14 06:59:26 Waiting for app: TestN1QLGraceSufficientTimeOut to get deployed
2021/12/14 06:59:31 Waiting for app: TestN1QLGraceSufficientTimeOut to get deployed
2021/12/14 06:59:31 App: TestN1QLGraceSufficientTimeOut got deployed
2021/12/14 06:59:31 Starting bucket ops to source bucket
2021/12/14 06:59:36 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/14 06:59:41 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/14 06:59:46 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/14 06:59:51 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/14 06:59:56 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/14 07:00:01 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/14 07:00:06 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/14 07:00:11 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/14 07:00:16 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/14 07:00:21 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/14 07:00:26 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/14 07:00:26 src & dst bucket item count matched up. src bucket count: 30001 dst bucket count: 30001
2021/12/14 07:00:26 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 07:00:26 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 07:00:31 Waiting for app: TestN1QLGraceSufficientTimeOut to get un-deployed
2021/12/14 07:00:37 Waiting for app: TestN1QLGraceSufficientTimeOut to get un-deployed
2021/12/14 07:00:42 Waiting for app: TestN1QLGraceSufficientTimeOut to get un-deployed
2021/12/14 07:00:47 Waiting for app: TestN1QLGraceSufficientTimeOut to get un-deployed
2021/12/14 07:00:47 App: TestN1QLGraceSufficientTimeOut got un-deployed
2021/12/14 07:00:47 No eventing-con process running
2021/12/14 07:00:47 Function: TestN1QLGraceSufficientTimeOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLGraceSufficientTimeOut deleting in the background"
}
2021/12/14 07:00:49 Context: Bucket flush, response code: 200 dump: 
2021/12/14 07:00:59 Waiting for dst bucket item count to get to: 0 curr count: 25489
2021/12/14 07:01:04 Waiting for dst bucket item count to get to: 0 curr count: 10489
2021/12/14 07:01:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 07:01:05 Context: Bucket flush, response code: 200 dump: 
2021/12/14 07:01:15 Waiting for dst bucket item count to get to: 0 curr count: 30001
2021/12/14 07:01:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLGraceSufficientTimeOut (179.32s)
=== RUN   TestN1QLQueryConsistency
2021/12/14 07:01:20 Waiting for app: n1qlquery_consistency to get un-deployed
2021/12/14 07:01:20 App: n1qlquery_consistency got un-deployed
2021/12/14 07:01:20 No eventing-con process running
2021/12/14 07:01:20 Function: n1qlquery_consistency delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: n1qlquery_consistency not found"
 }
}
2021/12/14 07:01:22 Context: Bucket flush, response code: 200 dump: 
2021/12/14 07:01:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 07:01:29 Context: Bucket flush, response code: 200 dump: 
2021/12/14 07:01:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 07:01:34 Starting bucket ops to source bucket
2021/12/14 07:01:37 Deploying app: n1qlquery_consistency. Item count src bucket: 0 dst bucket: 0
2021/12/14 07:01:38 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'n1qlquery_consistency' in metakv",
  "warnings": null
 }
}
2021/12/14 07:01:43 Waiting for app: n1qlquery_consistency to get deployed
2021/12/14 07:01:48 Waiting for app: n1qlquery_consistency to get deployed
2021/12/14 07:01:53 Waiting for app: n1qlquery_consistency to get deployed
2021/12/14 07:01:58 Waiting for app: n1qlquery_consistency to get deployed
2021/12/14 07:01:58 App: n1qlquery_consistency got deployed
2021/12/14 07:02:03 Waiting for dst bucket item count to get to: 5000 curr count: 12
2021/12/14 07:02:08 Waiting for dst bucket item count to get to: 5000 curr count: 4810
2021/12/14 07:02:13 Waiting for dst bucket item count to get to: 5000 curr count: 4810
2021/12/14 07:02:13 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/14 07:02:13 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 07:02:13 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 07:02:18 Waiting for app: n1qlquery_consistency to get un-deployed
2021/12/14 07:02:23 Waiting for app: n1qlquery_consistency to get un-deployed
2021/12/14 07:02:28 Waiting for app: n1qlquery_consistency to get un-deployed
2021/12/14 07:02:33 Waiting for app: n1qlquery_consistency to get un-deployed
2021/12/14 07:02:33 App: n1qlquery_consistency got un-deployed
2021/12/14 07:02:33 No eventing-con process running
2021/12/14 07:02:33 Function: n1qlquery_consistency delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: n1qlquery_consistency deleting in the background"
}
2021/12/14 07:02:35 Context: Bucket flush, response code: 200 dump: 
2021/12/14 07:02:45 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/14 07:02:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 07:02:47 Context: Bucket flush, response code: 200 dump: 
2021/12/14 07:02:57 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/14 07:02:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLQueryConsistency (101.49s)
PASS
ok  	github.com/couchbase/eventing/tests/functional_tests	1559.633s

N1ql tests took 0 hours, 26 minutes and 4 seconds.

Starting server
Starting server: attempt 1

Rebalance tests

2021/12/14 07:04:32 Started rebalance
Post "http://127.0.0.1:9000/pools/default": read tcp 127.0.0.1:41738->127.0.0.1:9000: read: connection reset by peer
Quota setup Post "http://127.0.0.1:9000/pools/default": read tcp 127.0.0.1:41738->127.0.0.1:9000: read: connection reset by peer
=== RUN   TestDuplicateWithEventingRebIn
2021/12/14 07:04:51 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
2021/12/14 07:04:51 App: TestDuplicateWithEventingRebIn got un-deployed
2021/12/14 07:04:51 No eventing-con process running
2021/12/14 07:04:51 Function: TestDuplicateWithEventingRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithEventingRebIn not found"
 }
}
2021/12/14 07:04:52 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:04:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 07:04:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 07:04:59 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:05:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 07:05:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:05:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 07:05:09 Deploying app: TestDuplicateWithEventingRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/14 07:05:10 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithEventingRebIn' in metakv",
  "warnings": null
 }
}
2021/12/14 07:05:15 Waiting for app: TestDuplicateWithEventingRebIn to get deployed
2021/12/14 07:05:20 Waiting for app: TestDuplicateWithEventingRebIn to get deployed
2021/12/14 07:05:25 Waiting for app: TestDuplicateWithEventingRebIn to get deployed
2021/12/14 07:05:30 Waiting for app: TestDuplicateWithEventingRebIn to get deployed
2021/12/14 07:05:30 App: TestDuplicateWithEventingRebIn got deployed
2021/12/14 07:05:30 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/14 07:05:30 Starting bucket ops to source bucket
2021/12/14 07:05:44 Removing node(s): [] from the cluster
2021/12/14 07:05:49 Rebalance progress: 0
2021/12/14 07:05:54 Rebalance progress: 0
2021/12/14 07:05:59 Rebalance progress: 25
2021/12/14 07:06:04 Rebalance progress: 25
2021/12/14 07:06:09 Rebalance progress: 25
2021/12/14 07:06:14 Rebalance progress: 25
2021/12/14 07:06:19 Rebalance progress: 25
2021/12/14 07:06:24 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:06:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:06:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 07:06:24 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9302/api/v1/stats": dial tcp 127.0.0.1:9302: connect: connection refused
2021/12/14 07:06:24 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/14 07:06:24 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/14 07:06:29 Rebalance progress: 25
2021/12/14 07:06:34 Rebalance progress: 25
2021/12/14 07:06:39 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:06:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:06:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 07:06:39 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9302/api/v1/stats": dial tcp 127.0.0.1:9302: connect: connection refused
2021/12/14 07:06:39 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/14 07:06:39 Waiting for src bucket item count to get to: 200000 curr count: 95130
2021/12/14 07:06:44 Waiting for src bucket item count to get to: 200000 curr count: 111335
2021/12/14 07:06:49 Waiting for src bucket item count to get to: 200000 curr count: 111335
2021/12/14 07:06:54 Waiting for src bucket item count to get to: 200000 curr count: 127875
2021/12/14 07:06:59 Waiting for src bucket item count to get to: 200000 curr count: 127875
2021/12/14 07:07:04 Waiting for src bucket item count to get to: 200000 curr count: 144291
2021/12/14 07:07:09 Waiting for src bucket item count to get to: 200000 curr count: 144291
2021/12/14 07:07:14 Waiting for src bucket item count to get to: 200000 curr count: 160521
2021/12/14 07:07:19 Waiting for src bucket item count to get to: 200000 curr count: 160521
2021/12/14 07:07:24 Waiting for src bucket item count to get to: 200000 curr count: 176539
2021/12/14 07:07:29 Waiting for src bucket item count to get to: 200000 curr count: 176539
2021/12/14 07:07:34 Waiting for src bucket item count to get to: 200000 curr count: 192938
2021/12/14 07:07:39 Waiting for src bucket item count to get to: 200000 curr count: 192938
2021/12/14 07:07:45 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/14 07:07:45 Post SET ops. Expected item count: 200000 got 200000
2021/12/14 07:07:45 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 07:07:50 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
2021/12/14 07:07:55 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
2021/12/14 07:08:00 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
2021/12/14 07:08:05 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
2021/12/14 07:08:05 App: TestDuplicateWithEventingRebIn got un-deployed
2021/12/14 07:08:05 No eventing-con process running
2021/12/14 07:08:05 Function: TestDuplicateWithEventingRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithEventingRebIn deleting in the background"
}
2021/12/14 07:08:06 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:08:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 07:08:16 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/14 07:08:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 07:08:18 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:08:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 07:08:28 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/14 07:08:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithEventingRebIn (227.48s)
=== RUN   TestDuplicateWithEventingRebOut
2021/12/14 07:08:33 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/14 07:08:33 Removing node(s): [] from the cluster
2021/12/14 07:08:38 Rebalance progress: 100
2021/12/14 07:08:43 Waiting for app: TestDuplicateWithEventingRebOut to get un-deployed
2021/12/14 07:08:43 App: TestDuplicateWithEventingRebOut got un-deployed
2021/12/14 07:08:43 No eventing-con process running
2021/12/14 07:08:43 Function: TestDuplicateWithEventingRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithEventingRebOut not found"
 }
}
2021/12/14 07:08:45 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:08:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 07:08:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 07:08:52 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:08:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 07:08:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:09:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 07:09:02 Deploying app: TestDuplicateWithEventingRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/14 07:09:03 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithEventingRebOut' in metakv",
  "warnings": null
 }
}
2021/12/14 07:09:08 Waiting for app: TestDuplicateWithEventingRebOut to get deployed
2021/12/14 07:09:13 Waiting for app: TestDuplicateWithEventingRebOut to get deployed
2021/12/14 07:09:18 Waiting for app: TestDuplicateWithEventingRebOut to get deployed
2021/12/14 07:09:23 Waiting for app: TestDuplicateWithEventingRebOut to get deployed
2021/12/14 07:09:23 App: TestDuplicateWithEventingRebOut got deployed
2021/12/14 07:09:23 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/14 07:09:23 Starting bucket ops to source bucket
2021/12/14 07:09:28 Rebalance progress: 25
2021/12/14 07:09:33 Rebalance progress: 25
2021/12/14 07:09:38 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:09:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:09:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 07:09:38 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9302/api/v1/stats": dial tcp 127.0.0.1:9302: connect: connection refused
2021/12/14 07:09:38 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/14 07:09:38 Waiting for src bucket item count to get to: 200000 curr count: 9604
2021/12/14 07:09:43 Waiting for src bucket item count to get to: 200000 curr count: 25906
2021/12/14 07:09:48 Waiting for src bucket item count to get to: 200000 curr count: 25906
2021/12/14 07:09:53 Waiting for src bucket item count to get to: 200000 curr count: 42601
2021/12/14 07:09:58 Waiting for src bucket item count to get to: 200000 curr count: 42601
2021/12/14 07:10:03 Waiting for src bucket item count to get to: 200000 curr count: 58929
2021/12/14 07:10:08 Waiting for src bucket item count to get to: 200000 curr count: 58929
2021/12/14 07:10:13 Waiting for src bucket item count to get to: 200000 curr count: 58929
2021/12/14 07:10:18 Waiting for src bucket item count to get to: 200000 curr count: 75059
2021/12/14 07:10:23 Waiting for src bucket item count to get to: 200000 curr count: 75059
2021/12/14 07:10:28 Waiting for src bucket item count to get to: 200000 curr count: 91211
2021/12/14 07:10:33 Waiting for src bucket item count to get to: 200000 curr count: 91211
2021/12/14 07:10:38 Waiting for src bucket item count to get to: 200000 curr count: 107629
2021/12/14 07:10:43 Waiting for src bucket item count to get to: 200000 curr count: 107629
2021/12/14 07:10:48 Waiting for src bucket item count to get to: 200000 curr count: 124276
2021/12/14 07:10:53 Waiting for src bucket item count to get to: 200000 curr count: 124276
2021/12/14 07:10:58 Waiting for src bucket item count to get to: 200000 curr count: 140575
2021/12/14 07:11:03 Waiting for src bucket item count to get to: 200000 curr count: 140575
2021/12/14 07:11:08 Waiting for src bucket item count to get to: 200000 curr count: 156939
2021/12/14 07:11:13 Waiting for src bucket item count to get to: 200000 curr count: 156939
2021/12/14 07:11:18 Waiting for src bucket item count to get to: 200000 curr count: 173135
2021/12/14 07:11:23 Waiting for src bucket item count to get to: 200000 curr count: 173135
2021/12/14 07:11:28 Waiting for src bucket item count to get to: 200000 curr count: 189479
2021/12/14 07:11:33 Waiting for src bucket item count to get to: 200000 curr count: 189479
2021/12/14 07:11:38 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/14 07:11:38 Post SET ops. Expected item count: 200000 got 200000
2021/12/14 07:11:38 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 07:11:43 Waiting for app: TestDuplicateWithEventingRebOut to get un-deployed
2021/12/14 07:11:48 Waiting for app: TestDuplicateWithEventingRebOut to get un-deployed
2021/12/14 07:11:53 Waiting for app: TestDuplicateWithEventingRebOut to get un-deployed
2021/12/14 07:11:58 Waiting for app: TestDuplicateWithEventingRebOut to get un-deployed
2021/12/14 07:11:58 App: TestDuplicateWithEventingRebOut got un-deployed
2021/12/14 07:11:58 No eventing-con process running
2021/12/14 07:11:58 Function: TestDuplicateWithEventingRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithEventingRebOut deleting in the background"
}
2021/12/14 07:12:00 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:12:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 07:12:10 Waiting for dst bucket item count to get to: 0 curr count: 52319
2021/12/14 07:12:15 Waiting for dst bucket item count to get to: 0 curr count: 52319
2021/12/14 07:12:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 07:12:17 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:12:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 07:12:27 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/14 07:12:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithEventingRebOut (238.86s)
=== RUN   TestDuplicateWithEventingSwapReb
2021/12/14 07:12:37 Waiting for app: TestDuplicateWithEventingSwapReb to get un-deployed
2021/12/14 07:12:37 App: TestDuplicateWithEventingSwapReb got un-deployed
2021/12/14 07:12:37 No eventing-con process running
2021/12/14 07:12:37 Function: TestDuplicateWithEventingSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithEventingSwapReb not found"
 }
}
2021/12/14 07:12:39 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:12:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 07:12:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 07:12:45 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:12:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 07:12:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:12:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 07:12:56 Deploying app: TestDuplicateWithEventingSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/14 07:12:57 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithEventingSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/14 07:13:02 Waiting for app: TestDuplicateWithEventingSwapReb to get deployed
2021/12/14 07:13:07 Waiting for app: TestDuplicateWithEventingSwapReb to get deployed
2021/12/14 07:13:12 Waiting for app: TestDuplicateWithEventingSwapReb to get deployed
2021/12/14 07:13:17 Waiting for app: TestDuplicateWithEventingSwapReb to get deployed
2021/12/14 07:13:17 App: TestDuplicateWithEventingSwapReb got deployed
2021/12/14 07:13:17 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/14 07:13:17 Starting bucket ops to source bucket
2021/12/14 07:13:17 Removing node(s): [] from the cluster
2021/12/14 07:13:22 Rebalance progress: 25
2021/12/14 07:13:27 Rebalance progress: 25
2021/12/14 07:13:32 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:13:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:13:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 07:13:32 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9302/api/v1/stats": dial tcp 127.0.0.1:9302: connect: connection refused
2021/12/14 07:13:32 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/14 07:13:32 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/14 07:13:39 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/14 07:13:44 Rebalance progress: 16.66666666666667
2021/12/14 07:13:49 Rebalance progress: 16.66666666666667
2021/12/14 07:13:54 Rebalance progress: 16.66666666666667
2021/12/14 07:13:59 Rebalance progress: 66.04938271604938
2021/12/14 07:14:04 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:14:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:14:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 07:14:04 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/14 07:14:04 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/14 07:14:09 Rebalance progress: 16.66666666666667
2021/12/14 07:14:14 Rebalance progress: 16.66666666666667
2021/12/14 07:14:19 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:14:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:14:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 07:14:19 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/14 07:14:19 Waiting for src bucket item count to get to: 200000 curr count: 78611
2021/12/14 07:14:24 Waiting for src bucket item count to get to: 200000 curr count: 94578
2021/12/14 07:14:29 Waiting for src bucket item count to get to: 200000 curr count: 94578
2021/12/14 07:14:34 Waiting for src bucket item count to get to: 200000 curr count: 110586
2021/12/14 07:14:39 Waiting for src bucket item count to get to: 200000 curr count: 110586
2021/12/14 07:14:44 Waiting for src bucket item count to get to: 200000 curr count: 126996
2021/12/14 07:14:49 Waiting for src bucket item count to get to: 200000 curr count: 126996
2021/12/14 07:14:54 Waiting for src bucket item count to get to: 200000 curr count: 142920
2021/12/14 07:14:59 Waiting for src bucket item count to get to: 200000 curr count: 142920
2021/12/14 07:15:04 Waiting for src bucket item count to get to: 200000 curr count: 159138
2021/12/14 07:15:09 Waiting for src bucket item count to get to: 200000 curr count: 159138
2021/12/14 07:15:14 Waiting for src bucket item count to get to: 200000 curr count: 174875
2021/12/14 07:15:19 Waiting for src bucket item count to get to: 200000 curr count: 174875
2021/12/14 07:15:24 Waiting for src bucket item count to get to: 200000 curr count: 191393
2021/12/14 07:15:30 Waiting for src bucket item count to get to: 200000 curr count: 191393
2021/12/14 07:15:35 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/14 07:15:35 Post SET ops. Expected item count: 200000 got 200000
2021/12/14 07:15:35 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 07:15:40 Waiting for app: TestDuplicateWithEventingSwapReb to get un-deployed
2021/12/14 07:15:45 Waiting for app: TestDuplicateWithEventingSwapReb to get un-deployed
2021/12/14 07:15:50 Waiting for app: TestDuplicateWithEventingSwapReb to get un-deployed
2021/12/14 07:15:55 Waiting for app: TestDuplicateWithEventingSwapReb to get un-deployed
2021/12/14 07:15:55 App: TestDuplicateWithEventingSwapReb got un-deployed
2021/12/14 07:15:55 No eventing-con process running
2021/12/14 07:15:55 Function: TestDuplicateWithEventingSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithEventingSwapReb deleting in the background"
}
2021/12/14 07:15:56 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:16:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 07:16:06 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/14 07:16:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 07:16:08 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:16:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 07:16:18 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/14 07:16:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithEventingSwapReb (231.20s)
=== RUN   TestDuplicateWithKVRebIn
2021/12/14 07:16:28 Waiting for app: TestDuplicateWithKVRebIn to get un-deployed
2021/12/14 07:16:28 App: TestDuplicateWithKVRebIn got un-deployed
2021/12/14 07:16:28 No eventing-con process running
2021/12/14 07:16:28 Function: TestDuplicateWithKVRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithKVRebIn not found"
 }
}
2021/12/14 07:16:30 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:16:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 07:16:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 07:16:37 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:16:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 07:16:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:16:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 07:16:47 Deploying app: TestDuplicateWithKVRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/14 07:16:48 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithKVRebIn' in metakv",
  "warnings": null
 }
}
2021/12/14 07:16:53 Waiting for app: TestDuplicateWithKVRebIn to get deployed
2021/12/14 07:16:58 Waiting for app: TestDuplicateWithKVRebIn to get deployed
2021/12/14 07:17:03 Waiting for app: TestDuplicateWithKVRebIn to get deployed
2021/12/14 07:17:08 Waiting for app: TestDuplicateWithKVRebIn to get deployed
2021/12/14 07:17:08 App: TestDuplicateWithKVRebIn got deployed
2021/12/14 07:17:08 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/14 07:17:08 Starting bucket ops to source bucket
2021/12/14 07:17:08 Removing node(s): [] from the cluster
2021/12/14 07:17:13 Rebalance progress: 16.66666666666667
2021/12/14 07:17:18 Rebalance progress: 16.66666666666667
2021/12/14 07:17:23 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:17:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:17:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 07:17:24 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/14 07:17:24 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/14 07:17:29 Rebalance progress: 16.66666666666667
2021/12/14 07:17:34 Rebalance progress: 16.66666666666667
2021/12/14 07:17:39 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:17:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:17:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 07:17:39 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/14 07:17:39 Waiting for src bucket item count to get to: 200000 curr count: 33394
2021/12/14 07:17:44 Waiting for src bucket item count to get to: 200000 curr count: 49667
2021/12/14 07:17:49 Waiting for src bucket item count to get to: 200000 curr count: 49667
2021/12/14 07:17:54 Waiting for src bucket item count to get to: 200000 curr count: 65767
2021/12/14 07:17:59 Waiting for src bucket item count to get to: 200000 curr count: 65767
2021/12/14 07:18:04 Waiting for src bucket item count to get to: 200000 curr count: 81265
2021/12/14 07:18:09 Waiting for src bucket item count to get to: 200000 curr count: 81265
2021/12/14 07:18:14 Waiting for src bucket item count to get to: 200000 curr count: 97136
2021/12/14 07:18:19 Waiting for src bucket item count to get to: 200000 curr count: 97136
2021/12/14 07:18:24 Waiting for src bucket item count to get to: 200000 curr count: 113747
2021/12/14 07:18:29 Waiting for src bucket item count to get to: 200000 curr count: 113747
2021/12/14 07:18:34 Waiting for src bucket item count to get to: 200000 curr count: 130091
2021/12/14 07:18:39 Waiting for src bucket item count to get to: 200000 curr count: 130091
2021/12/14 07:18:44 Waiting for src bucket item count to get to: 200000 curr count: 130091
2021/12/14 07:18:49 Waiting for src bucket item count to get to: 200000 curr count: 146597
2021/12/14 07:18:54 Waiting for src bucket item count to get to: 200000 curr count: 146597
2021/12/14 07:18:59 Waiting for src bucket item count to get to: 200000 curr count: 162771
2021/12/14 07:19:04 Waiting for src bucket item count to get to: 200000 curr count: 162771
2021/12/14 07:19:09 Waiting for src bucket item count to get to: 200000 curr count: 177917
2021/12/14 07:19:14 Waiting for src bucket item count to get to: 200000 curr count: 177917
2021/12/14 07:19:19 Waiting for src bucket item count to get to: 200000 curr count: 193849
2021/12/14 07:19:24 Waiting for src bucket item count to get to: 200000 curr count: 193849
2021/12/14 07:19:29 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/14 07:19:29 Post SET ops. Expected item count: 200000 got 200000
2021/12/14 07:19:29 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 07:19:34 Waiting for app: TestDuplicateWithKVRebIn to get un-deployed
2021/12/14 07:19:39 Waiting for app: TestDuplicateWithKVRebIn to get un-deployed
2021/12/14 07:19:44 Waiting for app: TestDuplicateWithKVRebIn to get un-deployed
2021/12/14 07:19:49 Waiting for app: TestDuplicateWithKVRebIn to get un-deployed
2021/12/14 07:19:49 App: TestDuplicateWithKVRebIn got un-deployed
2021/12/14 07:19:49 No eventing-con process running
2021/12/14 07:19:49 Function: TestDuplicateWithKVRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithKVRebIn deleting in the background"
}
2021/12/14 07:19:51 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:19:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 07:20:01 Waiting for dst bucket item count to get to: 0 curr count: 144485
2021/12/14 07:20:06 Waiting for dst bucket item count to get to: 0 curr count: 144485
2021/12/14 07:20:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 07:20:08 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:20:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 07:20:18 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/14 07:20:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithKVRebIn (239.36s)
=== RUN   TestDuplicateWithKVRebOut
2021/12/14 07:20:23 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/14 07:20:23 Removing node(s): [] from the cluster
2021/12/14 07:20:28 Rebalance progress: 100
2021/12/14 07:20:33 Waiting for app: TestDuplicateWithKVRebOut to get un-deployed
2021/12/14 07:20:33 App: TestDuplicateWithKVRebOut got un-deployed
2021/12/14 07:20:33 No eventing-con process running
2021/12/14 07:20:33 Function: TestDuplicateWithKVRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithKVRebOut not found"
 }
}
2021/12/14 07:20:35 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:20:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 07:20:40 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 07:20:41 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:20:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 07:20:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:20:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 07:20:51 Deploying app: TestDuplicateWithKVRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/14 07:20:52 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithKVRebOut' in metakv",
  "warnings": null
 }
}
2021/12/14 07:20:57 Waiting for app: TestDuplicateWithKVRebOut to get deployed
2021/12/14 07:21:02 Waiting for app: TestDuplicateWithKVRebOut to get deployed
2021/12/14 07:21:07 Waiting for app: TestDuplicateWithKVRebOut to get deployed
2021/12/14 07:21:12 Waiting for app: TestDuplicateWithKVRebOut to get deployed
2021/12/14 07:21:12 App: TestDuplicateWithKVRebOut got deployed
2021/12/14 07:21:12 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/14 07:21:12 Starting bucket ops to source bucket
2021/12/14 07:21:18 Rebalance progress: 16.66666666666667
2021/12/14 07:21:23 Rebalance progress: 16.66666666666667
2021/12/14 07:21:28 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:21:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:21:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 07:21:28 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/14 07:21:28 Waiting for src bucket item count to get to: 200000 curr count: 10391
2021/12/14 07:21:33 Waiting for src bucket item count to get to: 200000 curr count: 26660
2021/12/14 07:21:38 Waiting for src bucket item count to get to: 200000 curr count: 26660
2021/12/14 07:21:43 Waiting for src bucket item count to get to: 200000 curr count: 42797
2021/12/14 07:21:48 Waiting for src bucket item count to get to: 200000 curr count: 42797
2021/12/14 07:21:53 Waiting for src bucket item count to get to: 200000 curr count: 58988
2021/12/14 07:21:58 Waiting for src bucket item count to get to: 200000 curr count: 58988
2021/12/14 07:22:03 Waiting for src bucket item count to get to: 200000 curr count: 58988
2021/12/14 07:22:08 Waiting for src bucket item count to get to: 200000 curr count: 74806
2021/12/14 07:22:13 Waiting for src bucket item count to get to: 200000 curr count: 74806
2021/12/14 07:22:18 Waiting for src bucket item count to get to: 200000 curr count: 90893
2021/12/14 07:22:23 Waiting for src bucket item count to get to: 200000 curr count: 90893
2021/12/14 07:22:28 Waiting for src bucket item count to get to: 200000 curr count: 107412
2021/12/14 07:22:33 Waiting for src bucket item count to get to: 200000 curr count: 107412
2021/12/14 07:22:38 Waiting for src bucket item count to get to: 200000 curr count: 123945
2021/12/14 07:22:43 Waiting for src bucket item count to get to: 200000 curr count: 123945
2021/12/14 07:22:48 Waiting for src bucket item count to get to: 200000 curr count: 140376
2021/12/14 07:22:53 Waiting for src bucket item count to get to: 200000 curr count: 140376
2021/12/14 07:22:58 Waiting for src bucket item count to get to: 200000 curr count: 156597
2021/12/14 07:23:03 Waiting for src bucket item count to get to: 200000 curr count: 156597
2021/12/14 07:23:08 Waiting for src bucket item count to get to: 200000 curr count: 171971
2021/12/14 07:23:13 Waiting for src bucket item count to get to: 200000 curr count: 171971
2021/12/14 07:23:18 Waiting for src bucket item count to get to: 200000 curr count: 187535
2021/12/14 07:23:23 Waiting for src bucket item count to get to: 200000 curr count: 187535
2021/12/14 07:23:28 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/14 07:23:28 Post SET ops. Expected item count: 200000 got 200000
2021/12/14 07:23:28 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 07:23:33 Waiting for app: TestDuplicateWithKVRebOut to get un-deployed
2021/12/14 07:23:38 Waiting for app: TestDuplicateWithKVRebOut to get un-deployed
2021/12/14 07:23:43 Waiting for app: TestDuplicateWithKVRebOut to get un-deployed
2021/12/14 07:23:48 Waiting for app: TestDuplicateWithKVRebOut to get un-deployed
2021/12/14 07:23:48 App: TestDuplicateWithKVRebOut got un-deployed
2021/12/14 07:23:48 No eventing-con process running
2021/12/14 07:23:48 Function: TestDuplicateWithKVRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithKVRebOut deleting in the background"
}
2021/12/14 07:23:50 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:23:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 07:23:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 07:23:56 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:24:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 07:24:06 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/14 07:24:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithKVRebOut (228.83s)
=== RUN   TestDuplicateWithKVSwapReb
2021/12/14 07:24:16 Waiting for app: TestDuplicateWithKVSwapReb to get un-deployed
2021/12/14 07:24:16 App: TestDuplicateWithKVSwapReb got un-deployed
2021/12/14 07:24:16 No eventing-con process running
2021/12/14 07:24:16 Function: TestDuplicateWithKVSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithKVSwapReb not found"
 }
}
2021/12/14 07:24:18 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:24:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 07:24:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 07:24:25 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:24:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 07:24:30 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:24:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 07:24:35 Deploying app: TestDuplicateWithKVSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/14 07:24:36 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithKVSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/14 07:24:41 Waiting for app: TestDuplicateWithKVSwapReb to get deployed
2021/12/14 07:24:46 Waiting for app: TestDuplicateWithKVSwapReb to get deployed
2021/12/14 07:24:51 Waiting for app: TestDuplicateWithKVSwapReb to get deployed
2021/12/14 07:24:56 Waiting for app: TestDuplicateWithKVSwapReb to get deployed
2021/12/14 07:24:56 App: TestDuplicateWithKVSwapReb got deployed
2021/12/14 07:24:56 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/14 07:24:56 Starting bucket ops to source bucket
2021/12/14 07:24:57 Removing node(s): [] from the cluster
2021/12/14 07:25:02 Rebalance progress: 16.66666666666667
2021/12/14 07:25:07 Rebalance progress: 16.66666666666667
2021/12/14 07:25:12 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:25:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:25:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 07:25:12 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/14 07:25:12 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/14 07:25:12 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/14 07:25:17 Rebalance progress: 16.66666666666667
2021/12/14 07:25:22 Rebalance progress: 16.66666666666667
2021/12/14 07:25:27 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:25:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:25:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 07:25:27 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/14 07:25:27 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/14 07:25:33 Rebalance progress: 16.66666666666667
2021/12/14 07:25:38 Rebalance progress: 16.66666666666667
2021/12/14 07:25:43 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:25:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:25:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 07:25:43 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/14 07:25:43 Waiting for src bucket item count to get to: 200000 curr count: 65997
2021/12/14 07:25:48 Waiting for src bucket item count to get to: 200000 curr count: 65997
2021/12/14 07:25:53 Waiting for src bucket item count to get to: 200000 curr count: 80695
2021/12/14 07:25:58 Waiting for src bucket item count to get to: 200000 curr count: 80695
2021/12/14 07:26:03 Waiting for src bucket item count to get to: 200000 curr count: 96421
2021/12/14 07:26:08 Waiting for src bucket item count to get to: 200000 curr count: 96421
2021/12/14 07:26:13 Waiting for src bucket item count to get to: 200000 curr count: 112370
2021/12/14 07:26:18 Waiting for src bucket item count to get to: 200000 curr count: 112370
2021/12/14 07:26:23 Waiting for src bucket item count to get to: 200000 curr count: 128323
2021/12/14 07:26:28 Waiting for src bucket item count to get to: 200000 curr count: 128323
2021/12/14 07:26:33 Waiting for src bucket item count to get to: 200000 curr count: 144370
2021/12/14 07:26:38 Waiting for src bucket item count to get to: 200000 curr count: 144370
2021/12/14 07:26:43 Waiting for src bucket item count to get to: 200000 curr count: 160448
2021/12/14 07:26:48 Waiting for src bucket item count to get to: 200000 curr count: 160448
2021/12/14 07:26:53 Waiting for src bucket item count to get to: 200000 curr count: 175401
2021/12/14 07:26:58 Waiting for src bucket item count to get to: 200000 curr count: 175401
2021/12/14 07:27:03 Waiting for src bucket item count to get to: 200000 curr count: 191344
2021/12/14 07:27:08 Waiting for src bucket item count to get to: 200000 curr count: 191344
2021/12/14 07:27:13 Waiting for src bucket item count to get to: 200000 curr count: 191344
2021/12/14 07:27:18 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/14 07:27:18 Post SET ops. Expected item count: 200000 got 200000
2021/12/14 07:27:18 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 07:27:23 Waiting for app: TestDuplicateWithKVSwapReb to get un-deployed
2021/12/14 07:27:28 Waiting for app: TestDuplicateWithKVSwapReb to get un-deployed
2021/12/14 07:27:33 Waiting for app: TestDuplicateWithKVSwapReb to get un-deployed
2021/12/14 07:27:38 Waiting for app: TestDuplicateWithKVSwapReb to get un-deployed
2021/12/14 07:27:38 App: TestDuplicateWithKVSwapReb got un-deployed
2021/12/14 07:27:38 No eventing-con process running
2021/12/14 07:27:38 Function: TestDuplicateWithKVSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithKVSwapReb deleting in the background"
}
2021/12/14 07:27:40 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:27:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 07:27:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 07:27:46 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:27:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 07:27:56 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/14 07:27:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithKVSwapReb (230.00s)
=== RUN   TestEventingRebKVOpsOnUpdateBucketOpOneByOne
2021/12/14 07:28:06 Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/14 07:28:06 App: TestEventingRebKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/14 07:28:06 No eventing-con process running
2021/12/14 07:28:06 Function: TestEventingRebKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpOneByOne not found"
 }
}
2021/12/14 07:28:08 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:28:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 07:28:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 07:28:15 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:28:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 07:28:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:28:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 07:28:25 Deploying app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne. Item count src bucket: 0 dst bucket: 0
2021/12/14 07:28:26 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebKVOpsOnUpdateBucketOpOneByOne' in metakv",
  "warnings": null
 }
}
2021/12/14 07:28:31 Starting bucket ops to source bucket
2021/12/14 07:28:36 Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/14 07:28:41 Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/14 07:28:46 Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/14 07:28:46 App: TestEventingRebKVOpsOnUpdateBucketOpOneByOne got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:28:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:28:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 07:28:46 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/14 07:28:46 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/14 07:28:46 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/14 07:28:51 Rebalance progress: 16.66666666666667
2021/12/14 07:28:56 Rebalance progress: 16.66666666666667
2021/12/14 07:29:01 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:29:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:29:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 07:29:01 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/14 07:29:01 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/14 07:29:02 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/14 07:29:07 Rebalance progress: 16.66666666666667
2021/12/14 07:29:12 Rebalance progress: 16.66666666666667
2021/12/14 07:29:17 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:29:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:29:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 07:29:17 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/14 07:29:17 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/14 07:29:22 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/14 07:29:27 Rebalance progress: 12.5
2021/12/14 07:29:32 Rebalance progress: 12.5
2021/12/14 07:29:37 Rebalance progress: 12.5
2021/12/14 07:29:42 Rebalance progress: 96.58203125
2021/12/14 07:29:47 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:29:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:29:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 07:29:47 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/14 07:29:53 Rebalance progress: 12.5
2021/12/14 07:29:58 Rebalance progress: 12.5
2021/12/14 07:30:03 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:30:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:30:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 07:30:03 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/14 07:30:08 Rebalance progress: 12.5
2021/12/14 07:30:13 Rebalance progress: 12.5
2021/12/14 07:30:18 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:30:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:30:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 07:30:18 clusterSetup.go:253: Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/14 07:30:23 Rebalance progress: 12.5
2021/12/14 07:30:28 Rebalance progress: 12.5
2021/12/14 07:30:33 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:30:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:30:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 07:30:33 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 07:30:38 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/14 07:30:43 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/14 07:30:48 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/14 07:30:53 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/14 07:30:53 clusterSetup.go:405: App: TestEventingRebKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/14 07:30:53 eventingSetup.go:46: No eventing-con process running
2021/12/14 07:30:53 eventingSetup.go:418: Function: TestEventingRebKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpOneByOne deleting in the background"
}
2021/12/14 07:30:55 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:31:00 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 07:31:05 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12163
2021/12/14 07:31:05 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 07:31:06 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:31:11 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 07:31:16 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12163
2021/12/14 07:31:16 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebKVOpsOnUpdateBucketOpOneByOne (200.10s)
=== RUN   TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce
2021/12/14 07:31:26 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get un-deployed
2021/12/14 07:31:26 clusterSetup.go:405: App: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce got un-deployed
2021/12/14 07:31:27 eventingSetup.go:46: No eventing-con process running
2021/12/14 07:31:27 eventingSetup.go:418: Function: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce not found"
 }
}
2021/12/14 07:31:28 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:31:33 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 07:31:33 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 07:31:35 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:31:40 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 07:31:40 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:31:45 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 07:31:45 eventingSetup.go:125: Deploying app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce. Item count src bucket: 0 dst bucket: 0
2021/12/14 07:31:46 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce' in metakv",
  "warnings": null
 }
}
2021/12/14 07:31:51 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/14 07:31:56 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get deployed
2021/12/14 07:32:01 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get deployed
2021/12/14 07:32:06 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get deployed
2021/12/14 07:32:06 clusterSetup.go:354: App: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:32:06 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:32:06 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 07:32:06 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/14 07:32:06 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/14 07:32:07 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/14 07:32:07 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/14 07:32:12 Rebalance progress: 12.5
2021/12/14 07:32:17 Rebalance progress: 12.5
2021/12/14 07:32:22 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:32:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:32:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 07:32:22 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002 127.0.0.1:19003] from the cluster
2021/12/14 07:32:27 Rebalance progress: 12.5
2021/12/14 07:32:32 Rebalance progress: 12.5
2021/12/14 07:32:37 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:32:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:32:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 07:32:37 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 07:32:42 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get un-deployed
2021/12/14 07:32:47 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get un-deployed
2021/12/14 07:32:52 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get un-deployed
2021/12/14 07:32:57 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get un-deployed
2021/12/14 07:32:57 clusterSetup.go:405: App: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce got un-deployed
2021/12/14 07:32:57 eventingSetup.go:46: No eventing-con process running
2021/12/14 07:32:57 eventingSetup.go:418: Function: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce deleting in the background"
}
2021/12/14 07:32:59 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:33:04 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 07:33:09 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4578
2021/12/14 07:33:09 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 07:33:11 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:33:16 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 07:33:21 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4107
2021/12/14 07:33:26 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4107
2021/12/14 07:33:26 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce (129.43s)
=== RUN   TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings
2021/12/14 07:33:36 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/14 07:33:36 clusterSetup.go:405: App: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings got un-deployed
2021/12/14 07:33:36 eventingSetup.go:46: No eventing-con process running
2021/12/14 07:33:36 eventingSetup.go:418: Function: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings not found"
 }
}
2021/12/14 07:33:38 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:33:43 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 07:33:43 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 07:33:45 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:33:50 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 07:33:50 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:33:55 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 07:33:55 eventingSetup.go:125: Deploying app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/14 07:33:56 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/14 07:34:01 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/14 07:34:06 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get deployed
2021/12/14 07:34:11 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get deployed
2021/12/14 07:34:16 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get deployed
2021/12/14 07:34:16 clusterSetup.go:354: App: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:34:16 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:34:16 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 07:34:16 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/14 07:34:16 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/14 07:34:21 Rebalance progress: 12.5
2021/12/14 07:34:26 Rebalance progress: 12.5
2021/12/14 07:34:31 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:34:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:34:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 07:34:31 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/14 07:34:31 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/14 07:34:37 Rebalance progress: 12.5
2021/12/14 07:34:42 Rebalance progress: 12.5
2021/12/14 07:34:47 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:34:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:34:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 07:34:47 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/14 07:34:47 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/14 07:34:52 Rebalance progress: 12.5
2021/12/14 07:34:57 Rebalance progress: 12.5
2021/12/14 07:35:02 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:35:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:35:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 07:35:03 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/14 07:35:08 Rebalance progress: 12.5
2021/12/14 07:35:13 Rebalance progress: 12.5
2021/12/14 07:35:18 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:35:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:35:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 07:35:18 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/14 07:35:23 Rebalance progress: 12.5
2021/12/14 07:35:28 Rebalance progress: 12.5
2021/12/14 07:35:33 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:35:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:35:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 07:35:33 clusterSetup.go:253: Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/14 07:35:38 Rebalance progress: 12.5
2021/12/14 07:35:43 Rebalance progress: 12.5
2021/12/14 07:35:48 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:35:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:35:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 07:35:48 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 07:35:53 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/14 07:35:58 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/14 07:36:03 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/14 07:36:08 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/14 07:36:08 clusterSetup.go:405: App: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings got un-deployed
2021/12/14 07:36:08 eventingSetup.go:46: No eventing-con process running
2021/12/14 07:36:08 eventingSetup.go:418: Function: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings deleting in the background"
}
2021/12/14 07:36:10 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:36:15 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 07:36:15 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 07:36:17 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:36:22 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 07:36:27 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 10705
2021/12/14 07:36:27 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings (180.70s)
=== RUN   TestEventingSwapRebOnUpdateBucketOp
2021/12/14 07:36:37 clusterSetup.go:397: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get un-deployed
2021/12/14 07:36:37 clusterSetup.go:405: App: TestEventingSwapRebOnUpdateBucketOp got un-deployed
2021/12/14 07:36:37 eventingSetup.go:46: No eventing-con process running
2021/12/14 07:36:37 eventingSetup.go:418: Function: TestEventingSwapRebOnUpdateBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingSwapRebOnUpdateBucketOp not found"
 }
}
2021/12/14 07:36:38 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:36:43 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 07:36:43 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 07:36:45 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:36:50 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 07:36:50 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:36:50 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 07:36:50 eventingSetup.go:125: Deploying app: TestEventingSwapRebOnUpdateBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/14 07:36:51 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingSwapRebOnUpdateBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/14 07:36:56 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/14 07:37:01 clusterSetup.go:346: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get deployed
2021/12/14 07:37:06 clusterSetup.go:346: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get deployed
2021/12/14 07:37:11 clusterSetup.go:346: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get deployed
2021/12/14 07:37:11 clusterSetup.go:354: App: TestEventingSwapRebOnUpdateBucketOp got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:37:11 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:37:11 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 07:37:11 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/14 07:37:12 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/14 07:37:17 Rebalance progress: 12.5
2021/12/14 07:37:22 Rebalance progress: 12.5
2021/12/14 07:37:27 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:37:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:37:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 07:37:27 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/14 07:37:27 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/14 07:37:32 Rebalance progress: 12.5
2021/12/14 07:37:37 Rebalance progress: 12.5
2021/12/14 07:37:42 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:37:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:37:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 07:37:42 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/14 07:37:47 Rebalance progress: 12.5
2021/12/14 07:37:52 Rebalance progress: 12.5
2021/12/14 07:37:57 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:37:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:37:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 07:37:58 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 07:38:03 clusterSetup.go:397: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get un-deployed
2021/12/14 07:38:08 clusterSetup.go:397: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get un-deployed
2021/12/14 07:38:13 clusterSetup.go:397: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get un-deployed
2021/12/14 07:38:18 clusterSetup.go:397: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get un-deployed
2021/12/14 07:38:18 clusterSetup.go:405: App: TestEventingSwapRebOnUpdateBucketOp got un-deployed
2021/12/14 07:38:18 eventingSetup.go:46: No eventing-con process running
2021/12/14 07:38:18 eventingSetup.go:418: Function: TestEventingSwapRebOnUpdateBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingSwapRebOnUpdateBucketOp deleting in the background"
}
2021/12/14 07:38:19 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:38:24 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 07:38:24 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 07:38:26 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:38:31 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 07:38:36 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6105
2021/12/14 07:38:36 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingSwapRebOnUpdateBucketOp (129.61s)
=== RUN   TestMetaRollbackWithEventingReb
2021/12/14 07:38:41 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/14 07:38:41 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/14 07:38:46 Rebalance progress: 100
2021/12/14 07:38:51 Waiting for app: TestMetaRollbackWithEventingReb to get un-deployed
2021/12/14 07:38:51 App: TestMetaRollbackWithEventingReb got un-deployed
2021/12/14 07:38:51 No eventing-con process running
2021/12/14 07:38:51 Function: TestMetaRollbackWithEventingReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMetaRollbackWithEventingReb not found"
 }
}
2021/12/14 07:38:53 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:38:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 07:38:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 07:39:00 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:39:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 07:39:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:39:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 07:39:05 Deploying app: TestMetaRollbackWithEventingReb. Item count src bucket: 0 dst bucket: 0
2021/12/14 07:39:06 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMetaRollbackWithEventingReb' in metakv",
  "warnings": null
 }
}
2021/12/14 07:39:11 Starting bucket ops to source bucket
2021/12/14 07:39:16 Waiting for app: TestMetaRollbackWithEventingReb to get deployed
2021/12/14 07:39:21 Waiting for app: TestMetaRollbackWithEventingReb to get deployed
2021/12/14 07:39:26 Waiting for app: TestMetaRollbackWithEventingReb to get deployed
2021/12/14 07:39:26 App: TestMetaRollbackWithEventingReb got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:39:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:39:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 07:39:26 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/14 07:39:27 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/14 07:39:32 Rebalance progress: 12.5
2021/12/14 07:39:37 Rebalance progress: 12.5
2021/12/14 07:39:42 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:39:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:39:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 07:39:42 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002] from the cluster
2021/12/14 07:39:42 bucketOps.go:134: Metakv lookup failed, err: Get "/_metakv/eventing/tempApps/TestMetaRollbackWithEventingReb/0": Unable to initialize cbauth's revrpc: cbauth environment variable CBAUTH_REVRPC_URL is not set
2021/12/14 07:39:47 Rebalance progress: 12.5
2021/12/14 07:39:52 Rebalance progress: 12.5
2021/12/14 07:39:57 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:39:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:39:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 07:39:57 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 07:40:02 clusterSetup.go:397: Waiting for app: TestMetaRollbackWithEventingReb to get un-deployed
2021/12/14 07:40:07 clusterSetup.go:397: Waiting for app: TestMetaRollbackWithEventingReb to get un-deployed
2021/12/14 07:40:12 clusterSetup.go:397: Waiting for app: TestMetaRollbackWithEventingReb to get un-deployed
2021/12/14 07:40:17 clusterSetup.go:397: Waiting for app: TestMetaRollbackWithEventingReb to get un-deployed
2021/12/14 07:40:17 clusterSetup.go:405: App: TestMetaRollbackWithEventingReb got un-deployed
2021/12/14 07:40:17 eventingSetup.go:46: No eventing-con process running
2021/12/14 07:40:17 eventingSetup.go:418: Function: TestMetaRollbackWithEventingReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMetaRollbackWithEventingReb deleting in the background"
}
2021/12/14 07:40:19 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:40:24 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 07:40:29 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4570
2021/12/14 07:40:29 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 07:40:31 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:40:36 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 07:40:41 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3326
2021/12/14 07:40:46 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3326
2021/12/14 07:40:46 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestMetaRollbackWithEventingReb (129.45s)
=== RUN   TestMetaPartialRollbackWithEventingReb
2021/12/14 07:40:51 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/14 07:40:51 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/14 07:40:56 Rebalance progress: 100
2021/12/14 07:41:01 Waiting for app: TestMetaPartialRollbackWithEventingReb to get un-deployed
2021/12/14 07:41:01 App: TestMetaPartialRollbackWithEventingReb got un-deployed
2021/12/14 07:41:01 No eventing-con process running
2021/12/14 07:41:01 Function: TestMetaPartialRollbackWithEventingReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMetaPartialRollbackWithEventingReb not found"
 }
}
2021/12/14 07:41:03 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:41:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 07:41:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 07:41:10 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:41:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 07:41:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:41:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 07:41:15 Deploying app: TestMetaPartialRollbackWithEventingReb. Item count src bucket: 0 dst bucket: 0
2021/12/14 07:41:16 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMetaPartialRollbackWithEventingReb' in metakv",
  "warnings": null
 }
}
2021/12/14 07:41:21 Starting bucket ops to source bucket
2021/12/14 07:41:26 Waiting for app: TestMetaPartialRollbackWithEventingReb to get deployed
2021/12/14 07:41:31 Waiting for app: TestMetaPartialRollbackWithEventingReb to get deployed
2021/12/14 07:41:36 Waiting for app: TestMetaPartialRollbackWithEventingReb to get deployed
2021/12/14 07:41:36 App: TestMetaPartialRollbackWithEventingReb got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:41:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:41:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 07:41:36 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/14 07:41:37 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/14 07:41:42 Rebalance progress: 12.5
2021/12/14 07:41:47 Rebalance progress: 12.5
2021/12/14 07:41:52 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:41:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:41:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 07:41:52 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002] from the cluster
2021/12/14 07:41:52 bucketOps.go:65: Metakv lookup failed, err: Get "/_metakv/eventing/tempApps/TestMetaPartialRollbackWithEventingReb/0": Unable to initialize cbauth's revrpc: cbauth environment variable CBAUTH_REVRPC_URL is not set
2021/12/14 07:41:57 Rebalance progress: 12.5
2021/12/14 07:42:02 Rebalance progress: 12.5
2021/12/14 07:42:07 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:42:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:42:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 07:42:07 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 07:42:07 bucketOps.go:65: Metakv lookup failed, err: Get "/_metakv/eventing/tempApps/TestMetaPartialRollbackWithEventingReb/0": Unable to initialize cbauth's revrpc: cbauth environment variable CBAUTH_REVRPC_URL is not set
2021/12/14 07:42:12 clusterSetup.go:397: Waiting for app: TestMetaPartialRollbackWithEventingReb to get un-deployed
2021/12/14 07:42:17 clusterSetup.go:397: Waiting for app: TestMetaPartialRollbackWithEventingReb to get un-deployed
2021/12/14 07:42:22 clusterSetup.go:397: Waiting for app: TestMetaPartialRollbackWithEventingReb to get un-deployed
2021/12/14 07:42:27 clusterSetup.go:397: Waiting for app: TestMetaPartialRollbackWithEventingReb to get un-deployed
2021/12/14 07:42:27 clusterSetup.go:405: App: TestMetaPartialRollbackWithEventingReb got un-deployed
2021/12/14 07:42:27 eventingSetup.go:46: No eventing-con process running
2021/12/14 07:42:27 eventingSetup.go:418: Function: TestMetaPartialRollbackWithEventingReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMetaPartialRollbackWithEventingReb deleting in the background"
}
2021/12/14 07:42:29 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:42:34 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 07:42:39 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 37771
2021/12/14 07:42:39 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 07:42:41 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:42:46 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 07:42:51 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 24994
2021/12/14 07:42:56 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 24994
2021/12/14 07:42:56 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestMetaPartialRollbackWithEventingReb (129.95s)
=== RUN   TestEventingRebMultipleHandlersOneByOne
2021/12/14 07:43:06 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get un-deployed
2021/12/14 07:43:06 clusterSetup.go:405: App: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op got un-deployed
2021/12/14 07:43:06 eventingSetup.go:46: No eventing-con process running
2021/12/14 07:43:06 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op not found"
 }
}
2021/12/14 07:43:07 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:43:12 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 07:43:12 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 07:43:14 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:43:19 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 07:43:19 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:43:24 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 07:43:24 eventingSetup.go:125: Deploying app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op. Item count src bucket: 0 dst bucket: 0
2021/12/14 07:43:25 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op' in metakv",
  "warnings": null
 }
}
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:43:25 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 07:43:25 eventingSetup.go:125: Deploying app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/14 07:43:27 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebMultipleHandlersOneByOnebucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/14 07:43:32 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/14 07:43:37 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get deployed
2021/12/14 07:43:42 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get deployed
2021/12/14 07:43:47 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get deployed
2021/12/14 07:43:47 clusterSetup.go:354: App: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op got deployed
2021/12/14 07:43:52 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get deployed
2021/12/14 07:43:57 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get deployed
2021/12/14 07:44:02 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get deployed
2021/12/14 07:44:02 clusterSetup.go:354: App: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:44:02 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:44:02 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 07:44:02 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/14 07:44:02 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/14 07:44:07 Rebalance progress: 12.5
2021/12/14 07:44:12 Rebalance progress: 12.5
2021/12/14 07:44:17 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:44:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:44:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 07:44:17 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/14 07:44:17 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/14 07:44:22 Rebalance progress: 12.5
2021/12/14 07:44:27 Rebalance progress: 12.5
2021/12/14 07:44:32 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:44:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:44:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 07:44:33 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/14 07:44:33 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/14 07:44:38 Rebalance progress: 12.5
2021/12/14 07:44:43 Rebalance progress: 12.5
2021/12/14 07:44:48 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:44:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:44:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 07:44:48 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/14 07:44:53 Rebalance progress: 12.5
2021/12/14 07:44:58 Rebalance progress: 12.5
2021/12/14 07:45:03 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:45:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:45:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 07:45:03 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/14 07:45:08 Rebalance progress: 12.5
2021/12/14 07:45:13 Rebalance progress: 12.5
2021/12/14 07:45:18 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:45:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:45:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 07:45:18 clusterSetup.go:253: Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/14 07:45:23 Rebalance progress: 12.5
2021/12/14 07:45:28 Rebalance progress: 12.5
2021/12/14 07:45:33 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:45:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:45:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 07:45:33 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 07:45:38 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get un-deployed
2021/12/14 07:45:43 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get un-deployed
2021/12/14 07:45:48 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get un-deployed
2021/12/14 07:45:53 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get un-deployed
2021/12/14 07:45:53 clusterSetup.go:405: App: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op got un-deployed
2021/12/14 07:45:53 eventingSetup.go:46: No eventing-con process running
2021/12/14 07:45:53 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op deleting in the background"
}
2021/12/14 07:45:56 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:46:01 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 07:46:06 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12158
2021/12/14 07:46:06 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 07:46:08 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:46:13 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 07:46:18 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12158
2021/12/14 07:46:18 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 07:46:18 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 07:46:23 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get un-deployed
2021/12/14 07:46:28 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get un-deployed
2021/12/14 07:46:33 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get un-deployed
2021/12/14 07:46:38 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get un-deployed
2021/12/14 07:46:38 clusterSetup.go:405: App: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update got un-deployed
2021/12/14 07:46:38 eventingSetup.go:46: No eventing-con process running
2021/12/14 07:46:38 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update deleting in the background"
}
2021/12/14 07:46:40 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:46:45 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 07:46:45 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 07:46:47 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:46:52 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 07:46:52 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebMultipleHandlersOneByOne (236.44s)
=== RUN   TestEventingRebMultipleHandlersAllAtOnce
2021/12/14 07:47:02 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get un-deployed
2021/12/14 07:47:02 clusterSetup.go:405: App: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op got un-deployed
2021/12/14 07:47:02 eventingSetup.go:46: No eventing-con process running
2021/12/14 07:47:02 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op not found"
 }
}
2021/12/14 07:47:04 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:47:09 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 07:47:09 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 07:47:11 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:47:16 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 07:47:16 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:47:21 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 07:47:21 eventingSetup.go:125: Deploying app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op. Item count src bucket: 0 dst bucket: 0
2021/12/14 07:47:22 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op' in metakv",
  "warnings": null
 }
}
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:47:22 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 07:47:22 eventingSetup.go:125: Deploying app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/14 07:47:23 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/14 07:47:28 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/14 07:47:33 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get deployed
2021/12/14 07:47:38 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get deployed
2021/12/14 07:47:43 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get deployed
2021/12/14 07:47:43 clusterSetup.go:354: App: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op got deployed
2021/12/14 07:47:48 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get deployed
2021/12/14 07:47:53 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get deployed
2021/12/14 07:47:58 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get deployed
2021/12/14 07:47:58 clusterSetup.go:354: App: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:47:58 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:47:58 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 07:47:58 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/14 07:47:59 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/14 07:48:00 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/14 07:48:00 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/14 07:48:05 Rebalance progress: 12.5
2021/12/14 07:48:10 Rebalance progress: 12.5
2021/12/14 07:48:15 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:48:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:48:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 07:48:15 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002 127.0.0.1:19003] from the cluster
2021/12/14 07:48:20 Rebalance progress: 12.5
2021/12/14 07:48:25 Rebalance progress: 12.5
2021/12/14 07:48:30 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:48:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:48:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 07:48:31 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 07:48:36 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get un-deployed
2021/12/14 07:48:41 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get un-deployed
2021/12/14 07:48:46 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get un-deployed
2021/12/14 07:48:51 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get un-deployed
2021/12/14 07:48:51 clusterSetup.go:405: App: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op got un-deployed
2021/12/14 07:48:51 eventingSetup.go:46: No eventing-con process running
2021/12/14 07:48:51 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op deleting in the background"
}
2021/12/14 07:48:55 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:49:00 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 07:49:05 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6237
2021/12/14 07:49:05 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 07:49:06 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:49:11 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 07:49:16 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6237
2021/12/14 07:49:16 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 07:49:16 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 07:49:21 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get un-deployed
2021/12/14 07:49:27 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get un-deployed
2021/12/14 07:49:32 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get un-deployed
2021/12/14 07:49:37 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get un-deployed
2021/12/14 07:49:37 clusterSetup.go:405: App: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update got un-deployed
2021/12/14 07:49:37 eventingSetup.go:46: No eventing-con process running
2021/12/14 07:49:37 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update deleting in the background"
}
2021/12/14 07:49:38 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:49:43 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 07:49:43 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 07:49:45 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:49:50 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 07:49:50 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebMultipleHandlersAllAtOnce (178.05s)
=== RUN   TestEventingSwapRebMultipleHandlers
2021/12/14 07:50:00 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get un-deployed
2021/12/14 07:50:00 clusterSetup.go:405: App: TestEventingSwapRebMultipleHandlersbucket_op_on_update got un-deployed
2021/12/14 07:50:00 eventingSetup.go:46: No eventing-con process running
2021/12/14 07:50:00 eventingSetup.go:418: Function: TestEventingSwapRebMultipleHandlersbucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingSwapRebMultipleHandlersbucket_op_on_update not found"
 }
}
2021/12/14 07:50:02 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:50:07 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 07:50:07 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 07:50:09 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:50:14 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 07:50:14 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 07:50:19 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get un-deployed
2021/12/14 07:50:19 clusterSetup.go:405: App: TestEventingSwapRebMultipleHandlerssys_test_bucket_op got un-deployed
2021/12/14 07:50:19 eventingSetup.go:46: No eventing-con process running
2021/12/14 07:50:19 eventingSetup.go:418: Function: TestEventingSwapRebMultipleHandlerssys_test_bucket_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingSwapRebMultipleHandlerssys_test_bucket_op not found"
 }
}
2021/12/14 07:50:21 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:50:26 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 07:50:26 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 07:50:27 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:50:32 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 07:50:32 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:50:32 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 07:50:32 eventingSetup.go:125: Deploying app: TestEventingSwapRebMultipleHandlersbucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/14 07:50:33 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingSwapRebMultipleHandlersbucket_op_on_update' in metakv",
  "warnings": null
 }
}
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:50:33 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 07:50:33 eventingSetup.go:125: Deploying app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op. Item count src bucket: 0 dst bucket: 0
2021/12/14 07:50:35 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingSwapRebMultipleHandlerssys_test_bucket_op' in metakv",
  "warnings": null
 }
}
2021/12/14 07:50:40 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/14 07:50:45 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get deployed
2021/12/14 07:50:50 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get deployed
2021/12/14 07:50:55 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get deployed
2021/12/14 07:50:55 clusterSetup.go:354: App: TestEventingSwapRebMultipleHandlersbucket_op_on_update got deployed
2021/12/14 07:51:00 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get deployed
2021/12/14 07:51:05 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get deployed
2021/12/14 07:51:10 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get deployed
2021/12/14 07:51:10 clusterSetup.go:354: App: TestEventingSwapRebMultipleHandlerssys_test_bucket_op got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:51:10 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:51:10 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 07:51:10 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/14 07:51:10 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/14 07:51:15 Rebalance progress: 12.5
2021/12/14 07:51:20 Rebalance progress: 12.5
2021/12/14 07:51:25 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:51:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:51:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 07:51:25 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/14 07:51:25 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/14 07:51:31 Rebalance progress: 12.5
2021/12/14 07:51:36 Rebalance progress: 12.5
2021/12/14 07:51:41 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:51:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:51:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 07:51:41 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/14 07:51:46 Rebalance progress: 12.5
2021/12/14 07:51:51 Rebalance progress: 12.5
2021/12/14 07:51:56 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:51:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:51:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 07:51:56 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 07:52:01 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get un-deployed
2021/12/14 07:52:06 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get un-deployed
2021/12/14 07:52:11 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get un-deployed
2021/12/14 07:52:16 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get un-deployed
2021/12/14 07:52:16 clusterSetup.go:405: App: TestEventingSwapRebMultipleHandlersbucket_op_on_update got un-deployed
2021/12/14 07:52:16 eventingSetup.go:46: No eventing-con process running
2021/12/14 07:52:16 eventingSetup.go:418: Function: TestEventingSwapRebMultipleHandlersbucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingSwapRebMultipleHandlersbucket_op_on_update deleting in the background"
}
2021/12/14 07:52:20 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:52:25 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 07:52:25 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 07:52:27 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:52:32 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 07:52:37 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7597
2021/12/14 07:52:37 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 07:52:37 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 07:52:42 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get un-deployed
2021/12/14 07:52:47 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get un-deployed
2021/12/14 07:52:52 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get un-deployed
2021/12/14 07:52:57 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get un-deployed
2021/12/14 07:52:57 clusterSetup.go:405: App: TestEventingSwapRebMultipleHandlerssys_test_bucket_op got un-deployed
2021/12/14 07:52:57 eventingSetup.go:46: No eventing-con process running
2021/12/14 07:52:57 eventingSetup.go:418: Function: TestEventingSwapRebMultipleHandlerssys_test_bucket_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingSwapRebMultipleHandlerssys_test_bucket_op deleting in the background"
}
2021/12/14 07:52:59 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:53:04 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 07:53:04 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 07:53:06 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:53:11 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 07:53:11 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingSwapRebMultipleHandlers (200.59s)
=== RUN   TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne
2021/12/14 07:53:21 clusterSetup.go:397: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/14 07:53:21 clusterSetup.go:405: App: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/14 07:53:21 eventingSetup.go:46: No eventing-con process running
2021/12/14 07:53:21 eventingSetup.go:418: Function: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne not found"
 }
}
2021/12/14 07:53:23 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:53:28 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 07:53:28 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 07:53:29 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:53:34 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 07:53:34 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:53:39 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 07:53:39 eventingSetup.go:125: Deploying app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne. Item count src bucket: 0 dst bucket: 0
2021/12/14 07:53:40 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne' in metakv",
  "warnings": null
 }
}
2021/12/14 07:53:45 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/14 07:53:50 clusterSetup.go:346: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/14 07:53:55 clusterSetup.go:346: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/14 07:54:00 clusterSetup.go:346: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/14 07:54:00 clusterSetup.go:354: App: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:54:00 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:54:00 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 07:54:00 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/14 07:54:01 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/14 07:54:21 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:54:21 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:54:21 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:54:52 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:54:52 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 07:54:52 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/14 07:54:57 Rebalance progress: 12.5
2021/12/14 07:55:02 Rebalance progress: 12.5
2021/12/14 07:55:07 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:55:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:55:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 07:55:07 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/14 07:55:07 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/14 07:55:12 Rebalance progress: 12.5
2021/12/14 07:55:17 Rebalance progress: 12.5
2021/12/14 07:55:22 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:55:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:55:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 07:55:22 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/14 07:55:27 Rebalance progress: 12.5
2021/12/14 07:55:32 Rebalance progress: 12.5
2021/12/14 07:55:37 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:55:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:55:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 07:55:37 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 07:55:42 clusterSetup.go:397: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/14 07:55:47 clusterSetup.go:397: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/14 07:55:52 clusterSetup.go:397: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/14 07:55:57 clusterSetup.go:397: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/14 07:55:57 clusterSetup.go:405: App: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/14 07:55:57 eventingSetup.go:46: No eventing-con process running
2021/12/14 07:55:57 eventingSetup.go:418: Function: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne deleting in the background"
}
2021/12/14 07:55:59 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:56:04 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 07:56:09 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 11144
2021/12/14 07:56:09 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 07:56:11 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:56:16 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 07:56:21 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 8429
2021/12/14 07:56:26 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 8429
2021/12/14 07:56:26 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne (195.04s)
=== RUN   TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne
2021/12/14 07:56:36 clusterSetup.go:397: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/14 07:56:36 clusterSetup.go:405: App: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/14 07:56:36 eventingSetup.go:46: No eventing-con process running
2021/12/14 07:56:36 eventingSetup.go:418: Function: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne not found"
 }
}
2021/12/14 07:56:37 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:56:42 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 07:56:43 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 07:56:44 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:56:49 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 07:56:49 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:56:54 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 07:56:54 eventingSetup.go:125: Deploying app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne. Item count src bucket: 0 dst bucket: 0
2021/12/14 07:56:55 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne' in metakv",
  "warnings": null
 }
}
2021/12/14 07:57:00 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/14 07:57:05 clusterSetup.go:346: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/14 07:57:10 clusterSetup.go:346: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/14 07:57:15 clusterSetup.go:346: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/14 07:57:15 clusterSetup.go:354: App: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:57:15 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:57:15 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 07:57:15 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/14 07:57:16 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/14 07:57:36 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:57:36 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:57:36 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:57:46 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:57:46 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 07:57:46 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/14 07:57:47 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/14 07:58:07 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:58:07 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:58:07 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:58:17 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:58:17 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 07:58:17 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/14 07:58:18 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/14 07:58:38 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:58:39 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:58:39 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:58:49 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:58:49 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 07:58:49 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/14 07:58:49 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/14 07:59:09 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:59:10 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:59:10 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:59:20 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:59:20 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 07:59:20 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/14 07:59:20 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/14 07:59:40 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:59:40 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:59:40 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:59:50 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 07:59:50 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 07:59:50 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/14 07:59:56 Rebalance progress: 12.5
2021/12/14 08:00:01 Rebalance progress: 12.5
2021/12/14 08:00:06 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 08:00:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 08:00:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 08:00:06 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/14 08:00:06 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/14 08:00:11 Rebalance progress: 12.5
2021/12/14 08:00:16 Rebalance progress: 12.5
2021/12/14 08:00:21 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 08:00:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 08:00:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 08:00:21 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/14 08:00:26 Rebalance progress: 12.5
2021/12/14 08:00:31 Rebalance progress: 12.5
2021/12/14 08:00:36 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 08:00:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 08:00:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 08:00:36 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 08:00:41 clusterSetup.go:397: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/14 08:00:46 clusterSetup.go:397: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/14 08:00:51 clusterSetup.go:397: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/14 08:00:56 clusterSetup.go:397: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/14 08:00:56 clusterSetup.go:405: App: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/14 08:00:56 eventingSetup.go:46: No eventing-con process running
2021/12/14 08:00:56 eventingSetup.go:418: Function: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne deleting in the background"
}
2021/12/14 08:00:58 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 08:01:03 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 08:01:08 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 21571
2021/12/14 08:01:08 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 08:01:10 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 08:01:15 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 08:01:20 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4042
2021/12/14 08:01:25 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4042
2021/12/14 08:01:25 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne (299.35s)
=== RUN   TestEventingFailoverOnUpdateBucketOp
2021/12/14 08:01:35 clusterSetup.go:397: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get un-deployed
2021/12/14 08:01:35 clusterSetup.go:405: App: TestEventingFailoverOnUpdateBucketOp got un-deployed
2021/12/14 08:01:35 eventingSetup.go:46: No eventing-con process running
2021/12/14 08:01:35 eventingSetup.go:418: Function: TestEventingFailoverOnUpdateBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingFailoverOnUpdateBucketOp not found"
 }
}
2021/12/14 08:01:37 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 08:01:42 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 08:01:42 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 08:01:44 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 08:01:49 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 08:01:49 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 08:01:54 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 08:01:54 eventingSetup.go:125: Deploying app: TestEventingFailoverOnUpdateBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/14 08:01:55 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingFailoverOnUpdateBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/14 08:02:00 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/14 08:02:05 clusterSetup.go:346: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get deployed
2021/12/14 08:02:10 clusterSetup.go:346: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get deployed
2021/12/14 08:02:15 clusterSetup.go:346: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get deployed
2021/12/14 08:02:15 clusterSetup.go:354: App: TestEventingFailoverOnUpdateBucketOp got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 08:02:15 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 08:02:15 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 08:02:15 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/14 08:02:15 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/14 08:02:20 Rebalance progress: 12.5
2021/12/14 08:02:25 Rebalance progress: 12.5
2021/12/14 08:02:30 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 08:02:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 08:02:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 08:02:30 clusterSetup.go:229: Failing over: [127.0.0.1:19001]
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 08:02:40 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 08:02:40 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 08:02:40 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/14 08:02:45 Rebalance progress: 12.5
2021/12/14 08:02:50 Rebalance progress: 12.5
2021/12/14 08:02:55 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 08:02:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 08:02:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 08:02:55 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/14 08:02:56 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/14 08:03:01 Rebalance progress: 12.5
2021/12/14 08:03:06 Rebalance progress: 12.5
2021/12/14 08:03:11 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 08:03:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 08:03:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 08:03:11 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/14 08:03:16 Rebalance progress: 12.5
2021/12/14 08:03:21 Rebalance progress: 12.5
2021/12/14 08:03:26 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 08:03:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 08:03:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 08:03:26 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 08:03:31 clusterSetup.go:397: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get un-deployed
2021/12/14 08:03:36 clusterSetup.go:397: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get un-deployed
2021/12/14 08:03:41 clusterSetup.go:397: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get un-deployed
2021/12/14 08:03:46 clusterSetup.go:397: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get un-deployed
2021/12/14 08:03:46 clusterSetup.go:405: App: TestEventingFailoverOnUpdateBucketOp got un-deployed
2021/12/14 08:03:46 eventingSetup.go:46: No eventing-con process running
2021/12/14 08:03:46 eventingSetup.go:418: Function: TestEventingFailoverOnUpdateBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingFailoverOnUpdateBucketOp deleting in the background"
}
2021/12/14 08:03:48 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 08:03:53 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 08:03:58 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 8578
2021/12/14 08:03:58 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 08:03:59 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 08:04:04 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 08:04:04 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingFailoverOnUpdateBucketOp (159.38s)
=== RUN   TestEventingKVRebalanceOnUpdateBucketOp
2021/12/14 08:04:14 clusterSetup.go:397: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get un-deployed
2021/12/14 08:04:14 clusterSetup.go:405: App: TestEventingKVRebalanceOnUpdateBucketOp got un-deployed
2021/12/14 08:04:14 eventingSetup.go:46: No eventing-con process running
2021/12/14 08:04:15 eventingSetup.go:418: Function: TestEventingKVRebalanceOnUpdateBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingKVRebalanceOnUpdateBucketOp not found"
 }
}
2021/12/14 08:04:16 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 08:04:21 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 08:04:21 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 08:04:23 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 08:04:28 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 08:04:28 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 08:04:33 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 08:04:33 eventingSetup.go:125: Deploying app: TestEventingKVRebalanceOnUpdateBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/14 08:04:34 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingKVRebalanceOnUpdateBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/14 08:04:39 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/14 08:04:44 clusterSetup.go:346: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get deployed
2021/12/14 08:04:49 clusterSetup.go:346: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get deployed
2021/12/14 08:04:54 clusterSetup.go:346: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get deployed
2021/12/14 08:04:54 clusterSetup.go:354: App: TestEventingKVRebalanceOnUpdateBucketOp got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 08:04:54 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 08:04:54 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 08:04:54 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing,kv to the cluster
2021/12/14 08:04:55 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/14 08:05:00 Rebalance progress: 12.5
2021/12/14 08:05:05 Rebalance progress: 12.5
2021/12/14 08:05:10 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 08:05:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 08:05:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 08:05:10 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing,kv to the cluster
2021/12/14 08:05:10 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/14 08:05:15 Rebalance progress: 12.5
2021/12/14 08:05:20 Rebalance progress: 12.5
2021/12/14 08:05:25 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 08:05:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 08:05:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 08:05:25 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/14 08:05:30 Rebalance progress: 12.5
2021/12/14 08:05:35 Rebalance progress: 12.5
2021/12/14 08:05:40 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 08:05:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 08:05:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 08:05:41 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 08:05:46 clusterSetup.go:397: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get un-deployed
2021/12/14 08:05:51 clusterSetup.go:397: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get un-deployed
2021/12/14 08:05:56 clusterSetup.go:397: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get un-deployed
2021/12/14 08:06:01 clusterSetup.go:397: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get un-deployed
2021/12/14 08:06:01 clusterSetup.go:405: App: TestEventingKVRebalanceOnUpdateBucketOp got un-deployed
2021/12/14 08:06:01 eventingSetup.go:46: No eventing-con process running
2021/12/14 08:06:01 eventingSetup.go:418: Function: TestEventingKVRebalanceOnUpdateBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingKVRebalanceOnUpdateBucketOp deleting in the background"
}
2021/12/14 08:06:02 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 08:06:07 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 08:06:12 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6096
2021/12/14 08:06:17 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6096
2021/12/14 08:06:17 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 08:06:19 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 08:06:24 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 08:06:29 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6096
2021/12/14 08:06:29 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingKVRebalanceOnUpdateBucketOp (144.72s)
=== RUN   TestEventingRebInWithOps
2021/12/14 08:06:39 clusterSetup.go:397: Waiting for app: TestEventingRebInWithOps to get un-deployed
2021/12/14 08:06:39 clusterSetup.go:405: App: TestEventingRebInWithOps got un-deployed
2021/12/14 08:06:39 eventingSetup.go:46: No eventing-con process running
2021/12/14 08:06:39 eventingSetup.go:418: Function: TestEventingRebInWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebInWithOps not found"
 }
}
2021/12/14 08:06:41 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 08:06:46 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 08:06:46 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 08:06:48 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 08:06:53 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 08:06:53 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 08:06:58 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 08:06:58 eventingSetup.go:125: Deploying app: TestEventingRebInWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/14 08:06:59 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebInWithOps' in metakv",
  "warnings": null
 }
}
2021/12/14 08:07:04 clusterSetup.go:346: Waiting for app: TestEventingRebInWithOps to get deployed
2021/12/14 08:07:09 clusterSetup.go:346: Waiting for app: TestEventingRebInWithOps to get deployed
2021/12/14 08:07:14 clusterSetup.go:346: Waiting for app: TestEventingRebInWithOps to get deployed
2021/12/14 08:07:19 clusterSetup.go:346: Waiting for app: TestEventingRebInWithOps to get deployed
2021/12/14 08:07:19 clusterSetup.go:354: App: TestEventingRebInWithOps got deployed
2021/12/14 08:07:19 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/14 08:07:19 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/14 08:07:20 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/14 08:07:25 Rebalance progress: 12.5
2021/12/14 08:07:30 Rebalance progress: 12.5
2021/12/14 08:07:35 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 08:07:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 08:07:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 08:07:35 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/14 08:07:40 Rebalance progress: 12.5
2021/12/14 08:07:45 Rebalance progress: 12.5
2021/12/14 08:07:50 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 08:07:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 08:07:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 08:07:50 Waiting for src bucket item count to get to: 200000 curr count: 30598
2021/12/14 08:07:55 Waiting for src bucket item count to get to: 200000 curr count: 46308
2021/12/14 08:08:00 Waiting for src bucket item count to get to: 200000 curr count: 46308
2021/12/14 08:08:05 Waiting for src bucket item count to get to: 200000 curr count: 61835
2021/12/14 08:08:10 Waiting for src bucket item count to get to: 200000 curr count: 61835
2021/12/14 08:08:15 Waiting for src bucket item count to get to: 200000 curr count: 76943
2021/12/14 08:08:20 Waiting for src bucket item count to get to: 200000 curr count: 76943
2021/12/14 08:08:25 Waiting for src bucket item count to get to: 200000 curr count: 92993
2021/12/14 08:08:30 Waiting for src bucket item count to get to: 200000 curr count: 92993
2021/12/14 08:08:35 Waiting for src bucket item count to get to: 200000 curr count: 109207
2021/12/14 08:08:40 Waiting for src bucket item count to get to: 200000 curr count: 125344
2021/12/14 08:08:45 Waiting for src bucket item count to get to: 200000 curr count: 125344
2021/12/14 08:08:50 Waiting for src bucket item count to get to: 200000 curr count: 141495
2021/12/14 08:08:55 Waiting for src bucket item count to get to: 200000 curr count: 141495
2021/12/14 08:09:00 Waiting for src bucket item count to get to: 200000 curr count: 141495
2021/12/14 08:09:05 Waiting for src bucket item count to get to: 200000 curr count: 157611
2021/12/14 08:09:10 Waiting for src bucket item count to get to: 200000 curr count: 157611
2021/12/14 08:09:15 Waiting for src bucket item count to get to: 200000 curr count: 172613
2021/12/14 08:09:20 Waiting for src bucket item count to get to: 200000 curr count: 172613
2021/12/14 08:09:25 Waiting for src bucket item count to get to: 200000 curr count: 188275
2021/12/14 08:09:30 Waiting for src bucket item count to get to: 200000 curr count: 188275
2021/12/14 08:09:35 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/14 08:09:35 Post SET ops. Expected item count: 200000 got 200000
2021/12/14 08:09:35 Starting bucket ops to source bucket
2021/12/14 08:13:26 Waiting for dst bucket item count to get to: 0 curr count: 20871
2021/12/14 08:13:31 Waiting for dst bucket item count to get to: 0 curr count: 2779
2021/12/14 08:13:36 Waiting for dst bucket item count to get to: 0 curr count: 2779
2021/12/14 08:13:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 08:13:36 Post DELETE ops. Expected item count: 0 got 0
2021/12/14 08:13:36 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 08:13:41 Waiting for app: TestEventingRebInWithOps to get un-deployed
2021/12/14 08:13:46 Waiting for app: TestEventingRebInWithOps to get un-deployed
2021/12/14 08:13:51 Waiting for app: TestEventingRebInWithOps to get un-deployed
2021/12/14 08:13:56 Waiting for app: TestEventingRebInWithOps to get un-deployed
2021/12/14 08:13:56 App: TestEventingRebInWithOps got un-deployed
2021/12/14 08:13:56 No eventing-con process running
2021/12/14 08:13:56 Function: TestEventingRebInWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebInWithOps deleting in the background"
}
2021/12/14 08:13:58 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 08:14:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 08:14:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 08:14:05 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 08:14:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 08:14:10 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebInWithOps (460.43s)
=== RUN   TestEventingRebOutWithOps
2021/12/14 08:14:15 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/14 08:14:15 Removing node(s): [] from the cluster
2021/12/14 08:14:20 Rebalance progress: 100
2021/12/14 08:14:25 Waiting for app: TestEventingRebOutWithOps to get un-deployed
2021/12/14 08:14:25 App: TestEventingRebOutWithOps got un-deployed
2021/12/14 08:14:25 No eventing-con process running
2021/12/14 08:14:25 Function: TestEventingRebOutWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebOutWithOps not found"
 }
}
2021/12/14 08:14:27 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 08:14:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 08:14:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 08:14:34 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 08:14:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 08:14:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 08:14:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 08:14:44 Deploying app: TestEventingRebOutWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/14 08:14:45 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebOutWithOps' in metakv",
  "warnings": null
 }
}
2021/12/14 08:14:50 Waiting for app: TestEventingRebOutWithOps to get deployed
2021/12/14 08:14:55 Waiting for app: TestEventingRebOutWithOps to get deployed
2021/12/14 08:15:00 Waiting for app: TestEventingRebOutWithOps to get deployed
2021/12/14 08:15:05 Waiting for app: TestEventingRebOutWithOps to get deployed
2021/12/14 08:15:05 App: TestEventingRebOutWithOps got deployed
2021/12/14 08:15:05 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/14 08:15:05 Starting bucket ops to source bucket
2021/12/14 08:15:10 Rebalance progress: 12.5
2021/12/14 08:15:15 Rebalance progress: 12.5
2021/12/14 08:15:20 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 08:15:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 08:15:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 08:15:20 Waiting for src bucket item count to get to: 200000 curr count: 21831
2021/12/14 08:15:25 Waiting for src bucket item count to get to: 200000 curr count: 21831
2021/12/14 08:15:30 Waiting for src bucket item count to get to: 200000 curr count: 37772
2021/12/14 08:15:35 Waiting for src bucket item count to get to: 200000 curr count: 37772
2021/12/14 08:15:40 Waiting for src bucket item count to get to: 200000 curr count: 54007
2021/12/14 08:15:45 Waiting for src bucket item count to get to: 200000 curr count: 54007
2021/12/14 08:15:50 Waiting for src bucket item count to get to: 200000 curr count: 54007
2021/12/14 08:15:55 Waiting for src bucket item count to get to: 200000 curr count: 70331
2021/12/14 08:16:00 Waiting for src bucket item count to get to: 200000 curr count: 70331
2021/12/14 08:16:05 Waiting for src bucket item count to get to: 200000 curr count: 85334
2021/12/14 08:16:10 Waiting for src bucket item count to get to: 200000 curr count: 85334
2021/12/14 08:16:15 Waiting for src bucket item count to get to: 200000 curr count: 101364
2021/12/14 08:16:20 Waiting for src bucket item count to get to: 200000 curr count: 101364
2021/12/14 08:16:25 Waiting for src bucket item count to get to: 200000 curr count: 117200
2021/12/14 08:16:30 Waiting for src bucket item count to get to: 200000 curr count: 117200
2021/12/14 08:16:35 Waiting for src bucket item count to get to: 200000 curr count: 133289
2021/12/14 08:16:40 Waiting for src bucket item count to get to: 200000 curr count: 133289
2021/12/14 08:16:45 Waiting for src bucket item count to get to: 200000 curr count: 149339
2021/12/14 08:16:50 Waiting for src bucket item count to get to: 200000 curr count: 149339
2021/12/14 08:16:55 Waiting for src bucket item count to get to: 200000 curr count: 165212
2021/12/14 08:17:00 Waiting for src bucket item count to get to: 200000 curr count: 165212
2021/12/14 08:17:05 Waiting for src bucket item count to get to: 200000 curr count: 179826
2021/12/14 08:17:10 Waiting for src bucket item count to get to: 200000 curr count: 179826
2021/12/14 08:17:15 Waiting for src bucket item count to get to: 200000 curr count: 195607
2021/12/14 08:17:20 Waiting for src bucket item count to get to: 200000 curr count: 195607
2021/12/14 08:17:25 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/14 08:17:25 Post SET ops. Expected item count: 200000 got 200000
2021/12/14 08:17:25 Starting bucket ops to source bucket
2021/12/14 08:21:15 Waiting for dst bucket item count to get to: 0 curr count: 20184
2021/12/14 08:21:20 Waiting for dst bucket item count to get to: 0 curr count: 1854
2021/12/14 08:21:25 Waiting for dst bucket item count to get to: 0 curr count: 1854
2021/12/14 08:21:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 08:21:25 Post DELETE ops. Expected item count: 0 got 0
2021/12/14 08:21:25 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 08:21:30 Waiting for app: TestEventingRebOutWithOps to get un-deployed
2021/12/14 08:21:35 Waiting for app: TestEventingRebOutWithOps to get un-deployed
2021/12/14 08:21:40 Waiting for app: TestEventingRebOutWithOps to get un-deployed
2021/12/14 08:21:45 Waiting for app: TestEventingRebOutWithOps to get un-deployed
2021/12/14 08:21:45 App: TestEventingRebOutWithOps got un-deployed
2021/12/14 08:21:45 No eventing-con process running
2021/12/14 08:21:46 Function: TestEventingRebOutWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebOutWithOps deleting in the background"
}
2021/12/14 08:21:47 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 08:21:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 08:21:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 08:21:54 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 08:21:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 08:21:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebOutWithOps (469.47s)
=== RUN   TestEventingSwapRebWithOps
2021/12/14 08:22:09 Waiting for app: TestEventingSwapRebWithOps to get un-deployed
2021/12/14 08:22:09 App: TestEventingSwapRebWithOps got un-deployed
2021/12/14 08:22:09 No eventing-con process running
2021/12/14 08:22:09 Function: TestEventingSwapRebWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingSwapRebWithOps not found"
 }
}
2021/12/14 08:22:11 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 08:22:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 08:22:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 08:22:18 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 08:22:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 08:22:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 08:22:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 08:22:28 Deploying app: TestEventingSwapRebWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/14 08:22:29 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingSwapRebWithOps' in metakv",
  "warnings": null
 }
}
2021/12/14 08:22:34 Waiting for app: TestEventingSwapRebWithOps to get deployed
2021/12/14 08:22:39 Waiting for app: TestEventingSwapRebWithOps to get deployed
2021/12/14 08:22:44 Waiting for app: TestEventingSwapRebWithOps to get deployed
2021/12/14 08:22:49 Waiting for app: TestEventingSwapRebWithOps to get deployed
2021/12/14 08:22:49 App: TestEventingSwapRebWithOps got deployed
2021/12/14 08:22:49 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/14 08:22:49 Starting bucket ops to source bucket
2021/12/14 08:22:50 Removing node(s): [] from the cluster
2021/12/14 08:22:55 Rebalance progress: 12.5
2021/12/14 08:23:00 Rebalance progress: 12.5
2021/12/14 08:23:05 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 08:23:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 08:23:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 08:23:05 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/14 08:23:05 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/14 08:23:10 Rebalance progress: 12.5
2021/12/14 08:23:15 Rebalance progress: 12.5
2021/12/14 08:23:20 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 08:23:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 08:23:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 08:23:20 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/14 08:23:25 Rebalance progress: 12.5
2021/12/14 08:23:30 Rebalance progress: 12.5
2021/12/14 08:23:35 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 08:23:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 08:23:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 08:23:35 Waiting for src bucket item count to get to: 200000 curr count: 61997
2021/12/14 08:23:40 Waiting for src bucket item count to get to: 200000 curr count: 61997
2021/12/14 08:23:45 Waiting for src bucket item count to get to: 200000 curr count: 76860
2021/12/14 08:23:50 Waiting for src bucket item count to get to: 200000 curr count: 76860
2021/12/14 08:23:55 Waiting for src bucket item count to get to: 200000 curr count: 92467
2021/12/14 08:24:00 Waiting for src bucket item count to get to: 200000 curr count: 92467
2021/12/14 08:24:05 Waiting for src bucket item count to get to: 200000 curr count: 108547
2021/12/14 08:24:10 Waiting for src bucket item count to get to: 200000 curr count: 108547
2021/12/14 08:24:15 Waiting for src bucket item count to get to: 200000 curr count: 124441
2021/12/14 08:24:20 Waiting for src bucket item count to get to: 200000 curr count: 140643
2021/12/14 08:24:25 Waiting for src bucket item count to get to: 200000 curr count: 140643
2021/12/14 08:24:30 Waiting for src bucket item count to get to: 200000 curr count: 156488
2021/12/14 08:24:35 Waiting for src bucket item count to get to: 200000 curr count: 156488
2021/12/14 08:24:40 Waiting for src bucket item count to get to: 200000 curr count: 171555
2021/12/14 08:24:45 Waiting for src bucket item count to get to: 200000 curr count: 171555
2021/12/14 08:24:50 Waiting for src bucket item count to get to: 200000 curr count: 187301
2021/12/14 08:24:55 Waiting for src bucket item count to get to: 200000 curr count: 187301
2021/12/14 08:25:00 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/14 08:25:00 Post SET ops. Expected item count: 200000 got 200000
2021/12/14 08:25:00 Starting bucket ops to source bucket
2021/12/14 08:28:51 Waiting for dst bucket item count to get to: 0 curr count: 11343
2021/12/14 08:28:56 Waiting for dst bucket item count to get to: 0 curr count: 11343
2021/12/14 08:28:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 08:28:56 Post DELETE ops. Expected item count: 0 got 0
2021/12/14 08:28:56 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 08:29:01 Waiting for app: TestEventingSwapRebWithOps to get un-deployed
2021/12/14 08:29:06 Waiting for app: TestEventingSwapRebWithOps to get un-deployed
2021/12/14 08:29:11 Waiting for app: TestEventingSwapRebWithOps to get un-deployed
2021/12/14 08:29:16 Waiting for app: TestEventingSwapRebWithOps to get un-deployed
2021/12/14 08:29:16 App: TestEventingSwapRebWithOps got un-deployed
2021/12/14 08:29:16 No eventing-con process running
2021/12/14 08:29:16 Function: TestEventingSwapRebWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingSwapRebWithOps deleting in the background"
}
2021/12/14 08:29:18 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 08:29:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 08:29:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 08:29:24 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 08:29:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 08:29:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingSwapRebWithOps (450.25s)
=== RUN   TestKVRebInWithOps
2021/12/14 08:29:39 Waiting for app: TestKVRebInWithOps to get un-deployed
2021/12/14 08:29:39 App: TestKVRebInWithOps got un-deployed
2021/12/14 08:29:39 No eventing-con process running
2021/12/14 08:29:39 Function: TestKVRebInWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVRebInWithOps not found"
 }
}
2021/12/14 08:29:41 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 08:29:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 08:29:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 08:29:48 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 08:29:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 08:29:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 08:29:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 08:29:58 Deploying app: TestKVRebInWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/14 08:29:59 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVRebInWithOps' in metakv",
  "warnings": null
 }
}
2021/12/14 08:30:04 Waiting for app: TestKVRebInWithOps to get deployed
2021/12/14 08:30:09 Waiting for app: TestKVRebInWithOps to get deployed
2021/12/14 08:30:14 Waiting for app: TestKVRebInWithOps to get deployed
2021/12/14 08:30:19 Waiting for app: TestKVRebInWithOps to get deployed
2021/12/14 08:30:19 App: TestKVRebInWithOps got deployed
2021/12/14 08:30:19 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/14 08:30:19 Starting bucket ops to source bucket
2021/12/14 08:30:19 Removing node(s): [] from the cluster
2021/12/14 08:30:24 Rebalance progress: 12.5
2021/12/14 08:30:29 Rebalance progress: 12.5
2021/12/14 08:30:34 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 08:30:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 08:30:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 08:30:34 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/14 08:30:40 Rebalance progress: 12.5
2021/12/14 08:30:45 Rebalance progress: 12.5
2021/12/14 08:30:50 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 08:30:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 08:30:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 08:30:50 Waiting for src bucket item count to get to: 200000 curr count: 30164
2021/12/14 08:30:55 Waiting for src bucket item count to get to: 200000 curr count: 46171
2021/12/14 08:31:00 Waiting for src bucket item count to get to: 200000 curr count: 46171
2021/12/14 08:31:05 Waiting for src bucket item count to get to: 200000 curr count: 62457
2021/12/14 08:31:10 Waiting for src bucket item count to get to: 200000 curr count: 62457
2021/12/14 08:31:15 Waiting for src bucket item count to get to: 200000 curr count: 77528
2021/12/14 08:31:20 Waiting for src bucket item count to get to: 200000 curr count: 77528
2021/12/14 08:31:25 Waiting for src bucket item count to get to: 200000 curr count: 93565
2021/12/14 08:31:30 Waiting for src bucket item count to get to: 200000 curr count: 93565
2021/12/14 08:31:35 Waiting for src bucket item count to get to: 200000 curr count: 109446
2021/12/14 08:31:40 Waiting for src bucket item count to get to: 200000 curr count: 109446
2021/12/14 08:31:45 Waiting for src bucket item count to get to: 200000 curr count: 125799
2021/12/14 08:31:50 Waiting for src bucket item count to get to: 200000 curr count: 125799
2021/12/14 08:31:55 Waiting for src bucket item count to get to: 200000 curr count: 141891
2021/12/14 08:32:00 Waiting for src bucket item count to get to: 200000 curr count: 141891
2021/12/14 08:32:05 Waiting for src bucket item count to get to: 200000 curr count: 158003
2021/12/14 08:32:10 Waiting for src bucket item count to get to: 200000 curr count: 158003
2021/12/14 08:32:15 Waiting for src bucket item count to get to: 200000 curr count: 173086
2021/12/14 08:32:20 Waiting for src bucket item count to get to: 200000 curr count: 173086
2021/12/14 08:32:25 Waiting for src bucket item count to get to: 200000 curr count: 188845
2021/12/14 08:32:30 Waiting for src bucket item count to get to: 200000 curr count: 188845
2021/12/14 08:32:35 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/14 08:32:35 Post SET ops. Expected item count: 200000 got 200000
2021/12/14 08:32:35 Starting bucket ops to source bucket
2021/12/14 08:36:25 Waiting for dst bucket item count to get to: 0 curr count: 19479
2021/12/14 08:36:30 Waiting for dst bucket item count to get to: 0 curr count: 1352
2021/12/14 08:36:35 Waiting for dst bucket item count to get to: 0 curr count: 1352
2021/12/14 08:36:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 08:36:35 Post DELETE ops. Expected item count: 0 got 0
2021/12/14 08:36:35 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 08:36:40 Waiting for app: TestKVRebInWithOps to get un-deployed
2021/12/14 08:36:45 Waiting for app: TestKVRebInWithOps to get un-deployed
2021/12/14 08:36:50 Waiting for app: TestKVRebInWithOps to get un-deployed
2021/12/14 08:36:55 Waiting for app: TestKVRebInWithOps to get un-deployed
2021/12/14 08:36:55 App: TestKVRebInWithOps got un-deployed
2021/12/14 08:36:55 No eventing-con process running
2021/12/14 08:36:55 Function: TestKVRebInWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVRebInWithOps deleting in the background"
}
2021/12/14 08:36:57 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 08:37:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 08:37:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 08:37:04 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 08:37:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 08:37:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebInWithOps (459.50s)
=== RUN   TestKVRebOutWithOps
2021/12/14 08:37:14 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/14 08:37:14 Removing node(s): [] from the cluster
2021/12/14 08:37:19 Rebalance progress: 100
2021/12/14 08:37:24 Waiting for app: TestKVRebOutWithOps to get un-deployed
2021/12/14 08:37:24 App: TestKVRebOutWithOps got un-deployed
2021/12/14 08:37:24 No eventing-con process running
2021/12/14 08:37:24 Function: TestKVRebOutWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVRebOutWithOps not found"
 }
}
2021/12/14 08:37:26 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 08:37:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 08:37:31 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 08:37:33 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 08:37:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 08:37:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 08:37:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 08:37:43 Deploying app: TestKVRebOutWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/14 08:37:44 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVRebOutWithOps' in metakv",
  "warnings": null
 }
}
2021/12/14 08:37:49 Waiting for app: TestKVRebOutWithOps to get deployed
2021/12/14 08:37:54 Waiting for app: TestKVRebOutWithOps to get deployed
2021/12/14 08:37:59 Waiting for app: TestKVRebOutWithOps to get deployed
2021/12/14 08:38:04 Waiting for app: TestKVRebOutWithOps to get deployed
2021/12/14 08:38:04 App: TestKVRebOutWithOps got deployed
2021/12/14 08:38:04 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/14 08:38:04 Starting bucket ops to source bucket
2021/12/14 08:38:09 Rebalance progress: 12.5
2021/12/14 08:38:14 Rebalance progress: 12.5
2021/12/14 08:38:19 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 08:38:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 08:38:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 08:38:19 Waiting for src bucket item count to get to: 200000 curr count: 7677
2021/12/14 08:38:24 Waiting for src bucket item count to get to: 200000 curr count: 23477
2021/12/14 08:38:29 Waiting for src bucket item count to get to: 200000 curr count: 23477
2021/12/14 08:38:34 Waiting for src bucket item count to get to: 200000 curr count: 39315
2021/12/14 08:38:39 Waiting for src bucket item count to get to: 200000 curr count: 39315
2021/12/14 08:38:44 Waiting for src bucket item count to get to: 200000 curr count: 55155
2021/12/14 08:38:49 Waiting for src bucket item count to get to: 200000 curr count: 55155
2021/12/14 08:38:54 Waiting for src bucket item count to get to: 200000 curr count: 55155
2021/12/14 08:38:59 Waiting for src bucket item count to get to: 200000 curr count: 71213
2021/12/14 08:39:04 Waiting for src bucket item count to get to: 200000 curr count: 71213
2021/12/14 08:39:09 Waiting for src bucket item count to get to: 200000 curr count: 86147
2021/12/14 08:39:14 Waiting for src bucket item count to get to: 200000 curr count: 86147
2021/12/14 08:39:19 Waiting for src bucket item count to get to: 200000 curr count: 102215
2021/12/14 08:39:24 Waiting for src bucket item count to get to: 200000 curr count: 118240
2021/12/14 08:39:29 Waiting for src bucket item count to get to: 200000 curr count: 118240
2021/12/14 08:39:34 Waiting for src bucket item count to get to: 200000 curr count: 134438
2021/12/14 08:39:39 Waiting for src bucket item count to get to: 200000 curr count: 134438
2021/12/14 08:39:44 Waiting for src bucket item count to get to: 200000 curr count: 150732
2021/12/14 08:39:49 Waiting for src bucket item count to get to: 200000 curr count: 150732
2021/12/14 08:39:54 Waiting for src bucket item count to get to: 200000 curr count: 166908
2021/12/14 08:39:59 Waiting for src bucket item count to get to: 200000 curr count: 166908
2021/12/14 08:40:04 Waiting for src bucket item count to get to: 200000 curr count: 181917
2021/12/14 08:40:09 Waiting for src bucket item count to get to: 200000 curr count: 181917
2021/12/14 08:40:14 Waiting for src bucket item count to get to: 200000 curr count: 197766
2021/12/14 08:40:19 Waiting for src bucket item count to get to: 200000 curr count: 197766
2021/12/14 08:40:24 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/14 08:40:24 Post SET ops. Expected item count: 200000 got 200000
2021/12/14 08:40:24 Starting bucket ops to source bucket
2021/12/14 08:44:15 Waiting for dst bucket item count to get to: 0 curr count: 19571
2021/12/14 08:44:20 Waiting for dst bucket item count to get to: 0 curr count: 1410
2021/12/14 08:44:25 Waiting for dst bucket item count to get to: 0 curr count: 1410
2021/12/14 08:44:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 08:44:25 Post DELETE ops. Expected item count: 0 got 0
2021/12/14 08:44:25 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 08:44:30 Waiting for app: TestKVRebOutWithOps to get un-deployed
2021/12/14 08:44:35 Waiting for app: TestKVRebOutWithOps to get un-deployed
2021/12/14 08:44:40 Waiting for app: TestKVRebOutWithOps to get un-deployed
2021/12/14 08:44:45 Waiting for app: TestKVRebOutWithOps to get un-deployed
2021/12/14 08:44:45 App: TestKVRebOutWithOps got un-deployed
2021/12/14 08:44:45 No eventing-con process running
2021/12/14 08:44:45 Function: TestKVRebOutWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVRebOutWithOps deleting in the background"
}
2021/12/14 08:44:47 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 08:44:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 08:44:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 08:44:54 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 08:44:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 08:44:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebOutWithOps (469.99s)
=== RUN   TestKVSwapRebWithOps
2021/12/14 08:45:09 Waiting for app: TestKVSwapRebWithOps to get un-deployed
2021/12/14 08:45:09 App: TestKVSwapRebWithOps got un-deployed
2021/12/14 08:45:09 No eventing-con process running
2021/12/14 08:45:09 Function: TestKVSwapRebWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVSwapRebWithOps not found"
 }
}
2021/12/14 08:45:11 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 08:45:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 08:45:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 08:45:17 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 08:45:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 08:45:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 08:45:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 08:45:27 Deploying app: TestKVSwapRebWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/14 08:45:29 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVSwapRebWithOps' in metakv",
  "warnings": null
 }
}
2021/12/14 08:45:34 Waiting for app: TestKVSwapRebWithOps to get deployed
2021/12/14 08:45:39 Waiting for app: TestKVSwapRebWithOps to get deployed
2021/12/14 08:45:44 Waiting for app: TestKVSwapRebWithOps to get deployed
2021/12/14 08:45:49 Waiting for app: TestKVSwapRebWithOps to get deployed
2021/12/14 08:45:49 App: TestKVSwapRebWithOps got deployed
2021/12/14 08:45:49 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/14 08:45:49 Starting bucket ops to source bucket
2021/12/14 08:45:49 Removing node(s): [] from the cluster
2021/12/14 08:45:54 Rebalance progress: 12.5
2021/12/14 08:45:59 Rebalance progress: 12.5
2021/12/14 08:46:04 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 08:46:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 08:46:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 08:46:04 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/14 08:46:04 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/14 08:46:09 Rebalance progress: 12.5
2021/12/14 08:46:14 Rebalance progress: 12.5
2021/12/14 08:46:19 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 08:46:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 08:46:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 08:46:19 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/14 08:46:24 Rebalance progress: 12.5
2021/12/14 08:46:29 Rebalance progress: 12.5
2021/12/14 08:46:34 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 08:46:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 08:46:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 08:46:35 Waiting for src bucket item count to get to: 200000 curr count: 62171
2021/12/14 08:46:40 Waiting for src bucket item count to get to: 200000 curr count: 62171
2021/12/14 08:46:45 Waiting for src bucket item count to get to: 200000 curr count: 77029
2021/12/14 08:46:50 Waiting for src bucket item count to get to: 200000 curr count: 77029
2021/12/14 08:46:55 Waiting for src bucket item count to get to: 200000 curr count: 92850
2021/12/14 08:47:00 Waiting for src bucket item count to get to: 200000 curr count: 92850
2021/12/14 08:47:05 Waiting for src bucket item count to get to: 200000 curr count: 108951
2021/12/14 08:47:10 Waiting for src bucket item count to get to: 200000 curr count: 108951
2021/12/14 08:47:15 Waiting for src bucket item count to get to: 200000 curr count: 124906
2021/12/14 08:47:20 Waiting for src bucket item count to get to: 200000 curr count: 124906
2021/12/14 08:47:25 Waiting for src bucket item count to get to: 200000 curr count: 141014
2021/12/14 08:47:30 Waiting for src bucket item count to get to: 200000 curr count: 141014
2021/12/14 08:47:35 Waiting for src bucket item count to get to: 200000 curr count: 141014
2021/12/14 08:47:40 Waiting for src bucket item count to get to: 200000 curr count: 156965
2021/12/14 08:47:45 Waiting for src bucket item count to get to: 200000 curr count: 171933
2021/12/14 08:47:50 Waiting for src bucket item count to get to: 200000 curr count: 171933
2021/12/14 08:47:55 Waiting for src bucket item count to get to: 200000 curr count: 187705
2021/12/14 08:48:00 Waiting for src bucket item count to get to: 200000 curr count: 187705
2021/12/14 08:48:05 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/14 08:48:05 Post SET ops. Expected item count: 200000 got 200000
2021/12/14 08:48:05 Starting bucket ops to source bucket
2021/12/14 08:51:55 Waiting for dst bucket item count to get to: 0 curr count: 19624
2021/12/14 08:52:00 Waiting for dst bucket item count to get to: 0 curr count: 1689
2021/12/14 08:52:05 Waiting for dst bucket item count to get to: 0 curr count: 1689
2021/12/14 08:52:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 08:52:05 Post DELETE ops. Expected item count: 0 got 0
2021/12/14 08:52:05 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 08:52:10 Waiting for app: TestKVSwapRebWithOps to get un-deployed
2021/12/14 08:52:15 Waiting for app: TestKVSwapRebWithOps to get un-deployed
2021/12/14 08:52:20 Waiting for app: TestKVSwapRebWithOps to get un-deployed
2021/12/14 08:52:25 Waiting for app: TestKVSwapRebWithOps to get un-deployed
2021/12/14 08:52:25 App: TestKVSwapRebWithOps got un-deployed
2021/12/14 08:52:25 No eventing-con process running
2021/12/14 08:52:25 Function: TestKVSwapRebWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVSwapRebWithOps deleting in the background"
}
2021/12/14 08:52:27 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 08:52:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 08:52:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 08:52:34 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 08:52:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 08:52:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVSwapRebWithOps (460.13s)
=== RUN   TestEventingRebInWithTimers
2021/12/14 08:52:49 Waiting for app: TestEventingRebInWithTimers to get un-deployed
2021/12/14 08:52:49 App: TestEventingRebInWithTimers got un-deployed
2021/12/14 08:52:49 No eventing-con process running
2021/12/14 08:52:49 Function: TestEventingRebInWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebInWithTimers not found"
 }
}
2021/12/14 08:52:51 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 08:52:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 08:52:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 08:52:58 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 08:53:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 08:53:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 08:53:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 08:53:08 Deploying app: TestEventingRebInWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/14 08:53:09 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebInWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/14 08:53:14 Waiting for app: TestEventingRebInWithTimers to get deployed
2021/12/14 08:53:19 Waiting for app: TestEventingRebInWithTimers to get deployed
2021/12/14 08:53:24 Waiting for app: TestEventingRebInWithTimers to get deployed
2021/12/14 08:53:29 Waiting for app: TestEventingRebInWithTimers to get deployed
2021/12/14 08:53:29 App: TestEventingRebInWithTimers got deployed
2021/12/14 08:53:29 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/14 08:53:29 Starting bucket ops to source bucket
2021/12/14 08:53:29 Removing node(s): [] from the cluster
2021/12/14 08:53:34 Rebalance progress: 12.5
2021/12/14 08:53:39 Rebalance progress: 12.5
2021/12/14 08:53:44 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 08:53:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 08:53:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 08:53:44 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/14 08:53:49 Rebalance progress: 12.5
2021/12/14 08:53:54 Rebalance progress: 12.5
2021/12/14 08:53:59 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 08:53:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 08:53:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 08:53:59 Waiting for src bucket item count to get to: 50000 curr count: 25896
2021/12/14 08:54:04 Waiting for src bucket item count to get to: 50000 curr count: 34919
2021/12/14 08:54:09 Waiting for src bucket item count to get to: 50000 curr count: 34919
2021/12/14 08:54:14 Waiting for src bucket item count to get to: 50000 curr count: 47145
2021/12/14 08:54:19 Waiting for src bucket item count to get to: 50000 curr count: 47145
2021/12/14 08:54:29 Waiting for dst bucket item count to get to: 50000 curr count: 46046
2021/12/14 08:54:34 Waiting for dst bucket item count to get to: 50000 curr count: 46046
2021/12/14 08:54:34 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/14 08:54:34 Post SET ops. Expected item count: 50000 got 50000
2021/12/14 08:54:34 Starting bucket ops to source bucket
2021/12/14 08:55:44 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/14 08:55:49 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/14 08:55:54 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/14 08:55:59 Waiting for dst bucket item count to get to: 0 curr count: 44539
2021/12/14 08:56:04 Waiting for dst bucket item count to get to: 0 curr count: 44539
2021/12/14 08:56:09 Waiting for dst bucket item count to get to: 0 curr count: 26905
2021/12/14 08:56:14 Waiting for dst bucket item count to get to: 0 curr count: 26905
2021/12/14 08:56:19 Waiting for dst bucket item count to get to: 0 curr count: 13742
2021/12/14 08:56:24 Waiting for dst bucket item count to get to: 0 curr count: 13742
2021/12/14 08:56:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 08:56:24 Post DELETE ops. Expected item count: 0 got 0
2021/12/14 08:56:25 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 08:56:30 Waiting for app: TestEventingRebInWithTimers to get un-deployed
2021/12/14 08:56:35 Waiting for app: TestEventingRebInWithTimers to get un-deployed
2021/12/14 08:56:40 Waiting for app: TestEventingRebInWithTimers to get un-deployed
2021/12/14 08:56:45 Waiting for app: TestEventingRebInWithTimers to get un-deployed
2021/12/14 08:56:45 App: TestEventingRebInWithTimers got un-deployed
2021/12/14 08:56:45 No eventing-con process running
2021/12/14 08:56:45 Function: TestEventingRebInWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebInWithTimers deleting in the background"
}
2021/12/14 08:56:47 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 08:56:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 08:56:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 08:56:53 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 08:56:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 08:56:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebInWithTimers (259.32s)
=== RUN   TestEventingRebOutWithTimers
2021/12/14 08:57:03 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/14 08:57:04 Removing node(s): [] from the cluster
2021/12/14 08:57:09 Rebalance progress: 100
2021/12/14 08:57:14 Waiting for app: TestEventingRebOutWithTimers to get un-deployed
2021/12/14 08:57:14 App: TestEventingRebOutWithTimers got un-deployed
2021/12/14 08:57:14 No eventing-con process running
2021/12/14 08:57:14 Function: TestEventingRebOutWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebOutWithTimers not found"
 }
}
2021/12/14 08:57:15 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 08:57:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 08:57:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 08:57:22 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 08:57:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 08:57:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 08:57:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 08:57:32 Deploying app: TestEventingRebOutWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/14 08:57:33 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebOutWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/14 08:57:38 Waiting for app: TestEventingRebOutWithTimers to get deployed
2021/12/14 08:57:43 Waiting for app: TestEventingRebOutWithTimers to get deployed
2021/12/14 08:57:48 Waiting for app: TestEventingRebOutWithTimers to get deployed
2021/12/14 08:57:53 Waiting for app: TestEventingRebOutWithTimers to get deployed
2021/12/14 08:57:53 App: TestEventingRebOutWithTimers got deployed
2021/12/14 08:57:53 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/14 08:57:53 Starting bucket ops to source bucket
2021/12/14 08:57:59 Rebalance progress: 12.5
2021/12/14 08:58:04 Rebalance progress: 12.5
2021/12/14 08:58:09 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 08:58:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 08:58:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 08:58:09 Waiting for src bucket item count to get to: 50000 curr count: 7673
2021/12/14 08:58:14 Waiting for src bucket item count to get to: 50000 curr count: 20089
2021/12/14 08:58:19 Waiting for src bucket item count to get to: 50000 curr count: 20089
2021/12/14 08:58:24 Waiting for src bucket item count to get to: 50000 curr count: 31039
2021/12/14 08:58:29 Waiting for src bucket item count to get to: 50000 curr count: 31039
2021/12/14 08:58:34 Waiting for src bucket item count to get to: 50000 curr count: 31039
2021/12/14 08:58:39 Waiting for src bucket item count to get to: 50000 curr count: 42047
2021/12/14 08:58:44 Waiting for src bucket item count to get to: 50000 curr count: 42047
2021/12/14 08:58:54 Waiting for dst bucket item count to get to: 50000 curr count: 40590
2021/12/14 08:58:59 Waiting for dst bucket item count to get to: 50000 curr count: 40590
2021/12/14 08:58:59 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/14 08:58:59 Post SET ops. Expected item count: 50000 got 50000
2021/12/14 08:58:59 Starting bucket ops to source bucket
2021/12/14 09:00:09 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/14 09:00:14 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/14 09:00:19 Waiting for dst bucket item count to get to: 0 curr count: 45922
2021/12/14 09:00:24 Waiting for dst bucket item count to get to: 0 curr count: 45922
2021/12/14 09:00:29 Waiting for dst bucket item count to get to: 0 curr count: 38785
2021/12/14 09:00:34 Waiting for dst bucket item count to get to: 0 curr count: 38785
2021/12/14 09:00:39 Waiting for dst bucket item count to get to: 0 curr count: 14378
2021/12/14 09:00:44 Waiting for dst bucket item count to get to: 0 curr count: 14378
2021/12/14 09:00:49 Waiting for dst bucket item count to get to: 0 curr count: 820
2021/12/14 09:00:54 Waiting for dst bucket item count to get to: 0 curr count: 820
2021/12/14 09:00:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 09:00:54 Post DELETE ops. Expected item count: 0 got 0
2021/12/14 09:00:54 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 09:00:59 Waiting for app: TestEventingRebOutWithTimers to get un-deployed
2021/12/14 09:01:04 Waiting for app: TestEventingRebOutWithTimers to get un-deployed
2021/12/14 09:01:09 Waiting for app: TestEventingRebOutWithTimers to get un-deployed
2021/12/14 09:01:14 Waiting for app: TestEventingRebOutWithTimers to get un-deployed
2021/12/14 09:01:14 App: TestEventingRebOutWithTimers got un-deployed
2021/12/14 09:01:14 No eventing-con process running
2021/12/14 09:01:14 Function: TestEventingRebOutWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebOutWithTimers deleting in the background"
}
2021/12/14 09:01:16 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:01:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 09:01:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 09:01:23 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:01:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 09:01:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebOutWithTimers (269.46s)
=== RUN   TestEventingSwapRebWithTimers
2021/12/14 09:01:38 Waiting for app: TestEventingSwapRebWithTimers to get un-deployed
2021/12/14 09:01:38 App: TestEventingSwapRebWithTimers got un-deployed
2021/12/14 09:01:38 No eventing-con process running
2021/12/14 09:01:38 Function: TestEventingSwapRebWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingSwapRebWithTimers not found"
 }
}
2021/12/14 09:01:40 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:01:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 09:01:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 09:01:46 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:01:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 09:01:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:01:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 09:01:57 Deploying app: TestEventingSwapRebWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/14 09:01:58 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingSwapRebWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/14 09:02:03 Waiting for app: TestEventingSwapRebWithTimers to get deployed
2021/12/14 09:02:08 Waiting for app: TestEventingSwapRebWithTimers to get deployed
2021/12/14 09:02:13 Waiting for app: TestEventingSwapRebWithTimers to get deployed
2021/12/14 09:02:18 Waiting for app: TestEventingSwapRebWithTimers to get deployed
2021/12/14 09:02:18 App: TestEventingSwapRebWithTimers got deployed
2021/12/14 09:02:18 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/14 09:02:18 Starting bucket ops to source bucket
2021/12/14 09:02:18 Removing node(s): [] from the cluster
2021/12/14 09:02:23 Rebalance progress: 12.5
2021/12/14 09:02:28 Rebalance progress: 12.5
2021/12/14 09:02:33 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:02:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:02:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 09:02:33 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/14 09:02:34 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/14 09:02:39 Rebalance progress: 12.5
2021/12/14 09:02:44 Rebalance progress: 12.5
2021/12/14 09:02:49 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:02:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:02:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 09:02:49 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/14 09:02:54 Rebalance progress: 12.5
2021/12/14 09:02:59 Rebalance progress: 12.5
2021/12/14 09:03:04 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:03:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:03:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 09:03:04 Waiting for src bucket item count to get to: 50000 curr count: 49656
2021/12/14 09:03:09 Waiting for src bucket item count to get to: 50000 curr count: 49656
2021/12/14 09:03:19 Waiting for dst bucket item count to get to: 50000 curr count: 41213
2021/12/14 09:03:24 Waiting for dst bucket item count to get to: 50000 curr count: 41213
2021/12/14 09:03:29 Waiting for dst bucket item count to get to: 50000 curr count: 49406
2021/12/14 09:03:34 Waiting for dst bucket item count to get to: 50000 curr count: 49406
2021/12/14 09:03:34 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/14 09:03:34 Post SET ops. Expected item count: 50000 got 50000
2021/12/14 09:03:34 Starting bucket ops to source bucket
2021/12/14 09:04:45 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/14 09:04:50 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/14 09:04:55 Waiting for dst bucket item count to get to: 0 curr count: 48959
2021/12/14 09:05:00 Waiting for dst bucket item count to get to: 0 curr count: 48959
2021/12/14 09:05:05 Waiting for dst bucket item count to get to: 0 curr count: 33280
2021/12/14 09:05:10 Waiting for dst bucket item count to get to: 0 curr count: 33280
2021/12/14 09:05:15 Waiting for dst bucket item count to get to: 0 curr count: 17941
2021/12/14 09:05:20 Waiting for dst bucket item count to get to: 0 curr count: 17941
2021/12/14 09:05:25 Waiting for dst bucket item count to get to: 0 curr count: 4791
2021/12/14 09:05:30 Waiting for dst bucket item count to get to: 0 curr count: 4791
2021/12/14 09:05:30 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 09:05:30 Post DELETE ops. Expected item count: 0 got 0
2021/12/14 09:05:30 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 09:05:35 Waiting for app: TestEventingSwapRebWithTimers to get un-deployed
2021/12/14 09:05:40 Waiting for app: TestEventingSwapRebWithTimers to get un-deployed
2021/12/14 09:05:46 Waiting for app: TestEventingSwapRebWithTimers to get un-deployed
2021/12/14 09:05:51 Waiting for app: TestEventingSwapRebWithTimers to get un-deployed
2021/12/14 09:05:51 App: TestEventingSwapRebWithTimers got un-deployed
2021/12/14 09:05:51 No eventing-con process running
2021/12/14 09:05:51 Function: TestEventingSwapRebWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingSwapRebWithTimers deleting in the background"
}
2021/12/14 09:05:52 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:05:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 09:05:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 09:05:59 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:06:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 09:06:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingSwapRebWithTimers (276.37s)
=== RUN   TestKVRebInWithTimers
2021/12/14 09:06:14 Waiting for app: TestKVRebInWithTimers to get un-deployed
2021/12/14 09:06:14 App: TestKVRebInWithTimers got un-deployed
2021/12/14 09:06:14 No eventing-con process running
2021/12/14 09:06:14 Function: TestKVRebInWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVRebInWithTimers not found"
 }
}
2021/12/14 09:06:16 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:06:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 09:06:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 09:06:23 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:06:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 09:06:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:06:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 09:06:33 Deploying app: TestKVRebInWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/14 09:06:34 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVRebInWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/14 09:06:39 Waiting for app: TestKVRebInWithTimers to get deployed
2021/12/14 09:06:44 Waiting for app: TestKVRebInWithTimers to get deployed
2021/12/14 09:06:49 Waiting for app: TestKVRebInWithTimers to get deployed
2021/12/14 09:06:54 Waiting for app: TestKVRebInWithTimers to get deployed
2021/12/14 09:06:54 App: TestKVRebInWithTimers got deployed
2021/12/14 09:06:54 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/14 09:06:54 Starting bucket ops to source bucket
2021/12/14 09:06:54 Removing node(s): [] from the cluster
2021/12/14 09:06:59 Rebalance progress: 12.5
2021/12/14 09:07:04 Rebalance progress: 12.5
2021/12/14 09:07:09 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:07:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:07:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 09:07:09 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/14 09:07:14 Rebalance progress: 12.5
2021/12/14 09:07:19 Rebalance progress: 12.5
2021/12/14 09:07:24 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:07:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:07:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 09:07:24 Waiting for src bucket item count to get to: 50000 curr count: 30428
2021/12/14 09:07:29 Waiting for src bucket item count to get to: 50000 curr count: 30428
2021/12/14 09:07:34 Waiting for src bucket item count to get to: 50000 curr count: 41665
2021/12/14 09:07:39 Waiting for src bucket item count to get to: 50000 curr count: 41665
2021/12/14 09:07:49 Waiting for dst bucket item count to get to: 50000 curr count: 38360
2021/12/14 09:07:54 Waiting for dst bucket item count to get to: 50000 curr count: 38360
2021/12/14 09:07:59 Waiting for dst bucket item count to get to: 50000 curr count: 47121
2021/12/14 09:08:04 Waiting for dst bucket item count to get to: 50000 curr count: 47121
2021/12/14 09:08:04 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/14 09:08:04 Post SET ops. Expected item count: 50000 got 50000
2021/12/14 09:08:04 Starting bucket ops to source bucket
2021/12/14 09:09:15 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/14 09:09:20 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/14 09:09:25 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/14 09:09:30 Waiting for dst bucket item count to get to: 0 curr count: 49132
2021/12/14 09:09:35 Waiting for dst bucket item count to get to: 0 curr count: 49132
2021/12/14 09:09:40 Waiting for dst bucket item count to get to: 0 curr count: 31265
2021/12/14 09:09:45 Waiting for dst bucket item count to get to: 0 curr count: 31265
2021/12/14 09:09:50 Waiting for dst bucket item count to get to: 0 curr count: 13412
2021/12/14 09:09:55 Waiting for dst bucket item count to get to: 0 curr count: 13412
2021/12/14 09:09:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 09:09:55 Post DELETE ops. Expected item count: 0 got 0
2021/12/14 09:09:56 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 09:10:01 Waiting for app: TestKVRebInWithTimers to get un-deployed
2021/12/14 09:10:06 Waiting for app: TestKVRebInWithTimers to get un-deployed
2021/12/14 09:10:11 Waiting for app: TestKVRebInWithTimers to get un-deployed
2021/12/14 09:10:16 Waiting for app: TestKVRebInWithTimers to get un-deployed
2021/12/14 09:10:16 App: TestKVRebInWithTimers got un-deployed
2021/12/14 09:10:16 No eventing-con process running
2021/12/14 09:10:16 Function: TestKVRebInWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVRebInWithTimers deleting in the background"
}
2021/12/14 09:10:18 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:10:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 09:10:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 09:10:24 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:10:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 09:10:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebInWithTimers (265.32s)
=== RUN   TestKVRebOutWithTimers
2021/12/14 09:10:34 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/14 09:10:35 Removing node(s): [] from the cluster
2021/12/14 09:10:40 Rebalance progress: 100
2021/12/14 09:10:45 Waiting for app: TestKVRebOutWithTimers to get un-deployed
2021/12/14 09:10:45 App: TestKVRebOutWithTimers got un-deployed
2021/12/14 09:10:45 No eventing-con process running
2021/12/14 09:10:45 Function: TestKVRebOutWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVRebOutWithTimers not found"
 }
}
2021/12/14 09:10:46 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:10:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 09:10:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 09:10:53 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:10:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 09:10:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:11:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 09:11:03 Deploying app: TestKVRebOutWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/14 09:11:04 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVRebOutWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/14 09:11:09 Waiting for app: TestKVRebOutWithTimers to get deployed
2021/12/14 09:11:14 Waiting for app: TestKVRebOutWithTimers to get deployed
2021/12/14 09:11:19 Waiting for app: TestKVRebOutWithTimers to get deployed
2021/12/14 09:11:24 Waiting for app: TestKVRebOutWithTimers to get deployed
2021/12/14 09:11:24 App: TestKVRebOutWithTimers got deployed
2021/12/14 09:11:24 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/14 09:11:24 Starting bucket ops to source bucket
2021/12/14 09:11:30 Rebalance progress: 12.5
2021/12/14 09:11:35 Rebalance progress: 12.5
2021/12/14 09:11:40 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:11:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:11:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 09:11:40 Waiting for src bucket item count to get to: 50000 curr count: 6093
2021/12/14 09:11:45 Waiting for src bucket item count to get to: 50000 curr count: 19569
2021/12/14 09:11:50 Waiting for src bucket item count to get to: 50000 curr count: 19569
2021/12/14 09:11:55 Waiting for src bucket item count to get to: 50000 curr count: 29562
2021/12/14 09:12:00 Waiting for src bucket item count to get to: 50000 curr count: 29562
2021/12/14 09:12:05 Waiting for src bucket item count to get to: 50000 curr count: 41778
2021/12/14 09:12:10 Waiting for src bucket item count to get to: 50000 curr count: 41778
2021/12/14 09:12:20 Waiting for dst bucket item count to get to: 50000 curr count: 41350
2021/12/14 09:12:25 Waiting for dst bucket item count to get to: 50000 curr count: 41350
2021/12/14 09:12:30 Waiting for dst bucket item count to get to: 50000 curr count: 49492
2021/12/14 09:12:35 Waiting for dst bucket item count to get to: 50000 curr count: 49492
2021/12/14 09:12:35 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/14 09:12:35 Post SET ops. Expected item count: 50000 got 50000
2021/12/14 09:12:35 Starting bucket ops to source bucket
2021/12/14 09:13:45 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/14 09:13:50 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/14 09:13:55 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/14 09:14:00 Waiting for dst bucket item count to get to: 0 curr count: 42294
2021/12/14 09:14:05 Waiting for dst bucket item count to get to: 0 curr count: 42294
2021/12/14 09:14:10 Waiting for dst bucket item count to get to: 0 curr count: 25347
2021/12/14 09:14:15 Waiting for dst bucket item count to get to: 0 curr count: 25347
2021/12/14 09:14:20 Waiting for dst bucket item count to get to: 0 curr count: 8001
2021/12/14 09:14:25 Waiting for dst bucket item count to get to: 0 curr count: 8001
2021/12/14 09:14:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 09:14:25 Post DELETE ops. Expected item count: 0 got 0
2021/12/14 09:14:25 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 09:14:30 Waiting for app: TestKVRebOutWithTimers to get un-deployed
2021/12/14 09:14:35 Waiting for app: TestKVRebOutWithTimers to get un-deployed
2021/12/14 09:14:40 Waiting for app: TestKVRebOutWithTimers to get un-deployed
2021/12/14 09:14:45 Waiting for app: TestKVRebOutWithTimers to get un-deployed
2021/12/14 09:14:45 App: TestKVRebOutWithTimers got un-deployed
2021/12/14 09:14:46 No eventing-con process running
2021/12/14 09:14:46 Function: TestKVRebOutWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVRebOutWithTimers deleting in the background"
}
2021/12/14 09:14:47 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:14:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 09:14:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 09:14:54 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:14:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 09:14:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebOutWithTimers (269.67s)
=== RUN   TestKVSwapRebWithTimers
2021/12/14 09:15:09 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/14 09:15:09 App: TestKVSwapRebWithTimers got un-deployed
2021/12/14 09:15:09 No eventing-con process running
2021/12/14 09:15:09 Function: TestKVSwapRebWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVSwapRebWithTimers not found"
 }
}
2021/12/14 09:15:11 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:15:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 09:15:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 09:15:18 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:15:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 09:15:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:15:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 09:15:28 Deploying app: TestKVSwapRebWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/14 09:15:29 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVSwapRebWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/14 09:15:34 Waiting for app: TestKVSwapRebWithTimers to get deployed
2021/12/14 09:15:39 Waiting for app: TestKVSwapRebWithTimers to get deployed
2021/12/14 09:15:44 Waiting for app: TestKVSwapRebWithTimers to get deployed
2021/12/14 09:15:49 Waiting for app: TestKVSwapRebWithTimers to get deployed
2021/12/14 09:15:49 App: TestKVSwapRebWithTimers got deployed
2021/12/14 09:15:49 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/14 09:15:49 Starting bucket ops to source bucket
2021/12/14 09:15:49 Removing node(s): [] from the cluster
2021/12/14 09:15:54 Rebalance progress: 12.5
2021/12/14 09:15:59 Rebalance progress: 12.5
2021/12/14 09:16:04 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:16:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:16:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 09:16:04 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/14 09:16:05 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/14 09:16:10 Rebalance progress: 12.5
2021/12/14 09:16:15 Rebalance progress: 12.5
2021/12/14 09:16:20 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:16:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:16:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 09:16:20 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/14 09:16:25 Rebalance progress: 12.5
2021/12/14 09:16:30 Rebalance progress: 12.5
2021/12/14 09:16:35 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:16:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:16:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 09:16:35 Waiting for src bucket item count to get to: 50000 curr count: 46995
2021/12/14 09:16:40 Waiting for src bucket item count to get to: 50000 curr count: 46995
2021/12/14 09:16:50 Waiting for dst bucket item count to get to: 50000 curr count: 41417
2021/12/14 09:16:55 Waiting for dst bucket item count to get to: 50000 curr count: 41417
2021/12/14 09:17:00 Waiting for dst bucket item count to get to: 50000 curr count: 48272
2021/12/14 09:17:05 Waiting for dst bucket item count to get to: 50000 curr count: 48272
2021/12/14 09:17:05 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/14 09:17:05 Post SET ops. Expected item count: 50000 got 50000
2021/12/14 09:17:05 Starting bucket ops to source bucket
2021/12/14 09:18:15 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/14 09:18:20 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/14 09:18:25 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/14 09:18:30 Waiting for dst bucket item count to get to: 0 curr count: 46892
2021/12/14 09:18:35 Waiting for dst bucket item count to get to: 0 curr count: 46892
2021/12/14 09:18:40 Waiting for dst bucket item count to get to: 0 curr count: 35924
2021/12/14 09:18:45 Waiting for dst bucket item count to get to: 0 curr count: 35924
2021/12/14 09:18:50 Waiting for dst bucket item count to get to: 0 curr count: 14640
2021/12/14 09:18:55 Waiting for dst bucket item count to get to: 0 curr count: 14640
2021/12/14 09:19:00 Waiting for dst bucket item count to get to: 0 curr count: 2570
2021/12/14 09:19:05 Waiting for dst bucket item count to get to: 0 curr count: 2570
2021/12/14 09:19:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 09:19:05 Post DELETE ops. Expected item count: 0 got 0
2021/12/14 09:19:05 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 09:19:10 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/14 09:19:15 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/14 09:19:20 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/14 09:19:25 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/14 09:19:30 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/14 09:19:35 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/14 09:19:35 App: TestKVSwapRebWithTimers got un-deployed
2021/12/14 09:19:35 No eventing-con process running
2021/12/14 09:19:35 Function: TestKVSwapRebWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVSwapRebWithTimers deleting in the background"
}
2021/12/14 09:19:37 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:19:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 09:19:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 09:19:44 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:19:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 09:19:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVSwapRebWithTimers (289.89s)
=== RUN   TestDuplicateTimerCreationEventingRebIn
2021/12/14 09:19:59 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/14 09:19:59 App: TestDuplicateTimerCreationEventingRebIn got un-deployed
2021/12/14 09:19:59 No eventing-con process running
2021/12/14 09:19:59 Function: TestDuplicateTimerCreationEventingRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationEventingRebIn not found"
 }
}
2021/12/14 09:20:01 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:20:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 09:20:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 09:20:08 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:20:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 09:20:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:20:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 09:20:18 Deploying app: TestDuplicateTimerCreationEventingRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/14 09:20:19 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationEventingRebIn' in metakv",
  "warnings": null
 }
}
2021/12/14 09:20:24 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get deployed
2021/12/14 09:20:29 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get deployed
2021/12/14 09:20:34 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get deployed
2021/12/14 09:20:39 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get deployed
2021/12/14 09:20:39 App: TestDuplicateTimerCreationEventingRebIn got deployed
2021/12/14 09:20:39 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/14 09:20:39 Starting bucket ops to source bucket
2021/12/14 09:20:39 Removing node(s): [] from the cluster
2021/12/14 09:20:44 Rebalance progress: 12.5
2021/12/14 09:20:49 Rebalance progress: 12.5
2021/12/14 09:20:54 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:20:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:20:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 09:20:54 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/14 09:20:59 Rebalance progress: 12.5
2021/12/14 09:21:04 Rebalance progress: 12.5
2021/12/14 09:21:09 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:21:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:21:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 09:21:09 Waiting for src bucket item count to get to: 50000 curr count: 26350
2021/12/14 09:21:14 Waiting for src bucket item count to get to: 50000 curr count: 36589
2021/12/14 09:21:19 Waiting for src bucket item count to get to: 50000 curr count: 36589
2021/12/14 09:21:24 Waiting for src bucket item count to get to: 50000 curr count: 47329
2021/12/14 09:21:29 Waiting for src bucket item count to get to: 50000 curr count: 47329
2021/12/14 09:21:39 Waiting for dst bucket item count to get to: 50000 curr count: 38922
2021/12/14 09:21:44 Waiting for dst bucket item count to get to: 50000 curr count: 38922
2021/12/14 09:21:49 Waiting for dst bucket item count to get to: 50000 curr count: 46305
2021/12/14 09:21:54 Waiting for dst bucket item count to get to: 50000 curr count: 46305
2021/12/14 09:21:54 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/14 09:21:54 Post SET ops. Expected item count: 50000 got 50000
2021/12/14 09:21:54 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 09:21:59 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/14 09:22:04 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/14 09:22:10 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/14 09:22:15 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/14 09:22:20 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/14 09:22:25 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/14 09:22:25 App: TestDuplicateTimerCreationEventingRebIn got un-deployed
2021/12/14 09:22:25 No eventing-con process running
2021/12/14 09:22:25 Function: TestDuplicateTimerCreationEventingRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationEventingRebIn deleting in the background"
}
2021/12/14 09:22:26 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:22:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 09:22:36 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/14 09:22:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 09:22:38 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:22:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 09:22:48 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/14 09:22:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationEventingRebIn (179.27s)
=== RUN   TestDuplicateTimerCreationEventingRebOut
2021/12/14 09:22:53 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/14 09:22:53 Removing node(s): [] from the cluster
2021/12/14 09:22:58 Rebalance progress: 100
2021/12/14 09:23:03 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/14 09:23:03 App: TestDuplicateTimerCreationEventingRebOut got un-deployed
2021/12/14 09:23:03 No eventing-con process running
2021/12/14 09:23:03 Function: TestDuplicateTimerCreationEventingRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationEventingRebOut not found"
 }
}
2021/12/14 09:23:05 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:23:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 09:23:10 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 09:23:12 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:23:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 09:23:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:23:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 09:23:22 Deploying app: TestDuplicateTimerCreationEventingRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/14 09:23:23 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationEventingRebOut' in metakv",
  "warnings": null
 }
}
2021/12/14 09:23:28 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get deployed
2021/12/14 09:23:33 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get deployed
2021/12/14 09:23:38 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get deployed
2021/12/14 09:23:43 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get deployed
2021/12/14 09:23:43 App: TestDuplicateTimerCreationEventingRebOut got deployed
2021/12/14 09:23:43 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/14 09:23:43 Starting bucket ops to source bucket
2021/12/14 09:23:48 Rebalance progress: 12.5
2021/12/14 09:23:53 Rebalance progress: 12.5
2021/12/14 09:23:58 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:23:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:23:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 09:23:58 Waiting for src bucket item count to get to: 50000 curr count: 7564
2021/12/14 09:24:03 Waiting for src bucket item count to get to: 50000 curr count: 21555
2021/12/14 09:24:08 Waiting for src bucket item count to get to: 50000 curr count: 21555
2021/12/14 09:24:13 Waiting for src bucket item count to get to: 50000 curr count: 35139
2021/12/14 09:24:18 Waiting for src bucket item count to get to: 50000 curr count: 35139
2021/12/14 09:24:23 Waiting for src bucket item count to get to: 50000 curr count: 45041
2021/12/14 09:24:28 Waiting for src bucket item count to get to: 50000 curr count: 45041
2021/12/14 09:24:33 Waiting for src bucket item count to get to: 50000 curr count: 45041
2021/12/14 09:24:43 Waiting for dst bucket item count to get to: 50000 curr count: 31084
2021/12/14 09:24:48 Waiting for dst bucket item count to get to: 50000 curr count: 31084
2021/12/14 09:24:53 Waiting for dst bucket item count to get to: 50000 curr count: 39789
2021/12/14 09:24:58 Waiting for dst bucket item count to get to: 50000 curr count: 39789
2021/12/14 09:24:58 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/14 09:24:58 Post SET ops. Expected item count: 50000 got 50000
2021/12/14 09:24:59 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 09:25:04 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/14 09:25:09 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/14 09:25:14 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/14 09:25:19 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/14 09:25:24 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/14 09:25:29 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/14 09:25:29 App: TestDuplicateTimerCreationEventingRebOut got un-deployed
2021/12/14 09:25:29 No eventing-con process running
2021/12/14 09:25:29 Function: TestDuplicateTimerCreationEventingRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationEventingRebOut deleting in the background"
}
2021/12/14 09:25:31 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:25:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 09:25:41 Waiting for dst bucket item count to get to: 0 curr count: 30748
2021/12/14 09:25:46 Waiting for dst bucket item count to get to: 0 curr count: 30748
2021/12/14 09:25:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 09:25:47 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:25:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 09:25:57 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/14 09:25:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationEventingRebOut (189.13s)
=== RUN   TestDuplicateTimerCreationEventingSwapReb
2021/12/14 09:26:07 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/14 09:26:07 App: TestDuplicateTimerCreationEventingSwapReb got un-deployed
2021/12/14 09:26:07 No eventing-con process running
2021/12/14 09:26:07 Function: TestDuplicateTimerCreationEventingSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationEventingSwapReb not found"
 }
}
2021/12/14 09:26:09 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:26:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 09:26:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 09:26:16 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:26:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 09:26:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:26:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 09:26:26 Deploying app: TestDuplicateTimerCreationEventingSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/14 09:26:27 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationEventingSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/14 09:26:32 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get deployed
2021/12/14 09:26:37 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get deployed
2021/12/14 09:26:42 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get deployed
2021/12/14 09:26:47 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get deployed
2021/12/14 09:26:47 App: TestDuplicateTimerCreationEventingSwapReb got deployed
2021/12/14 09:26:47 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/14 09:26:47 Starting bucket ops to source bucket
2021/12/14 09:26:48 Removing node(s): [] from the cluster
2021/12/14 09:26:53 Rebalance progress: 12.5
2021/12/14 09:26:58 Rebalance progress: 12.5
2021/12/14 09:27:03 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:27:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:27:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 09:27:03 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/14 09:27:04 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/14 09:27:09 Rebalance progress: 12.5
2021/12/14 09:27:14 Rebalance progress: 12.5
2021/12/14 09:27:19 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:27:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:27:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 09:27:19 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/14 09:27:24 Rebalance progress: 12.5
2021/12/14 09:27:29 Rebalance progress: 12.5
2021/12/14 09:27:34 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:27:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:27:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 09:27:34 Waiting for src bucket item count to get to: 50000 curr count: 49700
2021/12/14 09:27:39 Waiting for src bucket item count to get to: 50000 curr count: 49700
2021/12/14 09:27:49 Waiting for dst bucket item count to get to: 50000 curr count: 36113
2021/12/14 09:27:54 Waiting for dst bucket item count to get to: 50000 curr count: 36113
2021/12/14 09:27:59 Waiting for dst bucket item count to get to: 50000 curr count: 42261
2021/12/14 09:28:04 Waiting for dst bucket item count to get to: 50000 curr count: 42261
2021/12/14 09:28:04 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/14 09:28:04 Post SET ops. Expected item count: 50000 got 50000
2021/12/14 09:28:04 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 09:28:09 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/14 09:28:14 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/14 09:28:19 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/14 09:28:24 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/14 09:28:29 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/14 09:28:34 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/14 09:28:34 App: TestDuplicateTimerCreationEventingSwapReb got un-deployed
2021/12/14 09:28:34 No eventing-con process running
2021/12/14 09:28:34 Function: TestDuplicateTimerCreationEventingSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationEventingSwapReb deleting in the background"
}
2021/12/14 09:28:36 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:28:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 09:28:46 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/14 09:28:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 09:28:48 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:28:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 09:28:58 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/14 09:28:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationEventingSwapReb (180.48s)
=== RUN   TestDuplicateTimerCreationKVRebIn
2021/12/14 09:29:08 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/14 09:29:08 App: TestDuplicateTimerCreationKVRebIn got un-deployed
2021/12/14 09:29:08 No eventing-con process running
2021/12/14 09:29:08 Function: TestDuplicateTimerCreationKVRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationKVRebIn not found"
 }
}
2021/12/14 09:29:10 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:29:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 09:29:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 09:29:17 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:29:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 09:29:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:29:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 09:29:27 Deploying app: TestDuplicateTimerCreationKVRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/14 09:29:28 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationKVRebIn' in metakv",
  "warnings": null
 }
}
2021/12/14 09:29:33 Waiting for app: TestDuplicateTimerCreationKVRebIn to get deployed
2021/12/14 09:29:38 Waiting for app: TestDuplicateTimerCreationKVRebIn to get deployed
2021/12/14 09:29:43 Waiting for app: TestDuplicateTimerCreationKVRebIn to get deployed
2021/12/14 09:29:48 Waiting for app: TestDuplicateTimerCreationKVRebIn to get deployed
2021/12/14 09:29:48 App: TestDuplicateTimerCreationKVRebIn got deployed
2021/12/14 09:29:48 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/14 09:29:48 Starting bucket ops to source bucket
2021/12/14 09:29:49 Removing node(s): [] from the cluster
2021/12/14 09:29:54 Rebalance progress: 12.5
2021/12/14 09:29:59 Rebalance progress: 12.5
2021/12/14 09:30:04 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:30:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:30:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 09:30:04 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/14 09:30:09 Rebalance progress: 12.5
2021/12/14 09:30:14 Rebalance progress: 12.5
2021/12/14 09:30:19 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:30:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:30:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 09:30:19 Waiting for src bucket item count to get to: 50000 curr count: 27487
2021/12/14 09:30:24 Waiting for src bucket item count to get to: 50000 curr count: 38584
2021/12/14 09:30:29 Waiting for src bucket item count to get to: 50000 curr count: 38584
2021/12/14 09:30:34 Waiting for src bucket item count to get to: 50000 curr count: 49700
2021/12/14 09:30:39 Waiting for src bucket item count to get to: 50000 curr count: 49700
2021/12/14 09:30:49 Waiting for dst bucket item count to get to: 50000 curr count: 35353
2021/12/14 09:30:54 Waiting for dst bucket item count to get to: 50000 curr count: 35353
2021/12/14 09:30:59 Waiting for dst bucket item count to get to: 50000 curr count: 42385
2021/12/14 09:31:04 Waiting for dst bucket item count to get to: 50000 curr count: 42385
2021/12/14 09:31:04 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/14 09:31:04 Post SET ops. Expected item count: 50000 got 50000
2021/12/14 09:31:04 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 09:31:09 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/14 09:31:14 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/14 09:31:19 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/14 09:31:24 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/14 09:31:29 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/14 09:31:34 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/14 09:31:34 App: TestDuplicateTimerCreationKVRebIn got un-deployed
2021/12/14 09:31:34 No eventing-con process running
2021/12/14 09:31:34 Function: TestDuplicateTimerCreationKVRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationKVRebIn deleting in the background"
}
2021/12/14 09:31:36 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:31:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 09:31:46 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/14 09:31:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 09:31:48 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:31:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 09:31:58 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/14 09:31:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationKVRebIn (179.83s)
=== RUN   TestDuplicateTimerCreationKVRebOut
2021/12/14 09:32:03 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/14 09:32:03 Removing node(s): [] from the cluster
2021/12/14 09:32:08 Rebalance progress: 100
2021/12/14 09:32:13 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/14 09:32:13 App: TestDuplicateTimerCreationKVRebOut got un-deployed
2021/12/14 09:32:13 No eventing-con process running
2021/12/14 09:32:13 Function: TestDuplicateTimerCreationKVRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationKVRebOut not found"
 }
}
2021/12/14 09:32:15 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:32:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 09:32:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 09:32:22 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:32:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 09:32:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:32:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 09:32:32 Deploying app: TestDuplicateTimerCreationKVRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/14 09:32:33 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationKVRebOut' in metakv",
  "warnings": null
 }
}
2021/12/14 09:32:38 Waiting for app: TestDuplicateTimerCreationKVRebOut to get deployed
2021/12/14 09:32:43 Waiting for app: TestDuplicateTimerCreationKVRebOut to get deployed
2021/12/14 09:32:48 Waiting for app: TestDuplicateTimerCreationKVRebOut to get deployed
2021/12/14 09:32:53 Waiting for app: TestDuplicateTimerCreationKVRebOut to get deployed
2021/12/14 09:32:53 App: TestDuplicateTimerCreationKVRebOut got deployed
2021/12/14 09:32:53 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/14 09:32:53 Starting bucket ops to source bucket
2021/12/14 09:32:58 Rebalance progress: 12.5
2021/12/14 09:33:03 Rebalance progress: 12.5
2021/12/14 09:33:08 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:33:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:33:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 09:33:08 Waiting for src bucket item count to get to: 50000 curr count: 7983
2021/12/14 09:33:13 Waiting for src bucket item count to get to: 50000 curr count: 21610
2021/12/14 09:33:18 Waiting for src bucket item count to get to: 50000 curr count: 21610
2021/12/14 09:33:23 Waiting for src bucket item count to get to: 50000 curr count: 33474
2021/12/14 09:33:28 Waiting for src bucket item count to get to: 50000 curr count: 33474
2021/12/14 09:33:33 Waiting for src bucket item count to get to: 50000 curr count: 43249
2021/12/14 09:33:38 Waiting for src bucket item count to get to: 50000 curr count: 43249
2021/12/14 09:33:43 Waiting for src bucket item count to get to: 50000 curr count: 43249
2021/12/14 09:33:53 Waiting for dst bucket item count to get to: 50000 curr count: 33714
2021/12/14 09:33:58 Waiting for dst bucket item count to get to: 50000 curr count: 33714
2021/12/14 09:34:03 Waiting for dst bucket item count to get to: 50000 curr count: 45217
2021/12/14 09:34:08 Waiting for dst bucket item count to get to: 50000 curr count: 45217
2021/12/14 09:34:08 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/14 09:34:08 Post SET ops. Expected item count: 50000 got 50000
2021/12/14 09:34:08 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 09:34:13 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/14 09:34:18 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/14 09:34:23 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/14 09:34:28 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/14 09:34:33 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/14 09:34:38 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/14 09:34:38 App: TestDuplicateTimerCreationKVRebOut got un-deployed
2021/12/14 09:34:38 No eventing-con process running
2021/12/14 09:34:39 Function: TestDuplicateTimerCreationKVRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationKVRebOut deleting in the background"
}
2021/12/14 09:34:40 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:34:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 09:34:50 Waiting for dst bucket item count to get to: 0 curr count: 21052
2021/12/14 09:34:55 Waiting for dst bucket item count to get to: 0 curr count: 21052
2021/12/14 09:34:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 09:34:57 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:35:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 09:35:07 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/14 09:35:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationKVRebOut (189.47s)
=== RUN   TestDuplicateTimerCreationKVSwapReb
2021/12/14 09:35:17 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/14 09:35:17 App: TestDuplicateTimerCreationKVSwapReb got un-deployed
2021/12/14 09:35:17 No eventing-con process running
2021/12/14 09:35:17 Function: TestDuplicateTimerCreationKVSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationKVSwapReb not found"
 }
}
2021/12/14 09:35:19 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:35:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 09:35:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 09:35:26 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:35:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 09:35:31 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:35:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 09:35:36 Deploying app: TestDuplicateTimerCreationKVSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/14 09:35:37 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationKVSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/14 09:35:42 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get deployed
2021/12/14 09:35:47 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get deployed
2021/12/14 09:35:52 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get deployed
2021/12/14 09:35:57 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get deployed
2021/12/14 09:35:57 App: TestDuplicateTimerCreationKVSwapReb got deployed
2021/12/14 09:35:57 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/14 09:35:57 Starting bucket ops to source bucket
2021/12/14 09:35:57 Removing node(s): [] from the cluster
2021/12/14 09:36:02 Rebalance progress: 12.5
2021/12/14 09:36:07 Rebalance progress: 12.5
2021/12/14 09:36:12 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:36:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:36:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 09:36:12 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/14 09:36:13 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/14 09:36:18 Rebalance progress: 12.5
2021/12/14 09:36:23 Rebalance progress: 12.5
2021/12/14 09:36:28 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:36:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:36:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 09:36:28 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/14 09:36:33 Rebalance progress: 12.5
2021/12/14 09:36:38 Rebalance progress: 12.5
2021/12/14 09:36:43 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:36:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:36:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 09:36:43 Waiting for src bucket item count to get to: 50000 curr count: 48264
2021/12/14 09:36:48 Waiting for src bucket item count to get to: 50000 curr count: 48264
2021/12/14 09:36:58 Waiting for dst bucket item count to get to: 50000 curr count: 36882
2021/12/14 09:37:03 Waiting for dst bucket item count to get to: 50000 curr count: 36882
2021/12/14 09:37:08 Waiting for dst bucket item count to get to: 50000 curr count: 44341
2021/12/14 09:37:13 Waiting for dst bucket item count to get to: 50000 curr count: 44341
2021/12/14 09:37:13 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/14 09:37:13 Post SET ops. Expected item count: 50000 got 50000
2021/12/14 09:37:13 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 09:37:18 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/14 09:37:23 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/14 09:37:28 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/14 09:37:33 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/14 09:37:38 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/14 09:37:43 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/14 09:37:43 App: TestDuplicateTimerCreationKVSwapReb got un-deployed
2021/12/14 09:37:43 No eventing-con process running
2021/12/14 09:37:43 Function: TestDuplicateTimerCreationKVSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationKVSwapReb deleting in the background"
}
2021/12/14 09:37:45 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:37:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 09:37:55 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/14 09:37:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 09:37:57 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:38:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 09:38:07 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/14 09:38:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationKVSwapReb (179.74s)
=== RUN   TestDuplicateTimerFiringEventingRebIn
2021/12/14 09:38:17 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/14 09:38:17 App: TestDuplicateTimerFiringEventingRebIn got un-deployed
2021/12/14 09:38:17 No eventing-con process running
2021/12/14 09:38:17 Function: TestDuplicateTimerFiringEventingRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringEventingRebIn not found"
 }
}
2021/12/14 09:38:19 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:38:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 09:38:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 09:38:26 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:38:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 09:38:31 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:38:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 09:38:36 Deploying app: TestDuplicateTimerFiringEventingRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/14 09:38:37 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringEventingRebIn' in metakv",
  "warnings": null
 }
}
2021/12/14 09:38:42 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get deployed
2021/12/14 09:38:47 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get deployed
2021/12/14 09:38:52 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get deployed
2021/12/14 09:38:57 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get deployed
2021/12/14 09:38:57 App: TestDuplicateTimerFiringEventingRebIn got deployed
2021/12/14 09:38:57 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/14 09:38:57 Starting bucket ops to source bucket
2021/12/14 09:38:57 Removing node(s): [] from the cluster
2021/12/14 09:39:02 Rebalance progress: 12.5
2021/12/14 09:39:07 Rebalance progress: 12.5
2021/12/14 09:39:12 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:39:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:39:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 09:39:12 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/14 09:39:17 Rebalance progress: 12.5
2021/12/14 09:39:22 Rebalance progress: 12.5
2021/12/14 09:39:27 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:39:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:39:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 09:39:27 Waiting for src bucket item count to get to: 50000 curr count: 25981
2021/12/14 09:39:32 Waiting for src bucket item count to get to: 50000 curr count: 37691
2021/12/14 09:39:37 Waiting for src bucket item count to get to: 50000 curr count: 37691
2021/12/14 09:39:42 Waiting for src bucket item count to get to: 50000 curr count: 49114
2021/12/14 09:39:47 Waiting for src bucket item count to get to: 50000 curr count: 49114
2021/12/14 09:39:57 Waiting for dst bucket item count to get to: 50000 curr count: 48253
2021/12/14 09:40:02 Waiting for dst bucket item count to get to: 50000 curr count: 48253
2021/12/14 09:40:02 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/14 09:40:02 Post SET ops. Expected item count: 50000 got 50000
2021/12/14 09:40:02 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 09:40:07 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/14 09:40:12 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/14 09:40:18 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/14 09:40:23 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/14 09:40:28 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/14 09:40:28 App: TestDuplicateTimerFiringEventingRebIn got un-deployed
2021/12/14 09:40:28 No eventing-con process running
2021/12/14 09:40:28 Function: TestDuplicateTimerFiringEventingRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringEventingRebIn deleting in the background"
}
2021/12/14 09:40:30 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:40:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 09:40:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 09:40:36 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:40:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 09:40:46 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/14 09:40:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringEventingRebIn (159.36s)
=== RUN   TestDuplicateTimerFiringEventingRebOut
2021/12/14 09:40:51 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/14 09:40:52 Removing node(s): [] from the cluster
2021/12/14 09:40:57 Rebalance progress: 100
2021/12/14 09:41:02 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/14 09:41:02 App: TestDuplicateTimerFiringEventingRebOut got un-deployed
2021/12/14 09:41:02 No eventing-con process running
2021/12/14 09:41:02 Function: TestDuplicateTimerFiringEventingRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringEventingRebOut not found"
 }
}
2021/12/14 09:41:03 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:41:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 09:41:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 09:41:10 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:41:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 09:41:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:41:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 09:41:20 Deploying app: TestDuplicateTimerFiringEventingRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/14 09:41:21 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringEventingRebOut' in metakv",
  "warnings": null
 }
}
2021/12/14 09:41:26 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get deployed
2021/12/14 09:41:31 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get deployed
2021/12/14 09:41:36 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get deployed
2021/12/14 09:41:41 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get deployed
2021/12/14 09:41:41 App: TestDuplicateTimerFiringEventingRebOut got deployed
2021/12/14 09:41:41 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/14 09:41:41 Starting bucket ops to source bucket
2021/12/14 09:41:47 Rebalance progress: 12.5
2021/12/14 09:41:52 Rebalance progress: 12.5
2021/12/14 09:41:57 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:41:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:41:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 09:41:57 Waiting for src bucket item count to get to: 50000 curr count: 9888
2021/12/14 09:42:02 Waiting for src bucket item count to get to: 50000 curr count: 22917
2021/12/14 09:42:07 Waiting for src bucket item count to get to: 50000 curr count: 22917
2021/12/14 09:42:12 Waiting for src bucket item count to get to: 50000 curr count: 33886
2021/12/14 09:42:17 Waiting for src bucket item count to get to: 50000 curr count: 33886
2021/12/14 09:42:22 Waiting for src bucket item count to get to: 50000 curr count: 33886
2021/12/14 09:42:27 Waiting for src bucket item count to get to: 50000 curr count: 44601
2021/12/14 09:42:32 Waiting for src bucket item count to get to: 50000 curr count: 44601
2021/12/14 09:42:42 Waiting for dst bucket item count to get to: 50000 curr count: 36365
2021/12/14 09:42:47 Waiting for dst bucket item count to get to: 50000 curr count: 36365
2021/12/14 09:42:47 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/14 09:42:47 Post SET ops. Expected item count: 50000 got 50000
2021/12/14 09:42:47 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 09:42:52 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/14 09:42:57 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/14 09:43:02 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/14 09:43:07 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/14 09:43:12 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/14 09:43:17 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/14 09:43:17 App: TestDuplicateTimerFiringEventingRebOut got un-deployed
2021/12/14 09:43:17 No eventing-con process running
2021/12/14 09:43:17 Function: TestDuplicateTimerFiringEventingRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringEventingRebOut deleting in the background"
}
2021/12/14 09:43:19 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:43:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 09:43:29 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/14 09:43:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 09:43:31 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:43:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 09:43:41 Waiting for dst bucket item count to get to: 0 curr count: 31151
2021/12/14 09:43:46 Waiting for dst bucket item count to get to: 0 curr count: 31151
2021/12/14 09:43:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringEventingRebOut (179.29s)
=== RUN   TestDuplicateTimerFiringEventingSwapReb
2021/12/14 09:43:56 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/14 09:43:56 App: TestDuplicateTimerFiringEventingSwapReb got un-deployed
2021/12/14 09:43:56 No eventing-con process running
2021/12/14 09:43:56 Function: TestDuplicateTimerFiringEventingSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringEventingSwapReb not found"
 }
}
2021/12/14 09:43:57 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:44:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 09:44:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 09:44:04 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:44:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 09:44:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:44:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 09:44:14 Deploying app: TestDuplicateTimerFiringEventingSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/14 09:44:15 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringEventingSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/14 09:44:20 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get deployed
2021/12/14 09:44:25 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get deployed
2021/12/14 09:44:30 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get deployed
2021/12/14 09:44:35 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get deployed
2021/12/14 09:44:35 App: TestDuplicateTimerFiringEventingSwapReb got deployed
2021/12/14 09:44:35 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/14 09:44:35 Starting bucket ops to source bucket
2021/12/14 09:44:36 Removing node(s): [] from the cluster
2021/12/14 09:44:41 Rebalance progress: 12.5
2021/12/14 09:44:46 Rebalance progress: 12.5
2021/12/14 09:44:51 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:44:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:44:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 09:44:51 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/14 09:44:51 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/14 09:44:56 Rebalance progress: 12.5
2021/12/14 09:45:01 Rebalance progress: 12.5
2021/12/14 09:45:06 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:45:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:45:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 09:45:07 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/14 09:45:12 Rebalance progress: 12.5
2021/12/14 09:45:17 Rebalance progress: 12.5
2021/12/14 09:45:22 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:45:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:45:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 09:45:22 Waiting for src bucket item count to get to: 50000 curr count: 48404
2021/12/14 09:45:27 Waiting for src bucket item count to get to: 50000 curr count: 48404
2021/12/14 09:45:37 Waiting for dst bucket item count to get to: 50000 curr count: 43459
2021/12/14 09:45:42 Waiting for dst bucket item count to get to: 50000 curr count: 43459
2021/12/14 09:45:42 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/14 09:45:42 Post SET ops. Expected item count: 50000 got 50000
2021/12/14 09:45:42 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 09:45:47 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/14 09:45:52 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/14 09:45:57 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/14 09:46:02 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/14 09:46:07 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/14 09:46:07 App: TestDuplicateTimerFiringEventingSwapReb got un-deployed
2021/12/14 09:46:07 No eventing-con process running
2021/12/14 09:46:07 Function: TestDuplicateTimerFiringEventingSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringEventingSwapReb deleting in the background"
}
2021/12/14 09:46:09 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:46:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 09:46:19 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/14 09:46:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 09:46:21 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:46:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 09:46:31 Waiting for dst bucket item count to get to: 0 curr count: 35881
2021/12/14 09:46:36 Waiting for dst bucket item count to get to: 0 curr count: 35881
2021/12/14 09:46:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringEventingSwapReb (170.09s)
=== RUN   TestDuplicateTimerFiringKVRebIn
2021/12/14 09:46:46 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/14 09:46:46 App: TestDuplicateTimerFiringKVRebIn got un-deployed
2021/12/14 09:46:46 No eventing-con process running
2021/12/14 09:46:46 Function: TestDuplicateTimerFiringKVRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringKVRebIn not found"
 }
}
2021/12/14 09:46:47 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:46:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 09:46:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 09:46:54 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:46:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 09:46:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:47:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 09:47:04 Deploying app: TestDuplicateTimerFiringKVRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/14 09:47:05 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringKVRebIn' in metakv",
  "warnings": null
 }
}
2021/12/14 09:47:10 Waiting for app: TestDuplicateTimerFiringKVRebIn to get deployed
2021/12/14 09:47:15 Waiting for app: TestDuplicateTimerFiringKVRebIn to get deployed
2021/12/14 09:47:20 Waiting for app: TestDuplicateTimerFiringKVRebIn to get deployed
2021/12/14 09:47:25 Waiting for app: TestDuplicateTimerFiringKVRebIn to get deployed
2021/12/14 09:47:25 App: TestDuplicateTimerFiringKVRebIn got deployed
2021/12/14 09:47:25 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/14 09:47:25 Starting bucket ops to source bucket
2021/12/14 09:47:26 Removing node(s): [] from the cluster
2021/12/14 09:47:31 Rebalance progress: 12.5
2021/12/14 09:47:36 Rebalance progress: 12.5
2021/12/14 09:47:41 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:47:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:47:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 09:47:41 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/14 09:47:46 Rebalance progress: 12.5
2021/12/14 09:47:51 Rebalance progress: 12.5
2021/12/14 09:47:56 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:47:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:47:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 09:47:56 Waiting for src bucket item count to get to: 50000 curr count: 28834
2021/12/14 09:48:01 Waiting for src bucket item count to get to: 50000 curr count: 40071
2021/12/14 09:48:06 Waiting for src bucket item count to get to: 50000 curr count: 40071
2021/12/14 09:48:16 Waiting for dst bucket item count to get to: 50000 curr count: 33420
2021/12/14 09:48:21 Waiting for dst bucket item count to get to: 50000 curr count: 33420
2021/12/14 09:48:26 Waiting for dst bucket item count to get to: 50000 curr count: 43107
2021/12/14 09:48:31 Waiting for dst bucket item count to get to: 50000 curr count: 43107
2021/12/14 09:48:31 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/14 09:48:31 Post SET ops. Expected item count: 50000 got 50000
2021/12/14 09:48:31 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 09:48:36 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/14 09:48:41 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/14 09:48:46 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/14 09:48:51 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/14 09:48:56 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/14 09:48:56 App: TestDuplicateTimerFiringKVRebIn got un-deployed
2021/12/14 09:48:57 No eventing-con process running
2021/12/14 09:48:57 Function: TestDuplicateTimerFiringKVRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringKVRebIn deleting in the background"
}
2021/12/14 09:48:58 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:49:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 09:49:08 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/14 09:49:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 09:49:10 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:49:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 09:49:20 Waiting for dst bucket item count to get to: 0 curr count: 11846
2021/12/14 09:49:25 Waiting for dst bucket item count to get to: 0 curr count: 11846
2021/12/14 09:49:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringKVRebIn (169.48s)
=== RUN   TestDuplicateTimerFiringKVRebOut
2021/12/14 09:49:30 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/14 09:49:30 Removing node(s): [] from the cluster
2021/12/14 09:49:36 Rebalance progress: 100
2021/12/14 09:49:41 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/14 09:49:41 App: TestDuplicateTimerFiringKVRebOut got un-deployed
2021/12/14 09:49:41 No eventing-con process running
2021/12/14 09:49:41 Function: TestDuplicateTimerFiringKVRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringKVRebOut not found"
 }
}
2021/12/14 09:49:42 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:49:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 09:49:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 09:49:49 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:49:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 09:49:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:49:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 09:49:59 Deploying app: TestDuplicateTimerFiringKVRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/14 09:50:00 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringKVRebOut' in metakv",
  "warnings": null
 }
}
2021/12/14 09:50:05 Waiting for app: TestDuplicateTimerFiringKVRebOut to get deployed
2021/12/14 09:50:10 Waiting for app: TestDuplicateTimerFiringKVRebOut to get deployed
2021/12/14 09:50:15 Waiting for app: TestDuplicateTimerFiringKVRebOut to get deployed
2021/12/14 09:50:20 Waiting for app: TestDuplicateTimerFiringKVRebOut to get deployed
2021/12/14 09:50:20 App: TestDuplicateTimerFiringKVRebOut got deployed
2021/12/14 09:50:20 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/14 09:50:20 Starting bucket ops to source bucket
2021/12/14 09:50:25 Rebalance progress: 12.5
2021/12/14 09:50:30 Rebalance progress: 12.5
2021/12/14 09:50:35 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:50:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:50:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 09:50:35 Waiting for src bucket item count to get to: 50000 curr count: 12017
2021/12/14 09:50:40 Waiting for src bucket item count to get to: 50000 curr count: 25566
2021/12/14 09:50:45 Waiting for src bucket item count to get to: 50000 curr count: 25566
2021/12/14 09:50:50 Waiting for src bucket item count to get to: 50000 curr count: 35673
2021/12/14 09:50:55 Waiting for src bucket item count to get to: 50000 curr count: 35673
2021/12/14 09:51:00 Waiting for src bucket item count to get to: 50000 curr count: 35673
2021/12/14 09:51:05 Waiting for src bucket item count to get to: 50000 curr count: 47171
2021/12/14 09:51:10 Waiting for src bucket item count to get to: 50000 curr count: 47171
2021/12/14 09:51:20 Waiting for dst bucket item count to get to: 50000 curr count: 37483
2021/12/14 09:51:25 Waiting for dst bucket item count to get to: 50000 curr count: 37483
2021/12/14 09:51:25 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/14 09:51:25 Post SET ops. Expected item count: 50000 got 50000
2021/12/14 09:51:26 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 09:51:31 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/14 09:51:36 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/14 09:51:41 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/14 09:51:46 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/14 09:51:51 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/14 09:51:56 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/14 09:51:56 App: TestDuplicateTimerFiringKVRebOut got un-deployed
2021/12/14 09:51:56 No eventing-con process running
2021/12/14 09:51:56 Function: TestDuplicateTimerFiringKVRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringKVRebOut deleting in the background"
}
2021/12/14 09:51:58 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:52:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 09:52:08 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/14 09:52:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 09:52:09 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:52:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 09:52:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringKVRebOut (169.28s)
=== RUN   TestDuplicateTimerFiringKVSwapReb
2021/12/14 09:52:24 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/14 09:52:24 App: TestDuplicateTimerFiringKVSwapReb got un-deployed
2021/12/14 09:52:24 No eventing-con process running
2021/12/14 09:52:24 Function: TestDuplicateTimerFiringKVSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringKVSwapReb not found"
 }
}
2021/12/14 09:52:26 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:52:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 09:52:31 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 09:52:33 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:52:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 09:52:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:52:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 09:52:43 Deploying app: TestDuplicateTimerFiringKVSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/14 09:52:44 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringKVSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/14 09:52:49 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get deployed
2021/12/14 09:52:54 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get deployed
2021/12/14 09:52:59 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get deployed
2021/12/14 09:53:04 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get deployed
2021/12/14 09:53:04 App: TestDuplicateTimerFiringKVSwapReb got deployed
2021/12/14 09:53:04 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/14 09:53:04 Starting bucket ops to source bucket
2021/12/14 09:53:05 Removing node(s): [] from the cluster
2021/12/14 09:53:10 Rebalance progress: 12.5
2021/12/14 09:53:15 Rebalance progress: 12.5
2021/12/14 09:53:20 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:53:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:53:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 09:53:20 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/14 09:53:20 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/14 09:53:25 Rebalance progress: 12.5
2021/12/14 09:53:30 Rebalance progress: 12.5
2021/12/14 09:53:35 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:53:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:53:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 09:53:35 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/14 09:53:40 Rebalance progress: 12.5
2021/12/14 09:53:45 Rebalance progress: 12.5
2021/12/14 09:53:50 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:53:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:53:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 09:53:50 Waiting for src bucket item count to get to: 50000 curr count: 39324
2021/12/14 09:54:00 Waiting for dst bucket item count to get to: 50000 curr count: 38576
2021/12/14 09:54:06 Waiting for dst bucket item count to get to: 50000 curr count: 38576
2021/12/14 09:54:11 Waiting for dst bucket item count to get to: 50000 curr count: 46545
2021/12/14 09:54:11 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/14 09:54:11 Post SET ops. Expected item count: 50000 got 50000
2021/12/14 09:54:11 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 09:54:16 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/14 09:54:21 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/14 09:54:26 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/14 09:54:31 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/14 09:54:36 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/14 09:54:41 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/14 09:54:41 App: TestDuplicateTimerFiringKVSwapReb got un-deployed
2021/12/14 09:54:41 No eventing-con process running
2021/12/14 09:54:41 Function: TestDuplicateTimerFiringKVSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringKVSwapReb deleting in the background"
}
2021/12/14 09:54:43 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:54:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 09:54:53 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/14 09:54:58 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/14 09:54:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 09:55:00 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:55:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 09:55:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringKVSwapReb (170.10s)
=== RUN   TestKVRebOnUpdateBucketOpOneByOne
2021/12/14 09:55:15 Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 09:55:15 App: bucket_op_on_update got un-deployed
2021/12/14 09:55:15 No eventing-con process running
2021/12/14 09:55:15 Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/14 09:55:16 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:55:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 09:55:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 09:55:23 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:55:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 09:55:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:55:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 09:55:33 Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/14 09:55:34 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/14 09:55:39 Starting bucket ops to source bucket
2021/12/14 09:55:44 Waiting for app: bucket_op_on_update to get deployed
2021/12/14 09:55:49 Waiting for app: bucket_op_on_update to get deployed
2021/12/14 09:55:54 Waiting for app: bucket_op_on_update to get deployed
2021/12/14 09:55:54 App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:55:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:55:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 09:55:54 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/14 09:55:55 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/14 09:56:00 Rebalance progress: 12.5
2021/12/14 09:56:05 Rebalance progress: 12.5
2021/12/14 09:56:10 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:56:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:56:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 09:56:10 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/14 09:56:10 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/14 09:56:15 Rebalance progress: 12.5
2021/12/14 09:56:20 Rebalance progress: 12.5
2021/12/14 09:56:25 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:56:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:56:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 09:56:25 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: kv to the cluster
2021/12/14 09:56:26 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/14 09:56:31 Rebalance progress: 12.5
2021/12/14 09:56:36 Rebalance progress: 12.5
2021/12/14 09:56:41 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:56:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:56:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 09:56:41 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/14 09:56:46 Rebalance progress: 12.5
2021/12/14 09:56:51 Rebalance progress: 12.5
2021/12/14 09:56:56 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:56:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:56:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 09:56:56 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/14 09:57:01 Rebalance progress: 12.5
2021/12/14 09:57:06 Rebalance progress: 12.5
2021/12/14 09:57:11 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:57:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:57:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 09:57:11 clusterSetup.go:253: Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/14 09:57:16 Rebalance progress: 12.5
2021/12/14 09:57:21 Rebalance progress: 12.5
2021/12/14 09:57:26 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:57:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:57:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 09:57:26 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 09:57:31 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 09:57:36 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 09:57:42 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 09:57:47 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 09:57:52 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 09:57:57 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 09:57:57 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/14 09:57:57 eventingSetup.go:46: No eventing-con process running
2021/12/14 09:57:57 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/14 09:57:58 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:58:03 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 09:58:08 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 10692
2021/12/14 09:58:08 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 09:58:10 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:58:15 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 09:58:20 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3368
2021/12/14 09:58:25 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3368
2021/12/14 09:58:25 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebOnUpdateBucketOpOneByOne (200.72s)
=== RUN   TestKVForceFailover
2021/12/14 09:58:40 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 09:58:40 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/14 09:58:40 eventingSetup.go:46: No eventing-con process running
2021/12/14 09:58:40 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/14 09:58:42 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:58:47 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 09:58:47 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 09:58:49 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:58:54 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 09:58:54 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 09:58:54 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/14 09:58:54 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/14 09:58:59 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:58:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 09:58:59 Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/14 09:59:00 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/14 09:59:00 Starting bucket ops to source bucket
2021/12/14 09:59:05 Waiting for app: bucket_op_on_update to get deployed
2021/12/14 09:59:10 Waiting for app: bucket_op_on_update to get deployed
2021/12/14 09:59:15 Waiting for app: bucket_op_on_update to get deployed
2021/12/14 09:59:20 Waiting for app: bucket_op_on_update to get deployed
2021/12/14 09:59:20 App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:59:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:59:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 09:59:20 clusterSetup.go:229: Failing over: [127.0.0.1:19001]
2021/12/14 09:59:40 clusterSetup.go:237: Kicking off failover recovery, type: full
2021/12/14 09:59:40 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/14 09:59:46 Rebalance progress: 12.5
2021/12/14 09:59:51 Rebalance progress: 12.5
2021/12/14 09:59:56 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:59:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 09:59:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 09:59:56 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002] from the cluster
2021/12/14 10:00:01 Rebalance progress: 12.5
2021/12/14 10:00:06 Rebalance progress: 12.5
2021/12/14 10:00:11 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 10:00:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 10:00:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 10:00:11 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 10:00:16 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 10:00:21 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 10:00:26 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 10:00:31 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 10:00:36 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 10:00:41 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 10:00:41 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/14 10:00:41 eventingSetup.go:46: No eventing-con process running
2021/12/14 10:00:41 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/14 10:00:43 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 10:00:48 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 10:00:53 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7038
2021/12/14 10:00:58 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7038
2021/12/14 10:00:58 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 10:01:00 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 10:01:05 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 10:01:05 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVForceFailover (159.50s)
=== RUN   TestKVHardFailoverRecovery
2021/12/14 10:01:15 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 10:01:15 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/14 10:01:15 eventingSetup.go:46: No eventing-con process running
2021/12/14 10:01:15 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/14 10:01:17 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 10:01:22 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 10:01:22 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 10:01:23 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 10:01:28 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 10:01:28 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 10:01:33 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 10:01:33 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/14 10:01:34 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/14 10:01:39 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/14 10:01:44 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/14 10:01:49 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/14 10:01:54 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/14 10:01:55 clusterSetup.go:354: App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 10:01:55 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 10:01:55 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 10:01:55 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/14 10:01:55 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/14 10:02:00 Rebalance progress: 12.5
2021/12/14 10:02:05 Rebalance progress: 12.5
2021/12/14 10:02:10 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 10:02:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 10:02:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 10:02:10 clusterSetup.go:229: Failing over: [127.0.0.1:19001]
2021/12/14 10:03:10 clusterSetup.go:237: Kicking off failover recovery, type: full
2021/12/14 10:03:15 Rebalance progress: 12.5
2021/12/14 10:03:20 Rebalance progress: 12.5
2021/12/14 10:03:25 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 10:03:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 10:03:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 10:03:25 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/14 10:03:30 Rebalance progress: 12.5
2021/12/14 10:03:35 Rebalance progress: 12.5
2021/12/14 10:03:40 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 10:03:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 10:03:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 10:03:40 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 10:03:45 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 10:03:50 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 10:03:55 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 10:04:00 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 10:04:05 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 10:04:05 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/14 10:04:05 eventingSetup.go:46: No eventing-con process running
2021/12/14 10:04:05 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/14 10:04:07 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 10:04:12 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 10:04:17 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12045
2021/12/14 10:04:17 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 10:04:19 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 10:04:24 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 10:04:29 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12045
2021/12/14 10:04:29 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVHardFailoverRecovery (204.41s)
=== RUN   TestKVRebOnUpdateBucketOpAllAtOnce
2021/12/14 10:04:39 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 10:04:39 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/14 10:04:39 eventingSetup.go:46: No eventing-con process running
2021/12/14 10:04:39 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/14 10:04:41 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 10:04:46 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 10:04:46 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 10:04:48 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 10:04:53 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 10:04:53 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 10:04:58 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 10:04:58 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/14 10:04:59 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/14 10:05:04 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/14 10:05:09 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/14 10:05:14 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/14 10:05:19 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/14 10:05:19 clusterSetup.go:354: App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 10:05:19 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 10:05:19 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 10:05:19 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/14 10:05:20 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/14 10:05:20 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: kv to the cluster
2021/12/14 10:05:21 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/14 10:05:26 Rebalance progress: 12.5
2021/12/14 10:05:31 Rebalance progress: 12.5
2021/12/14 10:05:36 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 10:05:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 10:05:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 10:05:36 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002 127.0.0.1:19003] from the cluster
2021/12/14 10:05:41 Rebalance progress: 12.5
2021/12/14 10:05:46 Rebalance progress: 12.5
2021/12/14 10:05:51 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 10:05:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 10:05:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 10:05:51 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 10:05:56 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 10:06:01 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 10:06:06 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 10:06:11 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 10:06:16 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 10:06:21 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 10:06:21 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/14 10:06:21 eventingSetup.go:46: No eventing-con process running
2021/12/14 10:06:21 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/14 10:06:23 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 10:06:28 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 10:06:33 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4691
2021/12/14 10:06:38 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4691
2021/12/14 10:06:38 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 10:06:40 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 10:06:45 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 10:06:50 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 379
2021/12/14 10:06:55 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 379
2021/12/14 10:06:55 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebOnUpdateBucketOpAllAtOnce (145.84s)
=== RUN   TestKVSwapRebOnUpdateBucketOp
2021/12/14 10:07:05 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 10:07:05 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/14 10:07:05 eventingSetup.go:46: No eventing-con process running
2021/12/14 10:07:05 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/14 10:07:07 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 10:07:12 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 10:07:12 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 10:07:14 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 10:07:19 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 10:07:19 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 10:07:19 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 10:07:19 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/14 10:07:20 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/14 10:07:25 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/14 10:07:30 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/14 10:07:35 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/14 10:07:40 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/14 10:07:40 clusterSetup.go:354: App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 10:07:40 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 10:07:40 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 10:07:40 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/14 10:07:40 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/14 10:07:45 Rebalance progress: 12.5
2021/12/14 10:07:50 Rebalance progress: 12.5
2021/12/14 10:07:55 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 10:07:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 10:07:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 10:07:55 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/14 10:07:55 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/14 10:08:00 Rebalance progress: 12.5
2021/12/14 10:08:05 Rebalance progress: 12.5
2021/12/14 10:08:10 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 10:08:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 10:08:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 10:08:10 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/14 10:08:16 Rebalance progress: 12.5
2021/12/14 10:08:21 Rebalance progress: 12.5
2021/12/14 10:08:26 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 10:08:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 10:08:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 10:08:26 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 10:08:31 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 10:08:36 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 10:08:41 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 10:08:46 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 10:08:51 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 10:08:56 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 10:08:56 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/14 10:08:56 eventingSetup.go:46: No eventing-con process running
2021/12/14 10:08:56 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/14 10:08:58 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 10:09:03 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 10:09:08 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6053
2021/12/14 10:09:08 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 10:09:09 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 10:09:14 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 10:09:14 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 10:09:19 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 10:09:19 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/14 10:09:19 eventingSetup.go:46: No eventing-con process running
2021/12/14 10:09:19 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/14 10:09:21 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 10:09:26 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 10:09:26 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 10:09:28 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 10:09:33 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 10:09:33 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVSwapRebOnUpdateBucketOp (158.04s)
=== RUN   TestKVRebalanceWithMultipleHandlers
2021/12/14 10:09:43 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 10:09:43 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/14 10:09:43 eventingSetup.go:46: No eventing-con process running
2021/12/14 10:09:43 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/14 10:09:45 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 10:09:50 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 10:09:50 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 10:09:52 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 10:09:57 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 10:09:57 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 10:10:02 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/14 10:10:02 clusterSetup.go:405: App: sys_test_bucket_op got un-deployed
2021/12/14 10:10:02 eventingSetup.go:46: No eventing-con process running
2021/12/14 10:10:02 eventingSetup.go:418: Function: sys_test_bucket_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: sys_test_bucket_op not found"
 }
}
2021/12/14 10:10:03 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 10:10:08 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 10:10:08 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 10:10:10 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 10:10:15 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 10:10:15 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 10:10:15 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 10:10:15 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/14 10:10:16 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 10:10:16 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 10:10:16 eventingSetup.go:125: Deploying app: sys_test_bucket_op. Item count src bucket: 0 dst bucket: 0
2021/12/14 10:10:17 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'sys_test_bucket_op' in metakv",
  "warnings": null
 }
}
2021/12/14 10:10:22 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/14 10:10:27 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/14 10:10:32 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/14 10:10:38 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/14 10:10:38 clusterSetup.go:354: App: bucket_op_on_update got deployed
2021/12/14 10:10:43 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/14 10:10:48 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/14 10:10:53 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/14 10:10:53 clusterSetup.go:354: App: sys_test_bucket_op got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 10:10:53 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 10:10:53 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 10:10:53 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/14 10:10:53 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/14 10:10:58 Rebalance progress: 12.5
2021/12/14 10:11:03 Rebalance progress: 12.5
2021/12/14 10:11:08 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 10:11:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 10:11:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 10:11:08 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/14 10:11:09 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/14 10:11:14 Rebalance progress: 12.5
2021/12/14 10:11:19 Rebalance progress: 12.5
2021/12/14 10:11:24 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 10:11:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 10:11:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 10:11:24 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: kv to the cluster
2021/12/14 10:11:24 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/14 10:11:29 Rebalance progress: 12.5
2021/12/14 10:11:34 Rebalance progress: 12.5
2021/12/14 10:11:39 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 10:11:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 10:11:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 10:11:39 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/14 10:11:44 Rebalance progress: 12.5
2021/12/14 10:11:49 Rebalance progress: 12.5
2021/12/14 10:11:54 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 10:11:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 10:11:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 10:11:54 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/14 10:11:59 Rebalance progress: 12.5
2021/12/14 10:12:04 Rebalance progress: 12.5
2021/12/14 10:12:09 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 10:12:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 10:12:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 10:12:10 clusterSetup.go:253: Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/14 10:12:15 Rebalance progress: 12.5
2021/12/14 10:12:20 Rebalance progress: 12.5
2021/12/14 10:12:25 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 10:12:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 10:12:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 10:12:25 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 10:12:30 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 10:12:35 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 10:12:40 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 10:12:45 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 10:12:50 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 10:12:55 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 10:12:55 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/14 10:12:55 eventingSetup.go:46: No eventing-con process running
2021/12/14 10:12:55 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/14 10:13:00 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 10:13:05 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 10:13:05 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 10:13:07 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 10:13:12 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 10:13:17 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12206
2021/12/14 10:13:17 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 10:13:17 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 10:13:22 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/14 10:13:27 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/14 10:13:32 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/14 10:13:37 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/14 10:13:42 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/14 10:13:47 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/14 10:13:47 clusterSetup.go:405: App: sys_test_bucket_op got un-deployed
2021/12/14 10:13:47 eventingSetup.go:46: No eventing-con process running
2021/12/14 10:13:47 eventingSetup.go:418: Function: sys_test_bucket_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: sys_test_bucket_op deleting in the background"
}
2021/12/14 10:13:49 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 10:13:54 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 10:13:54 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 10:13:56 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 10:14:01 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 10:14:01 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebalanceWithMultipleHandlers (267.94s)
=== RUN   TestKVSwapRebalanceWithMultipleHandlers
2021/12/14 10:14:11 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 10:14:11 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/14 10:14:11 eventingSetup.go:46: No eventing-con process running
2021/12/14 10:14:11 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/14 10:14:13 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 10:14:18 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 10:14:18 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 10:14:20 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 10:14:25 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 10:14:25 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 10:14:30 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/14 10:14:30 clusterSetup.go:405: App: sys_test_bucket_op got un-deployed
2021/12/14 10:14:30 eventingSetup.go:46: No eventing-con process running
2021/12/14 10:14:30 eventingSetup.go:418: Function: sys_test_bucket_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: sys_test_bucket_op not found"
 }
}
2021/12/14 10:14:31 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 10:14:36 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 10:14:36 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 10:14:38 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 10:14:43 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 10:14:43 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 10:14:43 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 10:14:43 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/14 10:14:44 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 10:14:44 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 10:14:44 eventingSetup.go:125: Deploying app: sys_test_bucket_op. Item count src bucket: 0 dst bucket: 0
2021/12/14 10:14:45 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'sys_test_bucket_op' in metakv",
  "warnings": null
 }
}
2021/12/14 10:14:50 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/14 10:14:55 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/14 10:15:00 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/14 10:15:05 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/14 10:15:05 clusterSetup.go:354: App: bucket_op_on_update got deployed
2021/12/14 10:15:10 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/14 10:15:15 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/14 10:15:20 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/14 10:15:21 clusterSetup.go:354: App: sys_test_bucket_op got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 10:15:21 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 10:15:21 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 10:15:21 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/14 10:15:21 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/14 10:15:26 Rebalance progress: 12.5
2021/12/14 10:15:31 Rebalance progress: 12.5
2021/12/14 10:15:36 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 10:15:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 10:15:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 10:15:36 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/14 10:15:36 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/14 10:15:41 Rebalance progress: 12.5
2021/12/14 10:15:46 Rebalance progress: 12.5
2021/12/14 10:15:51 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 10:15:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 10:15:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 10:15:51 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/14 10:15:56 Rebalance progress: 12.5
2021/12/14 10:16:01 Rebalance progress: 12.5
2021/12/14 10:16:06 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 10:16:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 10:16:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 10:16:07 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 10:16:12 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 10:16:17 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 10:16:22 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 10:16:27 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 10:16:32 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 10:16:37 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 10:16:37 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/14 10:16:37 eventingSetup.go:46: No eventing-con process running
2021/12/14 10:16:37 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/14 10:16:40 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 10:16:45 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 10:16:50 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 136
2021/12/14 10:16:55 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 136
2021/12/14 10:16:55 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 10:16:57 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 10:17:02 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 10:17:07 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7596
2021/12/14 10:17:07 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 10:17:07 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 10:17:12 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/14 10:17:17 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/14 10:17:22 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/14 10:17:27 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/14 10:17:33 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/14 10:17:38 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/14 10:17:38 clusterSetup.go:405: App: sys_test_bucket_op got un-deployed
2021/12/14 10:17:38 eventingSetup.go:46: No eventing-con process running
2021/12/14 10:17:38 eventingSetup.go:418: Function: sys_test_bucket_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: sys_test_bucket_op deleting in the background"
}
2021/12/14 10:17:39 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 10:17:44 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 10:17:44 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 10:17:46 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 10:17:51 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 10:17:51 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVSwapRebalanceWithMultipleHandlers (230.24s)
=== RUN   TestKVRebStopStartKVOpsOnUpdateBucketOpOneByOne
2021/12/14 10:18:01 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 10:18:01 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/14 10:18:01 eventingSetup.go:46: No eventing-con process running
2021/12/14 10:18:01 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/14 10:18:03 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 10:18:08 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 10:18:08 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 10:18:10 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 10:18:15 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 10:18:15 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 10:18:20 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 10:18:20 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/14 10:18:21 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/14 10:18:26 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/14 10:18:31 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/14 10:18:36 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/14 10:18:41 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/14 10:18:41 clusterSetup.go:354: App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 10:18:41 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 10:18:41 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 10:18:41 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/14 10:18:42 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/14 10:19:02 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 10:19:02 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 10:19:02 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 10:19:12 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 10:19:12 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 10:19:12 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/14 10:19:17 Rebalance progress: 12.5
2021/12/14 10:19:22 Rebalance progress: 12.5
2021/12/14 10:19:27 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 10:19:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 10:19:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 10:19:27 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/14 10:19:28 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/14 10:19:33 Rebalance progress: 12.5
2021/12/14 10:19:38 Rebalance progress: 12.5
2021/12/14 10:19:43 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 10:19:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 10:19:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 10:19:43 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/14 10:19:48 Rebalance progress: 12.5
2021/12/14 10:19:53 Rebalance progress: 12.5
2021/12/14 10:19:58 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 10:19:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 10:19:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 10:19:58 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 10:20:03 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 10:20:08 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 10:20:13 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 10:20:18 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 10:20:23 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 10:20:28 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 10:20:28 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/14 10:20:28 eventingSetup.go:46: No eventing-con process running
2021/12/14 10:20:28 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/14 10:20:30 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 10:20:35 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 10:20:40 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2709
2021/12/14 10:20:45 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2709
2021/12/14 10:20:45 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 10:20:47 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 10:20:52 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 10:20:57 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 9202
2021/12/14 10:20:57 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebStopStartKVOpsOnUpdateBucketOpOneByOne (185.86s)
=== RUN   TestKVFailoverOnUpdateBucketOp
2021/12/14 10:21:07 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 10:21:07 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/14 10:21:07 eventingSetup.go:46: No eventing-con process running
2021/12/14 10:21:07 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/14 10:21:09 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 10:21:14 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 10:21:14 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 10:21:16 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 10:21:21 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 10:21:21 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 10:21:26 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 10:21:26 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/14 10:21:27 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/14 10:21:32 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/14 10:21:37 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/14 10:21:42 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/14 10:21:47 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/14 10:21:47 clusterSetup.go:354: App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 10:21:47 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 10:21:47 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 10:21:47 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/14 10:21:47 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/14 10:21:52 Rebalance progress: 12.5
2021/12/14 10:21:57 Rebalance progress: 12.5
2021/12/14 10:22:02 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 10:22:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 10:22:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 10:22:02 clusterSetup.go:229: Failing over: [127.0.0.1:19001]
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 10:22:12 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 10:22:12 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 10:22:12 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/14 10:22:17 Rebalance progress: 12.5
2021/12/14 10:22:22 Rebalance progress: 12.5
2021/12/14 10:22:27 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 10:22:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 10:22:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 10:22:27 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/14 10:22:28 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/14 10:22:33 Rebalance progress: 12.5
2021/12/14 10:22:38 Rebalance progress: 12.5
2021/12/14 10:22:43 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 10:22:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 10:22:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 10:22:43 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/14 10:22:48 Rebalance progress: 12.5
2021/12/14 10:22:53 Rebalance progress: 12.5
2021/12/14 10:22:58 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 10:22:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 10:22:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 10:22:58 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 10:23:03 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 10:23:08 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 10:23:13 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 10:23:18 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 10:23:23 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 10:23:28 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 10:23:28 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/14 10:23:28 eventingSetup.go:46: No eventing-con process running
2021/12/14 10:23:29 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/14 10:23:30 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 10:23:35 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 10:23:40 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3028
2021/12/14 10:23:45 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3028
2021/12/14 10:23:45 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 10:23:47 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 10:23:52 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 10:23:57 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 8621
2021/12/14 10:23:57 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVFailoverOnUpdateBucketOp (180.01s)
=== RUN   TestBootstrapAfterKVHardFailover
2021/12/14 10:24:02 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/14 10:24:02 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/14 10:24:07 Rebalance progress: 100
2021/12/14 10:24:07 Failing over: [127.0.0.1:19001]
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 10:24:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 10:24:37 Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/14 10:24:39 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/14 10:24:44 Removing node(s): [] from the cluster
2021/12/14 10:24:44 Starting bucket ops to source bucket
2021/12/14 10:24:49 Rebalance failed. See logs for detailed reason. You can try again.
2021/12/14 10:24:54 Waiting for app: bucket_op_on_update to get deployed
2021/12/14 10:24:59 Waiting for app: bucket_op_on_update to get deployed
2021/12/14 10:24:59 App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 10:24:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 10:24:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 10:24:59 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 10:25:04 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 10:25:09 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 10:25:14 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 10:25:19 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 10:25:24 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 10:25:29 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 10:25:29 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/14 10:25:29 eventingSetup.go:46: No eventing-con process running
2021/12/14 10:25:29 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/14 10:25:31 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 10:25:36 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 10:25:41 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1162
2021/12/14 10:25:46 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1162
2021/12/14 10:25:46 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 10:25:48 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 10:25:53 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 10:25:58 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1504
2021/12/14 10:25:58 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestBootstrapAfterKVHardFailover (120.49s)
PASS
ok  	github.com/couchbase/eventing/tests/functional_tests	12083.568s

Rebalance tests took 3 hours, 21 minutes and 26 seconds.

Testing Succeeded


Version: versions-14.12.2021-02.46.cfg
Build Log: make-14.12.2021-02.46.log
Server Log: logs-14.12.2021-02.46.tar.gz

Finished