Building

Started building at 2021/12/10 01:52:11
Calculating base
Updating mirror
Basing run on 7.1.0-1883 cb38490c9
Updating tree for run 10.12.2021-01.52
gocb is at 635fa6c, changes since last good build: none
libcouchbase is at f6822fc, changes since last good build: none
indexing is at b908442, changes since last good build: 
 b908442a Merge remote-tracking branch 'couchbase/unstable' into HEAD
 cea82180 MB-49855: Add system events for indexes scheduled for background creation
 71150787 MB-49926 : Skip calling ServerGroups API in CE
 9cc3bff4 MB-49501 Part 5 (7.1.0 1861): Log long heartbeats; reduce dump frequency
kv_engine is at 77fe9b6, changes since last good build: 
 77fe9b629 MB-49349: [Eph] Only update purge seqno for time based purges
 60704d00f MB-48904: Expire secondary items from primary timeToExpireFrom
 e97a6fe99 MB-49802: Add extra uid checks to doCollectionsMetadataChecks
 5944dc0a8 MB-49461: Allow MagmaKVStore to return a file handle even if DiskFull
ns_server is at 068a047, changes since last good build: 
 068a04757 MB-49765 Remove no longer used query_vbucket_states
couchstore is at 5aa7c4c, changes since last good build: none
couchdb is at fca2e6b, changes since last good build: none
testrunner is at 6975b4e, changes since last good build: 
 6975b4ef8 fix typo init type.
 f1b9beed2 Adding configurable timeout for compression to happen
 b5797e177 add Azure vms dispatcher to testrunner.
Switching eventing to unstable
eventing is at 9fce322, changes since last good build: none
Switching eventing-ee to unstable
eventing-ee is at de9f5f4, changes since last good build: none
Pulling in uncommitted change 160397 at refs/changes/97/160397/3
Total 23 (delta 15), reused 17 (delta 15)
[unstable 017417e2] MB-100 : Do not fail n1ql timeout test if we are catching timeout exceptions successfully
 Author: abhijpes 
 Date: Mon Aug 30 08:24:38 2021 +0530
 1 file changed, 2 insertions(+), 2 deletions(-)
Pulling in uncommitted change 163449 at refs/changes/49/163449/1
Total 48 (delta 27), reused 39 (delta 27)
[unstable e9cbdb96] MB-100: collect data when test fails
 Author: AnkitPrabhu 
 Date: Tue Oct 12 14:02:31 2021 +0530
 1 file changed, 2 insertions(+)
Pulling in uncommitted change 165976 at refs/changes/76/165976/17
Total 4 (delta 3), reused 4 (delta 3)
[unstable 9c201ed3] MB-48538: Add new audit ids for different events
 Author: AnkitPrabhu 
 Date: Tue Nov 23 14:29:10 2021 +0530
 1 file changed, 337 insertions(+), 61 deletions(-)
Pulling in uncommitted change 166697 at refs/changes/97/166697/4
Total 10 (delta 9), reused 10 (delta 9)
[unstable a27154e7] MB-49547: Check bucket instance to get kv node for dcp request
 Author: AnkitPrabhu 
 Date: Thu Dec 2 12:22:17 2021 +0530
 6 files changed, 38 insertions(+), 109 deletions(-)
Pulling in uncommitted change 166698 at refs/changes/98/166698/3
Total 4 (delta 3), reused 4 (delta 3)
[unstable 7d90a1f2] MB-48334 : Fail rebalance unless all functions deployed on current encryption level
 Author: abhijpes 
 Date: Wed Dec 1 14:51:29 2021 +0530
 1 file changed, 9 insertions(+)
Pulling in uncommitted change 166812 at refs/changes/12/166812/5
Total 24 (delta 20), reused 24 (delta 20)
[unstable 7d8f6943] MB-49547: Refresh bucket info from streaming endpoint instance
 Author: AnkitPrabhu 
 Date: Thu Dec 2 13:55:55 2021 +0530
 7 files changed, 147 insertions(+), 235 deletions(-)
Pulling in uncommitted change 166999 at refs/changes/99/166999/5
Total 56 (delta 47), reused 56 (delta 47)
[unstable 519493c3] MB-46304: Add functionality to disable eventing exposed javascript features
 Author: AnkitPrabhu 
 Date: Mon Dec 6 13:26:12 2021 +0530
 20 files changed, 217 insertions(+), 30 deletions(-)
Pulling in uncommitted change 167196 at refs/changes/96/167196/6
Total 67 (delta 57), reused 67 (delta 57)
[unstable ffa31b09] MB-49720: Store failover log in checkpoint blob
 Author: AnkitPrabhu 
 Date: Wed Dec 8 15:16:22 2021 +0530
 5 files changed, 147 insertions(+), 72 deletions(-)
Pulling in uncommitted change 167183 at refs/changes/83/167183/2
Total 5 (delta 2), reused 4 (delta 2)
[unstable e724f94] MB-46304: Throw runtime exception when curl feature is disabled
 Author: AnkitPrabhu 
 Date: Wed Dec 8 10:39:19 2021 +0530
 1 file changed, 9 insertions(+), 3 deletions(-)
Building cmakefiles and deps
Building main product

Testing

Started testing at 2021/12/10 02:08:28
Testing mode: rebalance,n1ql,functional
Starting server
Starting server: attempt 1

Functional tests

2021/12/10 02:09:57 Started functional
Post "http://127.0.0.1:9000/pools/default": read tcp 127.0.0.1:50032->127.0.0.1:9000: read: connection reset by peer
Quota setup Post "http://127.0.0.1:9000/pools/default": read tcp 127.0.0.1:50032->127.0.0.1:9000: read: connection reset by peer
2021/12/10 02:10:06 Adding node: https://127.0.0.1:19001 with role: kv,index,n1ql to the cluster
2021/12/10 02:10:12 Error in adding nodes : , response: {"otpNode":"n_1@127.0.0.1"}
2021/12/10 02:10:12 Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/10 02:10:18 Removing node(s): [] from the cluster
2021/12/10 02:10:23 Rebalance progress: 0
2021/12/10 02:10:28 Rebalance progress: 0
2021/12/10 02:10:33 Rebalance progress: 4.756221064814815
2021/12/10 02:10:38 Rebalance progress: 9.25925925925926
2021/12/10 02:10:43 Rebalance progress: 12.93041087962963
2021/12/10 02:10:48 Rebalance progress: 17.61429398148148
2021/12/10 02:10:53 Rebalance progress: 20.97800925925926
2021/12/10 02:10:58 Rebalance progress: 25.66189236111111
2021/12/10 02:11:03 Rebalance progress: 100
2021/12/10 02:11:06 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/10 02:11:21 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/10 02:11:36 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/10 02:11:51 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/10 02:12:06 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/10 02:12:21 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/10 02:12:36 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/10 02:12:51 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/10 02:13:06 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/10 02:13:21 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/10 02:13:36 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/10 02:13:51 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
=== RUN   TestAdvancedGetOps
2021/12/10 02:14:08 Deploying app: TestAdvancedGetOps. Item count src bucket: 0 dst bucket: 0
2021/12/10 02:14:09 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedGetOps' in metakv",
  "warnings": null
 }
}
2021/12/10 02:14:14 Waiting for app: TestAdvancedGetOps to get deployed
2021/12/10 02:14:20 Waiting for app: TestAdvancedGetOps to get deployed
2021/12/10 02:14:25 Waiting for app: TestAdvancedGetOps to get deployed
2021/12/10 02:14:30 Waiting for app: TestAdvancedGetOps to get deployed
2021/12/10 02:14:30 App: TestAdvancedGetOps got deployed
2021/12/10 02:14:30 Starting bucket ops to source bucket
2021/12/10 02:14:35 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/10 02:14:40 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/10 02:14:45 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/10 02:14:45 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/10 02:14:45 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 02:14:45 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/10 02:14:45 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 02:14:50 Waiting for app: TestAdvancedGetOps to get un-deployed
2021/12/10 02:14:55 Waiting for app: TestAdvancedGetOps to get un-deployed
2021/12/10 02:15:00 Waiting for app: TestAdvancedGetOps to get un-deployed
2021/12/10 02:15:05 Waiting for app: TestAdvancedGetOps to get un-deployed
2021/12/10 02:15:05 App: TestAdvancedGetOps got un-deployed
2021/12/10 02:15:05 No eventing-con process running
2021/12/10 02:15:05 Function: TestAdvancedGetOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedGetOps deleting in the background"
}
2021/12/10 02:15:07 Context: Bucket flush, response code: 200 dump: 
2021/12/10 02:15:17 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/10 02:15:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 02:15:18 Context: Bucket flush, response code: 200 dump: 
2021/12/10 02:15:28 Waiting for dst bucket item count to get to: 0 curr count: 79
2021/12/10 02:15:33 Waiting for dst bucket item count to get to: 0 curr count: 29
2021/12/10 02:15:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedGetOps (85.07s)
=== RUN   TestAdvancedInsertOps
2021/12/10 02:15:33 Deploying app: TestAdvancedInsertOps. Item count src bucket: 0 dst bucket: 0
2021/12/10 02:15:34 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedInsertOps' in metakv",
  "warnings": null
 }
}
2021/12/10 02:15:39 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/10 02:15:44 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/10 02:15:49 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/10 02:15:54 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/10 02:15:54 App: TestAdvancedInsertOps got deployed
2021/12/10 02:15:54 Starting bucket ops to source bucket
2021/12/10 02:16:00 Waiting for dst bucket item count to get to: 1024 curr count: 0
2021/12/10 02:16:05 Waiting for dst bucket item count to get to: 1024 curr count: 512
2021/12/10 02:16:05 src & dst bucket item count matched up. src bucket count: 1024 dst bucket count: 1024
2021/12/10 02:16:05 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 02:16:05 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/10 02:16:05 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 02:16:10 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/10 02:16:16 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/10 02:16:21 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/10 02:16:26 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/10 02:16:26 App: TestAdvancedInsertOps got un-deployed
2021/12/10 02:16:26 No eventing-con process running
2021/12/10 02:16:26 Function: TestAdvancedInsertOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedInsertOps deleting in the background"
}
2021/12/10 02:16:27 Context: Bucket flush, response code: 200 dump: 
2021/12/10 02:16:37 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/10 02:16:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 02:16:39 Context: Bucket flush, response code: 200 dump: 
2021/12/10 02:16:49 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/10 02:16:54 Waiting for dst bucket item count to get to: 0 curr count: 512
2021/12/10 02:16:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 02:16:54 Testing insert operation on source bucket
2021/12/10 02:16:54 Deploying app: TestAdvancedInsertOps. Item count src bucket: 0 dst bucket: 0
2021/12/10 02:16:55 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedInsertOps' in metakv",
  "warnings": null
 }
}
2021/12/10 02:17:00 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/10 02:17:05 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/10 02:17:10 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/10 02:17:15 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/10 02:17:15 App: TestAdvancedInsertOps got deployed
2021/12/10 02:17:15 Starting bucket ops to source bucket
2021/12/10 02:17:21 Waiting for dst bucket item count to get to: 2048 curr count: 0
2021/12/10 02:17:21 src & dst bucket item count matched up. src bucket count: 2048 dst bucket count: 2048
2021/12/10 02:17:21 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 02:17:21 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/10 02:17:21 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 02:17:26 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/10 02:17:31 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/10 02:17:36 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/10 02:17:41 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/10 02:17:41 App: TestAdvancedInsertOps got un-deployed
2021/12/10 02:17:41 No eventing-con process running
2021/12/10 02:17:41 Function: TestAdvancedInsertOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedInsertOps deleting in the background"
}
2021/12/10 02:17:43 Context: Bucket flush, response code: 200 dump: 
2021/12/10 02:17:53 Waiting for dst bucket item count to get to: 0 curr count: 2048
2021/12/10 02:17:58 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/10 02:17:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 02:18:00 Context: Bucket flush, response code: 200 dump: 
2021/12/10 02:18:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedInsertOps (151.48s)
=== RUN   TestAdvancedUpsertOps
2021/12/10 02:18:05 Starting bucket ops to source bucket
2021/12/10 02:18:05 Deploying app: TestAdvancedUpsertOps. Item count src bucket: 0 dst bucket: 0
2021/12/10 02:18:07 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedUpsertOps' in metakv",
  "warnings": null
 }
}
2021/12/10 02:18:12 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/10 02:18:17 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/10 02:18:22 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/10 02:18:27 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/10 02:18:27 App: TestAdvancedUpsertOps got deployed
2021/12/10 02:18:27 Starting bucket ops to source bucket
2021/12/10 02:18:33 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/10 02:18:38 Waiting for dst bucket item count to get to: 0 curr count: 243
2021/12/10 02:18:43 Waiting for dst bucket item count to get to: 0 curr count: 243
2021/12/10 02:18:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 02:18:43 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 02:18:43 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/10 02:18:43 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 02:18:48 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/10 02:18:53 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/10 02:18:58 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/10 02:19:03 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/10 02:19:03 App: TestAdvancedUpsertOps got un-deployed
2021/12/10 02:19:03 No eventing-con process running
2021/12/10 02:19:03 Function: TestAdvancedUpsertOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedUpsertOps deleting in the background"
}
2021/12/10 02:19:04 Context: Bucket flush, response code: 200 dump: 
2021/12/10 02:19:14 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/10 02:19:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 02:19:16 Context: Bucket flush, response code: 200 dump: 
2021/12/10 02:19:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 02:19:21 Testing upsert operation on source bucket
2021/12/10 02:19:21 Deploying app: TestAdvancedUpsertOps. Item count src bucket: 0 dst bucket: 0
2021/12/10 02:19:23 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedUpsertOps' in metakv",
  "warnings": null
 }
}
2021/12/10 02:19:28 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/10 02:19:33 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/10 02:19:38 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/10 02:19:43 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/10 02:19:43 App: TestAdvancedUpsertOps got deployed
2021/12/10 02:19:43 Starting bucket ops to source bucket
2021/12/10 02:19:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 02:19:44 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 02:19:44 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/10 02:19:44 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 02:19:49 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/10 02:19:54 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/10 02:19:59 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/10 02:20:04 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/10 02:20:04 App: TestAdvancedUpsertOps got un-deployed
2021/12/10 02:20:04 No eventing-con process running
2021/12/10 02:20:04 Function: TestAdvancedUpsertOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedUpsertOps deleting in the background"
}
2021/12/10 02:20:05 Context: Bucket flush, response code: 200 dump: 
2021/12/10 02:20:10 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 02:20:13 Context: Bucket flush, response code: 200 dump: 
2021/12/10 02:20:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedUpsertOps (132.88s)
=== RUN   TestAdvancedReplaceOps
2021/12/10 02:20:18 Starting bucket ops to source bucket
2021/12/10 02:20:19 Deploying app: TestAdvancedReplaceOps. Item count src bucket: 0 dst bucket: 0
2021/12/10 02:20:20 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedReplaceOps' in metakv",
  "warnings": null
 }
}
2021/12/10 02:20:25 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/10 02:20:30 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/10 02:20:35 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/10 02:20:40 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/10 02:20:40 App: TestAdvancedReplaceOps got deployed
2021/12/10 02:20:40 Starting bucket ops to source bucket
2021/12/10 02:20:46 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/10 02:20:51 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/10 02:20:56 Waiting for dst bucket item count to get to: 0 curr count: 512
2021/12/10 02:20:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 02:20:56 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 02:20:56 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/10 02:20:56 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 02:21:01 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/10 02:21:06 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/10 02:21:11 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/10 02:21:16 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/10 02:21:16 App: TestAdvancedReplaceOps got un-deployed
2021/12/10 02:21:16 No eventing-con process running
2021/12/10 02:21:16 Function: TestAdvancedReplaceOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedReplaceOps deleting in the background"
}
2021/12/10 02:21:18 Context: Bucket flush, response code: 200 dump: 
2021/12/10 02:21:28 Waiting for dst bucket item count to get to: 0 curr count: 584
2021/12/10 02:21:33 Waiting for dst bucket item count to get to: 0 curr count: 72
2021/12/10 02:21:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 02:21:35 Context: Bucket flush, response code: 200 dump: 
2021/12/10 02:21:40 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 02:21:40 Testing upsert operation on source bucket
2021/12/10 02:21:40 Deploying app: TestAdvancedReplaceOps. Item count src bucket: 0 dst bucket: 0
2021/12/10 02:21:41 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedReplaceOps' in metakv",
  "warnings": null
 }
}
2021/12/10 02:21:46 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/10 02:21:51 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/10 02:21:56 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/10 02:22:01 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/10 02:22:01 App: TestAdvancedReplaceOps got deployed
2021/12/10 02:22:01 Starting bucket ops to source bucket
2021/12/10 02:22:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 02:22:02 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 02:22:02 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/10 02:22:02 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 02:22:07 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/10 02:22:12 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/10 02:22:17 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/10 02:22:22 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/10 02:22:22 App: TestAdvancedReplaceOps got un-deployed
2021/12/10 02:22:22 No eventing-con process running
2021/12/10 02:22:22 Function: TestAdvancedReplaceOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedReplaceOps deleting in the background"
}
2021/12/10 02:22:24 Context: Bucket flush, response code: 200 dump: 
2021/12/10 02:22:34 Waiting for dst bucket item count to get to: 0 curr count: 119
2021/12/10 02:22:39 Waiting for dst bucket item count to get to: 0 curr count: 41
2021/12/10 02:22:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 02:22:41 Context: Bucket flush, response code: 200 dump: 
2021/12/10 02:22:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedReplaceOps (147.94s)
=== RUN   TestAdvancedDeleteOps
2021/12/10 02:22:46 Starting bucket ops to source bucket
2021/12/10 02:22:46 Deploying app: TestAdvancedDeleteOps. Item count src bucket: 0 dst bucket: 0
2021/12/10 02:22:47 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedDeleteOps' in metakv",
  "warnings": null
 }
}
2021/12/10 02:22:52 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/10 02:22:57 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/10 02:23:02 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/10 02:23:07 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/10 02:23:07 App: TestAdvancedDeleteOps got deployed
2021/12/10 02:23:07 Starting bucket ops to source bucket
2021/12/10 02:23:12 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/10 02:23:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 02:23:12 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 02:23:12 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/10 02:23:12 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 02:23:17 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/10 02:23:22 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/10 02:23:27 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/10 02:23:32 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/10 02:23:32 App: TestAdvancedDeleteOps got un-deployed
2021/12/10 02:23:32 No eventing-con process running
2021/12/10 02:23:32 Function: TestAdvancedDeleteOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedDeleteOps deleting in the background"
}
2021/12/10 02:23:34 Context: Bucket flush, response code: 200 dump: 
2021/12/10 02:23:44 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/10 02:23:49 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/10 02:23:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 02:23:51 Context: Bucket flush, response code: 200 dump: 
2021/12/10 02:23:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 02:23:56 Testing delete operation on source bucket
2021/12/10 02:23:56 Deploying app: TestAdvancedDeleteOps. Item count src bucket: 0 dst bucket: 0
2021/12/10 02:23:57 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedDeleteOps' in metakv",
  "warnings": null
 }
}
2021/12/10 02:24:02 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/10 02:24:07 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/10 02:24:12 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/10 02:24:17 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/10 02:24:17 App: TestAdvancedDeleteOps got deployed
2021/12/10 02:24:17 Starting bucket ops to source bucket
2021/12/10 02:24:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 02:24:17 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 02:24:17 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/10 02:24:17 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 02:24:22 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/10 02:24:27 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/10 02:24:32 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/10 02:24:37 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/10 02:24:37 App: TestAdvancedDeleteOps got un-deployed
2021/12/10 02:24:37 No eventing-con process running
2021/12/10 02:24:37 Function: TestAdvancedDeleteOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedDeleteOps deleting in the background"
}
2021/12/10 02:24:39 Context: Bucket flush, response code: 200 dump: 
2021/12/10 02:24:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 02:24:46 Context: Bucket flush, response code: 200 dump: 
2021/12/10 02:24:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedDeleteOps (125.60s)
=== RUN   TestEnoentAdvancedGet
2021/12/10 02:24:51 Deploying app: TestEnoentAdvancedGet. Item count src bucket: 0 dst bucket: 0
2021/12/10 02:24:52 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedGet' in metakv",
  "warnings": null
 }
}
2021/12/10 02:24:57 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/10 02:25:02 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/10 02:25:07 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/10 02:25:12 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/10 02:25:12 App: TestEnoentAdvancedGet got deployed
2021/12/10 02:25:12 Starting bucket ops to source bucket
2021/12/10 02:25:18 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/10 02:25:23 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/10 02:25:23 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/10 02:25:23 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 02:25:23 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/10 02:25:23 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 02:25:28 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/10 02:25:33 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/10 02:25:38 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/10 02:25:43 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/10 02:25:43 App: TestEnoentAdvancedGet got un-deployed
2021/12/10 02:25:43 No eventing-con process running
2021/12/10 02:25:43 Function: TestEnoentAdvancedGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedGet deleting in the background"
}
2021/12/10 02:25:44 Context: Bucket flush, response code: 200 dump: 
2021/12/10 02:25:54 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/10 02:25:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 02:25:56 Context: Bucket flush, response code: 200 dump: 
2021/12/10 02:26:06 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/10 02:26:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 02:26:06 Testing get enoent operation on source bucket
2021/12/10 02:26:06 Deploying app: TestEnoentAdvancedGet. Item count src bucket: 0 dst bucket: 0
2021/12/10 02:26:07 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedGet' in metakv",
  "warnings": null
 }
}
2021/12/10 02:26:12 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/10 02:26:17 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/10 02:26:22 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/10 02:26:27 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/10 02:26:27 App: TestEnoentAdvancedGet got deployed
2021/12/10 02:26:27 Starting bucket ops to source bucket
2021/12/10 02:26:33 Waiting for dst bucket item count to get to: 200 curr count: 0
2021/12/10 02:26:38 Waiting for dst bucket item count to get to: 200 curr count: 100
2021/12/10 02:26:43 Waiting for dst bucket item count to get to: 200 curr count: 100
2021/12/10 02:26:43 src & dst bucket item count matched up. src bucket count: 200 dst bucket count: 200
2021/12/10 02:26:43 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 02:26:43 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/10 02:26:43 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 02:26:48 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/10 02:26:53 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/10 02:26:58 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/10 02:27:03 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/10 02:27:03 App: TestEnoentAdvancedGet got un-deployed
2021/12/10 02:27:03 No eventing-con process running
2021/12/10 02:27:03 Function: TestEnoentAdvancedGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedGet deleting in the background"
}
2021/12/10 02:27:04 Context: Bucket flush, response code: 200 dump: 
2021/12/10 02:27:14 Waiting for dst bucket item count to get to: 0 curr count: 200
2021/12/10 02:27:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 02:27:16 Context: Bucket flush, response code: 200 dump: 
2021/12/10 02:27:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEnoentAdvancedGet (149.90s)
=== RUN   TestEnoentAdvancedReplace
2021/12/10 02:27:21 Deploying app: TestEnoentAdvancedReplace. Item count src bucket: 0 dst bucket: 0
2021/12/10 02:27:22 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedReplace' in metakv",
  "warnings": null
 }
}
2021/12/10 02:27:27 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/10 02:27:32 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/10 02:27:37 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/10 02:27:42 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/10 02:27:42 App: TestEnoentAdvancedReplace got deployed
2021/12/10 02:27:42 Starting bucket ops to source bucket
2021/12/10 02:27:47 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/10 02:27:52 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/10 02:27:52 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/10 02:27:52 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 02:27:52 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/10 02:27:53 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 02:27:58 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/10 02:28:03 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/10 02:28:08 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/10 02:28:13 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/10 02:28:13 App: TestEnoentAdvancedReplace got un-deployed
2021/12/10 02:28:13 No eventing-con process running
2021/12/10 02:28:13 Function: TestEnoentAdvancedReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedReplace deleting in the background"
}
2021/12/10 02:28:14 Context: Bucket flush, response code: 200 dump: 
2021/12/10 02:28:24 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/10 02:28:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 02:28:26 Context: Bucket flush, response code: 200 dump: 
2021/12/10 02:28:36 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/10 02:28:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 02:28:36 Testing get enoent operation on source bucket
2021/12/10 02:28:36 Deploying app: TestEnoentAdvancedReplace. Item count src bucket: 0 dst bucket: 0
2021/12/10 02:28:37 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedReplace' in metakv",
  "warnings": null
 }
}
2021/12/10 02:28:42 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/10 02:28:47 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/10 02:28:52 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/10 02:28:57 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/10 02:28:57 App: TestEnoentAdvancedReplace got deployed
2021/12/10 02:28:57 Starting bucket ops to source bucket
2021/12/10 02:29:02 Waiting for dst bucket item count to get to: 200 curr count: 0
2021/12/10 02:29:07 Waiting for dst bucket item count to get to: 200 curr count: 100
2021/12/10 02:29:13 Waiting for dst bucket item count to get to: 200 curr count: 100
2021/12/10 02:29:13 src & dst bucket item count matched up. src bucket count: 200 dst bucket count: 200
2021/12/10 02:29:13 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 02:29:13 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/10 02:29:13 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 02:29:18 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/10 02:29:23 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/10 02:29:28 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/10 02:29:33 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/10 02:29:33 App: TestEnoentAdvancedReplace got un-deployed
2021/12/10 02:29:33 No eventing-con process running
2021/12/10 02:29:33 Function: TestEnoentAdvancedReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedReplace deleting in the background"
}
2021/12/10 02:29:34 Context: Bucket flush, response code: 200 dump: 
2021/12/10 02:29:44 Waiting for dst bucket item count to get to: 0 curr count: 200
2021/12/10 02:29:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 02:29:46 Context: Bucket flush, response code: 200 dump: 
2021/12/10 02:29:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEnoentAdvancedReplace (149.94s)
=== RUN   TestEnoentAdvancedDelete
2021/12/10 02:29:51 Deploying app: TestEnoentAdvancedDelete. Item count src bucket: 0 dst bucket: 0
2021/12/10 02:29:52 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedDelete' in metakv",
  "warnings": null
 }
}
2021/12/10 02:29:57 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/10 02:30:02 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/10 02:30:07 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/10 02:30:12 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/10 02:30:12 App: TestEnoentAdvancedDelete got deployed
2021/12/10 02:30:12 Starting bucket ops to source bucket
2021/12/10 02:30:17 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/10 02:30:22 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/10 02:30:22 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/10 02:30:22 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 02:30:22 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/10 02:30:22 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 02:30:27 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/10 02:30:32 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/10 02:30:37 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/10 02:30:42 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/10 02:30:42 App: TestEnoentAdvancedDelete got un-deployed
2021/12/10 02:30:42 No eventing-con process running
2021/12/10 02:30:43 Function: TestEnoentAdvancedDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedDelete deleting in the background"
}
2021/12/10 02:30:44 Context: Bucket flush, response code: 200 dump: 
2021/12/10 02:30:54 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/10 02:30:59 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/10 02:30:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 02:31:01 Context: Bucket flush, response code: 200 dump: 
2021/12/10 02:31:11 Waiting for dst bucket item count to get to: 0 curr count: 98
2021/12/10 02:31:16 Waiting for dst bucket item count to get to: 0 curr count: 98
2021/12/10 02:31:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 02:31:16 Testing get enoent operation on source bucket
2021/12/10 02:31:16 Deploying app: TestEnoentAdvancedDelete. Item count src bucket: 0 dst bucket: 0
2021/12/10 02:31:17 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedDelete' in metakv",
  "warnings": null
 }
}
2021/12/10 02:31:22 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/10 02:31:27 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/10 02:31:32 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/10 02:31:37 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/10 02:31:37 App: TestEnoentAdvancedDelete got deployed
2021/12/10 02:31:37 Starting bucket ops to source bucket
2021/12/10 02:31:42 Waiting for dst bucket item count to get to: 200 curr count: 0
2021/12/10 02:31:47 Waiting for dst bucket item count to get to: 200 curr count: 100
2021/12/10 02:31:52 Waiting for dst bucket item count to get to: 200 curr count: 100
2021/12/10 02:31:52 src & dst bucket item count matched up. src bucket count: 200 dst bucket count: 200
2021/12/10 02:31:52 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 02:31:52 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/10 02:31:52 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 02:31:57 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/10 02:32:02 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/10 02:32:07 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/10 02:32:12 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/10 02:32:12 App: TestEnoentAdvancedDelete got un-deployed
2021/12/10 02:32:12 No eventing-con process running
2021/12/10 02:32:12 Function: TestEnoentAdvancedDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedDelete deleting in the background"
}
2021/12/10 02:32:14 Context: Bucket flush, response code: 200 dump: 
2021/12/10 02:32:24 Waiting for dst bucket item count to get to: 0 curr count: 200
2021/12/10 02:32:29 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/10 02:32:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 02:32:31 Context: Bucket flush, response code: 200 dump: 
2021/12/10 02:32:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEnoentAdvancedDelete (164.88s)
=== RUN   TestAdvancedInsertKeyExist
2021/12/10 02:32:36 Deploying app: TestAdvancedInsertKeyExist. Item count src bucket: 0 dst bucket: 0
2021/12/10 02:32:37 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedInsertKeyExist' in metakv",
  "warnings": null
 }
}
2021/12/10 02:32:42 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/10 02:32:47 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/10 02:32:52 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/10 02:32:57 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/10 02:32:57 App: TestAdvancedInsertKeyExist got deployed
2021/12/10 02:32:57 Starting bucket ops to source bucket
2021/12/10 02:32:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 02:32:57 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 02:32:57 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/10 02:32:57 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 02:33:02 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/10 02:33:07 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/10 02:33:12 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/10 02:33:17 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/10 02:33:17 App: TestAdvancedInsertKeyExist got un-deployed
2021/12/10 02:33:17 No eventing-con process running
2021/12/10 02:33:17 Function: TestAdvancedInsertKeyExist delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedInsertKeyExist deleting in the background"
}
2021/12/10 02:33:19 Context: Bucket flush, response code: 200 dump: 
2021/12/10 02:33:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 02:33:27 Context: Bucket flush, response code: 200 dump: 
2021/12/10 02:33:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 02:33:32 Pumping document in the destination bucket
2021/12/10 02:33:32 Starting bucket ops to source bucket
2021/12/10 02:33:32 Deploying app: TestAdvancedInsertKeyExist. Item count src bucket: 0 dst bucket: 0
2021/12/10 02:33:33 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedInsertKeyExist' in metakv",
  "warnings": null
 }
}
2021/12/10 02:33:38 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/10 02:33:43 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/10 02:33:48 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/10 02:33:53 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/10 02:33:53 App: TestAdvancedInsertKeyExist got deployed
2021/12/10 02:33:53 Starting bucket ops to source bucket
2021/12/10 02:33:58 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/10 02:34:03 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/10 02:34:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 02:34:03 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 02:34:03 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/10 02:34:03 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 02:34:08 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/10 02:34:13 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/10 02:34:18 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/10 02:34:23 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/10 02:34:23 App: TestAdvancedInsertKeyExist got un-deployed
2021/12/10 02:34:23 No eventing-con process running
2021/12/10 02:34:23 Function: TestAdvancedInsertKeyExist delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedInsertKeyExist deleting in the background"
}
2021/12/10 02:34:25 Context: Bucket flush, response code: 200 dump: 
2021/12/10 02:34:35 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/10 02:34:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 02:34:37 Context: Bucket flush, response code: 200 dump: 
2021/12/10 02:34:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedInsertKeyExist (125.59s)
=== RUN   TestExipryGet
2021/12/10 02:34:42 Deploying app: TestExipryGet. Item count src bucket: 0 dst bucket: 0
2021/12/10 02:34:43 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryGet' in metakv",
  "warnings": null
 }
}
2021/12/10 02:34:48 Waiting for app: TestExipryGet to get deployed
2021/12/10 02:34:53 Waiting for app: TestExipryGet to get deployed
2021/12/10 02:34:58 Waiting for app: TestExipryGet to get deployed
2021/12/10 02:35:03 Waiting for app: TestExipryGet to get deployed
2021/12/10 02:35:03 App: TestExipryGet got deployed
2021/12/10 02:35:03 Starting bucket ops to source bucket
2021/12/10 02:35:48 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/10 02:35:53 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/10 02:35:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 02:35:53 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 02:35:53 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/10 02:35:53 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 02:35:58 Waiting for app: TestExipryGet to get un-deployed
2021/12/10 02:36:03 Waiting for app: TestExipryGet to get un-deployed
2021/12/10 02:36:08 Waiting for app: TestExipryGet to get un-deployed
2021/12/10 02:36:13 Waiting for app: TestExipryGet to get un-deployed
2021/12/10 02:36:13 App: TestExipryGet got un-deployed
2021/12/10 02:36:13 No eventing-con process running
2021/12/10 02:36:13 Function: TestExipryGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryGet deleting in the background"
}
2021/12/10 02:36:15 Context: Bucket flush, response code: 200 dump: 
2021/12/10 02:36:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 02:36:22 Context: Bucket flush, response code: 200 dump: 
2021/12/10 02:36:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 02:36:27 Pumping document in the destination bucket for expiry get
2021/12/10 02:36:27 Starting bucket ops to source bucket
2021/12/10 02:36:27 Deploying app: TestExipryGet. Item count src bucket: 0 dst bucket: 0
2021/12/10 02:36:29 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryGet' in metakv",
  "warnings": null
 }
}
2021/12/10 02:36:34 Waiting for app: TestExipryGet to get deployed
2021/12/10 02:36:39 Waiting for app: TestExipryGet to get deployed
2021/12/10 02:36:44 Waiting for app: TestExipryGet to get deployed
2021/12/10 02:36:49 Waiting for app: TestExipryGet to get deployed
2021/12/10 02:36:49 App: TestExipryGet got deployed
2021/12/10 02:36:49 Starting bucket ops to source bucket
2021/12/10 02:37:34 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/10 02:37:39 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/10 02:37:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 02:37:39 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 02:37:39 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/10 02:37:39 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 02:37:44 Waiting for app: TestExipryGet to get un-deployed
2021/12/10 02:37:49 Waiting for app: TestExipryGet to get un-deployed
2021/12/10 02:37:54 Waiting for app: TestExipryGet to get un-deployed
2021/12/10 02:37:59 Waiting for app: TestExipryGet to get un-deployed
2021/12/10 02:37:59 App: TestExipryGet got un-deployed
2021/12/10 02:37:59 No eventing-con process running
2021/12/10 02:37:59 Function: TestExipryGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryGet deleting in the background"
}
2021/12/10 02:38:01 Context: Bucket flush, response code: 200 dump: 
2021/12/10 02:38:11 Waiting for dst bucket item count to get to: 0 curr count: 89
2021/12/10 02:38:16 Waiting for dst bucket item count to get to: 0 curr count: 89
2021/12/10 02:38:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 02:38:17 Context: Bucket flush, response code: 200 dump: 
2021/12/10 02:38:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestExipryGet (220.87s)
=== RUN   TestExipryInsert
2021/12/10 02:38:22 Deploying app: TestExipryInsert. Item count src bucket: 0 dst bucket: 0
2021/12/10 02:38:24 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryInsert' in metakv",
  "warnings": null
 }
}
2021/12/10 02:38:29 Waiting for app: TestExipryInsert to get deployed
2021/12/10 02:38:34 Waiting for app: TestExipryInsert to get deployed
2021/12/10 02:38:39 Waiting for app: TestExipryInsert to get deployed
2021/12/10 02:38:44 Waiting for app: TestExipryInsert to get deployed
2021/12/10 02:38:44 App: TestExipryInsert got deployed
2021/12/10 02:38:44 Starting bucket ops to source bucket
2021/12/10 02:39:29 Waiting for dst bucket item count to get to: 0 curr count: 200
2021/12/10 02:39:34 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/10 02:39:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 02:39:34 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 02:39:34 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/10 02:39:34 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 02:39:39 Waiting for app: TestExipryInsert to get un-deployed
2021/12/10 02:39:44 Waiting for app: TestExipryInsert to get un-deployed
2021/12/10 02:39:49 Waiting for app: TestExipryInsert to get un-deployed
2021/12/10 02:39:54 Waiting for app: TestExipryInsert to get un-deployed
2021/12/10 02:39:54 App: TestExipryInsert got un-deployed
2021/12/10 02:39:54 No eventing-con process running
2021/12/10 02:39:54 Function: TestExipryInsert delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryInsert deleting in the background"
}
2021/12/10 02:39:56 Context: Bucket flush, response code: 200 dump: 
2021/12/10 02:40:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 02:40:03 Context: Bucket flush, response code: 200 dump: 
2021/12/10 02:40:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 02:40:08 Testing on the destination bucket for expiry insert
2021/12/10 02:40:08 Deploying app: TestExipryInsert. Item count src bucket: 0 dst bucket: 0
2021/12/10 02:40:09 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryInsert' in metakv",
  "warnings": null
 }
}
2021/12/10 02:40:14 Waiting for app: TestExipryInsert to get deployed
2021/12/10 02:40:19 Waiting for app: TestExipryInsert to get deployed
2021/12/10 02:40:24 Waiting for app: TestExipryInsert to get deployed
2021/12/10 02:40:29 Waiting for app: TestExipryInsert to get deployed
2021/12/10 02:40:29 App: TestExipryInsert got deployed
2021/12/10 02:40:29 Starting bucket ops to source bucket
2021/12/10 02:41:15 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/10 02:41:20 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/10 02:41:25 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/10 02:41:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 02:41:25 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 02:41:25 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/10 02:41:25 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 02:41:30 Waiting for app: TestExipryInsert to get un-deployed
2021/12/10 02:41:35 Waiting for app: TestExipryInsert to get un-deployed
2021/12/10 02:41:40 Waiting for app: TestExipryInsert to get un-deployed
2021/12/10 02:41:45 Waiting for app: TestExipryInsert to get un-deployed
2021/12/10 02:41:45 App: TestExipryInsert got un-deployed
2021/12/10 02:41:45 No eventing-con process running
2021/12/10 02:41:45 Function: TestExipryInsert delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryInsert deleting in the background"
}
2021/12/10 02:41:46 Context: Bucket flush, response code: 200 dump: 
2021/12/10 02:41:56 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/10 02:41:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 02:41:58 Context: Bucket flush, response code: 200 dump: 
2021/12/10 02:42:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestExipryInsert (220.73s)
=== RUN   TestExipryReplace
2021/12/10 02:42:03 Deploying app: TestExipryReplace. Item count src bucket: 0 dst bucket: 0
2021/12/10 02:42:04 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryReplace' in metakv",
  "warnings": null
 }
}
2021/12/10 02:42:09 Waiting for app: TestExipryReplace to get deployed
2021/12/10 02:42:14 Waiting for app: TestExipryReplace to get deployed
2021/12/10 02:42:19 Waiting for app: TestExipryReplace to get deployed
2021/12/10 02:42:24 Waiting for app: TestExipryReplace to get deployed
2021/12/10 02:42:24 App: TestExipryReplace got deployed
2021/12/10 02:42:24 Starting bucket ops to source bucket
2021/12/10 02:43:10 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/10 02:43:15 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/10 02:43:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 02:43:15 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 02:43:15 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/10 02:43:15 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 02:43:20 Waiting for app: TestExipryReplace to get un-deployed
2021/12/10 02:43:25 Waiting for app: TestExipryReplace to get un-deployed
2021/12/10 02:43:30 Waiting for app: TestExipryReplace to get un-deployed
2021/12/10 02:43:35 Waiting for app: TestExipryReplace to get un-deployed
2021/12/10 02:43:35 App: TestExipryReplace got un-deployed
2021/12/10 02:43:35 No eventing-con process running
2021/12/10 02:43:35 Function: TestExipryReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryReplace deleting in the background"
}
2021/12/10 02:43:36 Context: Bucket flush, response code: 200 dump: 
2021/12/10 02:43:41 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 02:43:44 Context: Bucket flush, response code: 200 dump: 
2021/12/10 02:43:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 02:43:49 Testing on the destination bucket for expiry insert
2021/12/10 02:43:49 Deploying app: TestExipryReplace. Item count src bucket: 0 dst bucket: 0
2021/12/10 02:43:50 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryReplace' in metakv",
  "warnings": null
 }
}
2021/12/10 02:43:55 Waiting for app: TestExipryReplace to get deployed
2021/12/10 02:44:00 Waiting for app: TestExipryReplace to get deployed
2021/12/10 02:44:05 Waiting for app: TestExipryReplace to get deployed
2021/12/10 02:44:10 Waiting for app: TestExipryReplace to get deployed
2021/12/10 02:44:10 App: TestExipryReplace got deployed
2021/12/10 02:44:10 Starting bucket ops to source bucket
2021/12/10 02:44:55 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/10 02:45:00 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/10 02:45:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 02:45:00 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 02:45:00 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/10 02:45:00 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 02:45:05 Waiting for app: TestExipryReplace to get un-deployed
2021/12/10 02:45:10 Waiting for app: TestExipryReplace to get un-deployed
2021/12/10 02:45:15 Waiting for app: TestExipryReplace to get un-deployed
2021/12/10 02:45:20 Waiting for app: TestExipryReplace to get un-deployed
2021/12/10 02:45:20 App: TestExipryReplace got un-deployed
2021/12/10 02:45:20 No eventing-con process running
2021/12/10 02:45:20 Function: TestExipryReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryReplace deleting in the background"
}
2021/12/10 02:45:22 Context: Bucket flush, response code: 200 dump: 
2021/12/10 02:45:32 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/10 02:45:37 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/10 02:45:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 02:45:39 Context: Bucket flush, response code: 200 dump: 
2021/12/10 02:45:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestExipryReplace (220.69s)
=== RUN   TestExipryUpsert
2021/12/10 02:45:44 Deploying app: TestExipryUpsert. Item count src bucket: 0 dst bucket: 0
2021/12/10 02:45:45 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryUpsert' in metakv",
  "warnings": null
 }
}
2021/12/10 02:45:50 Waiting for app: TestExipryUpsert to get deployed
2021/12/10 02:45:55 Waiting for app: TestExipryUpsert to get deployed
2021/12/10 02:46:00 Waiting for app: TestExipryUpsert to get deployed
2021/12/10 02:46:05 Waiting for app: TestExipryUpsert to get deployed
2021/12/10 02:46:05 App: TestExipryUpsert got deployed
2021/12/10 02:46:05 Starting bucket ops to source bucket
2021/12/10 02:46:50 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/10 02:46:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 02:46:50 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 02:46:50 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/10 02:46:50 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 02:46:55 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/10 02:47:00 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/10 02:47:05 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/10 02:47:10 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/10 02:47:10 App: TestExipryUpsert got un-deployed
2021/12/10 02:47:10 No eventing-con process running
2021/12/10 02:47:10 Function: TestExipryUpsert delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryUpsert deleting in the background"
}
2021/12/10 02:47:12 Context: Bucket flush, response code: 200 dump: 
2021/12/10 02:47:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 02:47:19 Context: Bucket flush, response code: 200 dump: 
2021/12/10 02:47:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 02:47:24 Testing on the destination bucket for expiry insert
2021/12/10 02:47:24 Deploying app: TestExipryUpsert. Item count src bucket: 0 dst bucket: 0
2021/12/10 02:47:25 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryUpsert' in metakv",
  "warnings": null
 }
}
2021/12/10 02:47:30 Waiting for app: TestExipryUpsert to get deployed
2021/12/10 02:47:36 Waiting for app: TestExipryUpsert to get deployed
2021/12/10 02:47:41 Waiting for app: TestExipryUpsert to get deployed
2021/12/10 02:47:46 Waiting for app: TestExipryUpsert to get deployed
2021/12/10 02:47:46 App: TestExipryUpsert got deployed
2021/12/10 02:47:46 Starting bucket ops to source bucket
2021/12/10 02:48:31 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/10 02:48:31 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 02:48:31 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 02:48:31 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/10 02:48:31 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 02:48:36 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/10 02:48:41 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/10 02:48:46 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/10 02:48:51 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/10 02:48:51 App: TestExipryUpsert got un-deployed
2021/12/10 02:48:51 No eventing-con process running
2021/12/10 02:48:51 Function: TestExipryUpsert delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryUpsert deleting in the background"
}
2021/12/10 02:48:53 Context: Bucket flush, response code: 200 dump: 
2021/12/10 02:49:03 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/10 02:49:08 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/10 02:49:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 02:49:09 Context: Bucket flush, response code: 200 dump: 
2021/12/10 02:49:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestExipryUpsert (210.45s)
=== RUN   TestCasReplace
2021/12/10 02:49:14 Deploying app: TestCasReplace. Item count src bucket: 0 dst bucket: 0
2021/12/10 02:49:15 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCasReplace' in metakv",
  "warnings": null
 }
}
2021/12/10 02:49:20 Waiting for app: TestCasReplace to get deployed
2021/12/10 02:49:25 Waiting for app: TestCasReplace to get deployed
2021/12/10 02:49:30 Waiting for app: TestCasReplace to get deployed
2021/12/10 02:49:35 Waiting for app: TestCasReplace to get deployed
2021/12/10 02:49:35 App: TestCasReplace got deployed
2021/12/10 02:49:35 Starting bucket ops to source bucket
2021/12/10 02:49:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 02:49:36 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 02:49:36 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/10 02:49:36 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 02:49:41 Waiting for app: TestCasReplace to get un-deployed
2021/12/10 02:49:46 Waiting for app: TestCasReplace to get un-deployed
2021/12/10 02:49:51 Waiting for app: TestCasReplace to get un-deployed
2021/12/10 02:49:56 Waiting for app: TestCasReplace to get un-deployed
2021/12/10 02:49:56 App: TestCasReplace got un-deployed
2021/12/10 02:49:56 No eventing-con process running
2021/12/10 02:49:56 Function: TestCasReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCasReplace deleting in the background"
}
2021/12/10 02:49:58 Context: Bucket flush, response code: 200 dump: 
2021/12/10 02:50:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 02:50:05 Context: Bucket flush, response code: 200 dump: 
2021/12/10 02:50:10 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 02:50:10 Testing on the destination bucket for upsert with cas
2021/12/10 02:50:10 Starting bucket ops to source bucket
2021/12/10 02:50:10 Deploying app: TestCasReplace. Item count src bucket: 0 dst bucket: 0
2021/12/10 02:50:11 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCasReplace' in metakv",
  "warnings": null
 }
}
2021/12/10 02:50:16 Waiting for app: TestCasReplace to get deployed
2021/12/10 02:50:21 Waiting for app: TestCasReplace to get deployed
2021/12/10 02:50:26 Waiting for app: TestCasReplace to get deployed
2021/12/10 02:50:31 Waiting for app: TestCasReplace to get deployed
2021/12/10 02:50:31 App: TestCasReplace got deployed
2021/12/10 02:50:31 Starting bucket ops to source bucket
2021/12/10 02:50:36 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/10 02:50:41 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/10 02:50:41 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 02:50:41 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 02:50:41 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/10 02:50:41 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 02:50:46 Waiting for app: TestCasReplace to get un-deployed
2021/12/10 02:50:51 Waiting for app: TestCasReplace to get un-deployed
2021/12/10 02:50:56 Waiting for app: TestCasReplace to get un-deployed
2021/12/10 02:51:01 Waiting for app: TestCasReplace to get un-deployed
2021/12/10 02:51:01 App: TestCasReplace got un-deployed
2021/12/10 02:51:01 No eventing-con process running
2021/12/10 02:51:01 Function: TestCasReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCasReplace deleting in the background"
}
2021/12/10 02:51:03 Context: Bucket flush, response code: 200 dump: 
2021/12/10 02:51:13 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/10 02:51:18 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/10 02:51:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 02:51:20 Context: Bucket flush, response code: 200 dump: 
2021/12/10 02:51:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCasReplace (130.62s)
=== RUN   TestCasDelete
2021/12/10 02:51:25 Deploying app: TestCasDelete. Item count src bucket: 0 dst bucket: 0
2021/12/10 02:51:26 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCasDelete' in metakv",
  "warnings": null
 }
}
2021/12/10 02:51:31 Waiting for app: TestCasDelete to get deployed
2021/12/10 02:51:36 Waiting for app: TestCasDelete to get deployed
2021/12/10 02:51:41 Waiting for app: TestCasDelete to get deployed
2021/12/10 02:51:46 Waiting for app: TestCasDelete to get deployed
2021/12/10 02:51:46 App: TestCasDelete got deployed
2021/12/10 02:51:46 Starting bucket ops to source bucket
2021/12/10 02:51:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 02:51:46 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 02:51:46 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/10 02:51:46 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 02:51:51 Waiting for app: TestCasDelete to get un-deployed
2021/12/10 02:51:56 Waiting for app: TestCasDelete to get un-deployed
2021/12/10 02:52:01 Waiting for app: TestCasDelete to get un-deployed
2021/12/10 02:52:06 Waiting for app: TestCasDelete to get un-deployed
2021/12/10 02:52:07 App: TestCasDelete got un-deployed
2021/12/10 02:52:07 No eventing-con process running
2021/12/10 02:52:07 Function: TestCasDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCasDelete deleting in the background"
}
2021/12/10 02:52:08 Context: Bucket flush, response code: 200 dump: 
2021/12/10 02:52:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 02:52:16 Context: Bucket flush, response code: 200 dump: 
2021/12/10 02:52:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 02:52:21 Testing on the destination bucket for delete with cas
2021/12/10 02:52:21 Starting bucket ops to source bucket
2021/12/10 02:52:21 Deploying app: TestCasDelete. Item count src bucket: 0 dst bucket: 0
2021/12/10 02:52:22 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCasDelete' in metakv",
  "warnings": null
 }
}
2021/12/10 02:52:27 Waiting for app: TestCasDelete to get deployed
2021/12/10 02:52:32 Waiting for app: TestCasDelete to get deployed
2021/12/10 02:52:37 Waiting for app: TestCasDelete to get deployed
2021/12/10 02:52:42 Waiting for app: TestCasDelete to get deployed
2021/12/10 02:52:42 App: TestCasDelete got deployed
2021/12/10 02:52:42 Starting bucket ops to source bucket
2021/12/10 02:52:47 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/10 02:52:52 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/10 02:52:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 02:52:52 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 02:52:52 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/10 02:52:52 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 02:52:57 Waiting for app: TestCasDelete to get un-deployed
2021/12/10 02:53:02 Waiting for app: TestCasDelete to get un-deployed
2021/12/10 02:53:07 Waiting for app: TestCasDelete to get un-deployed
2021/12/10 02:53:12 Waiting for app: TestCasDelete to get un-deployed
2021/12/10 02:53:12 App: TestCasDelete got un-deployed
2021/12/10 02:53:12 No eventing-con process running
2021/12/10 02:53:12 Function: TestCasDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCasDelete deleting in the background"
}
2021/12/10 02:53:14 Context: Bucket flush, response code: 200 dump: 
2021/12/10 02:53:24 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/10 02:53:29 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/10 02:53:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 02:53:31 Context: Bucket flush, response code: 200 dump: 
2021/12/10 02:53:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCasDelete (130.73s)
=== RUN   TestCountersIncrement
2021/12/10 02:53:36 Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/10 02:53:41 Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/10 02:53:41 Removing node(s): [] from the cluster
2021/12/10 02:53:47 Rebalance progress: 100
2021/12/10 02:53:47 Deploying app: TestCountersIncrement. Item count src bucket: 0 dst bucket: 0
2021/12/10 02:53:48 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCountersIncrement' in metakv",
  "warnings": null
 }
}
2021/12/10 02:53:53 Waiting for app: TestCountersIncrement to get deployed
2021/12/10 02:53:58 Waiting for app: TestCountersIncrement to get deployed
2021/12/10 02:54:03 Waiting for app: TestCountersIncrement to get deployed
2021/12/10 02:54:08 Waiting for app: TestCountersIncrement to get deployed
2021/12/10 02:54:08 App: TestCountersIncrement got deployed
2021/12/10 02:54:08 Starting bucket ops to source bucket
2021/12/10 02:54:14 Waiting for dst bucket item count to get to: 1024 curr count: 0
2021/12/10 02:54:19 Waiting for dst bucket item count to get to: 1024 curr count: 512
2021/12/10 02:54:24 Waiting for dst bucket item count to get to: 1024 curr count: 512
2021/12/10 02:54:24 src & dst bucket item count matched up. src bucket count: 1024 dst bucket count: 1024
2021/12/10 02:54:24 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 02:54:24 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 02:54:29 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/10 02:54:34 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/10 02:54:39 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/10 02:54:44 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/10 02:54:44 App: TestCountersIncrement got un-deployed
2021/12/10 02:54:44 No eventing-con process running
2021/12/10 02:54:44 Function: TestCountersIncrement delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCountersIncrement deleting in the background"
}
2021/12/10 02:54:46 Context: Bucket flush, response code: 200 dump: 
2021/12/10 02:54:56 Waiting for dst bucket item count to get to: 0 curr count: 1025
2021/12/10 02:54:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 02:54:58 Context: Bucket flush, response code: 200 dump: 
2021/12/10 02:55:08 Waiting for dst bucket item count to get to: 0 curr count: 544
2021/12/10 02:55:13 Waiting for dst bucket item count to get to: 0 curr count: 32
2021/12/10 02:55:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 02:55:13 Deploying app: TestCountersIncrement. Item count src bucket: 0 dst bucket: 0
2021/12/10 02:55:14 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCountersIncrement' in metakv",
  "warnings": null
 }
}
2021/12/10 02:55:19 Waiting for app: TestCountersIncrement to get deployed
2021/12/10 02:55:24 Waiting for app: TestCountersIncrement to get deployed
2021/12/10 02:55:29 Waiting for app: TestCountersIncrement to get deployed
2021/12/10 02:55:34 Waiting for app: TestCountersIncrement to get deployed
2021/12/10 02:55:34 App: TestCountersIncrement got deployed
2021/12/10 02:55:34 Starting bucket ops to source bucket
2021/12/10 02:55:40 Waiting for dst bucket item count to get to: 2048 curr count: 0
2021/12/10 02:55:45 Waiting for dst bucket item count to get to: 2048 curr count: 1024
2021/12/10 02:55:45 src & dst bucket item count matched up. src bucket count: 2048 dst bucket count: 2048
2021/12/10 02:55:45 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 02:55:45 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 02:55:50 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/10 02:55:55 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/10 02:56:00 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/10 02:56:05 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/10 02:56:05 App: TestCountersIncrement got un-deployed
2021/12/10 02:56:05 No eventing-con process running
2021/12/10 02:56:05 Function: TestCountersIncrement delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCountersIncrement deleting in the background"
}
2021/12/10 02:56:07 Context: Bucket flush, response code: 200 dump: 
2021/12/10 02:56:17 Waiting for dst bucket item count to get to: 0 curr count: 2048
2021/12/10 02:56:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 02:56:19 Context: Bucket flush, response code: 200 dump: 
2021/12/10 02:56:29 Waiting for dst bucket item count to get to: 0 curr count: 1
2021/12/10 02:56:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 02:56:29 Removing node(s): [http://127.0.0.1:19002 http://127.0.0.1:19003] from the cluster
2021/12/10 02:56:34 Rebalance progress: 100
--- PASS: TestCountersIncrement (178.34s)
=== RUN   TestCountersDecrement
2021/12/10 02:56:34 Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/10 02:56:34 Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/10 02:56:35 Removing node(s): [] from the cluster
2021/12/10 02:56:40 Rebalance progress: 100
2021/12/10 02:56:40 Deploying app: TestCountersDecrement. Item count src bucket: 0 dst bucket: 0
2021/12/10 02:56:41 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCountersDecrement' in metakv",
  "warnings": null
 }
}
2021/12/10 02:56:46 Waiting for app: TestCountersDecrement to get deployed
2021/12/10 02:56:51 Waiting for app: TestCountersDecrement to get deployed
2021/12/10 02:56:56 Waiting for app: TestCountersDecrement to get deployed
2021/12/10 02:57:01 Waiting for app: TestCountersDecrement to get deployed
2021/12/10 02:57:01 App: TestCountersDecrement got deployed
2021/12/10 02:57:01 Starting bucket ops to source bucket
2021/12/10 02:57:07 Waiting for dst bucket item count to get to: 1024 curr count: 0
2021/12/10 02:57:12 Waiting for dst bucket item count to get to: 1024 curr count: 0
2021/12/10 02:57:12 src & dst bucket item count matched up. src bucket count: 1024 dst bucket count: 1024
2021/12/10 02:57:12 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 02:57:12 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 02:57:17 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/10 02:57:22 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/10 02:57:27 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/10 02:57:32 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/10 02:57:32 App: TestCountersDecrement got un-deployed
2021/12/10 02:57:32 No eventing-con process running
2021/12/10 02:57:32 Function: TestCountersDecrement delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCountersDecrement deleting in the background"
}
2021/12/10 02:57:34 Context: Bucket flush, response code: 200 dump: 
2021/12/10 02:57:44 Waiting for dst bucket item count to get to: 0 curr count: 1025
2021/12/10 02:57:49 Waiting for dst bucket item count to get to: 0 curr count: 513
2021/12/10 02:57:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 02:57:51 Context: Bucket flush, response code: 200 dump: 
2021/12/10 02:58:01 Waiting for dst bucket item count to get to: 0 curr count: 943
2021/12/10 02:58:06 Waiting for dst bucket item count to get to: 0 curr count: 943
2021/12/10 02:58:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 02:58:06 Deploying app: TestCountersDecrement. Item count src bucket: 0 dst bucket: 0
2021/12/10 02:58:07 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCountersDecrement' in metakv",
  "warnings": null
 }
}
2021/12/10 02:58:12 Waiting for app: TestCountersDecrement to get deployed
2021/12/10 02:58:17 Waiting for app: TestCountersDecrement to get deployed
2021/12/10 02:58:22 Waiting for app: TestCountersDecrement to get deployed
2021/12/10 02:58:27 Waiting for app: TestCountersDecrement to get deployed
2021/12/10 02:58:27 App: TestCountersDecrement got deployed
2021/12/10 02:58:27 Starting bucket ops to source bucket
2021/12/10 02:58:33 Waiting for dst bucket item count to get to: 2048 curr count: 0
2021/12/10 02:58:38 Waiting for dst bucket item count to get to: 2048 curr count: 1139
2021/12/10 02:58:43 Waiting for dst bucket item count to get to: 2048 curr count: 1139
2021/12/10 02:58:43 src & dst bucket item count matched up. src bucket count: 2048 dst bucket count: 2048
2021/12/10 02:58:43 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 02:58:43 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 02:58:48 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/10 02:58:53 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/10 02:58:58 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/10 02:59:03 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/10 02:59:03 App: TestCountersDecrement got un-deployed
2021/12/10 02:59:03 No eventing-con process running
2021/12/10 02:59:03 Function: TestCountersDecrement delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCountersDecrement deleting in the background"
}
2021/12/10 02:59:05 Context: Bucket flush, response code: 200 dump: 
2021/12/10 02:59:15 Waiting for dst bucket item count to get to: 0 curr count: 2048
2021/12/10 02:59:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 02:59:17 Context: Bucket flush, response code: 200 dump: 
2021/12/10 02:59:27 Waiting for dst bucket item count to get to: 0 curr count: 1
2021/12/10 02:59:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 02:59:27 Removing node(s): [http://127.0.0.1:19002 http://127.0.0.1:19003] from the cluster
2021/12/10 02:59:32 Rebalance progress: 100
--- PASS: TestCountersDecrement (177.90s)
=== RUN   TestCacheGet
2021/12/10 02:59:32 Deploying app: TestCacheGet. Item count src bucket: 0 dst bucket: 0
2021/12/10 02:59:33 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCacheGet' in metakv",
  "warnings": null
 }
}
2021/12/10 02:59:38 Waiting for app: TestCacheGet to get deployed
2021/12/10 02:59:43 Waiting for app: TestCacheGet to get deployed
2021/12/10 02:59:48 Waiting for app: TestCacheGet to get deployed
2021/12/10 02:59:53 Waiting for app: TestCacheGet to get deployed
2021/12/10 02:59:53 App: TestCacheGet got deployed
2021/12/10 02:59:53 Starting bucket ops to source bucket
2021/12/10 02:59:58 Waiting for dst bucket item count to get to: 20 curr count: 0
2021/12/10 03:00:03 Waiting for dst bucket item count to get to: 20 curr count: 0
2021/12/10 03:00:08 Waiting for dst bucket item count to get to: 20 curr count: 0
2021/12/10 03:00:13 Waiting for dst bucket item count to get to: 20 curr count: 8
2021/12/10 03:00:18 Waiting for dst bucket item count to get to: 20 curr count: 16
2021/12/10 03:00:23 Waiting for dst bucket item count to get to: 20 curr count: 18
2021/12/10 03:00:23 src & dst bucket item count matched up. src bucket count: 20 dst bucket count: 20
2021/12/10 03:00:23 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 03:00:23 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 03:00:28 Waiting for app: TestCacheGet to get un-deployed
2021/12/10 03:00:33 Waiting for app: TestCacheGet to get un-deployed
2021/12/10 03:00:38 Waiting for app: TestCacheGet to get un-deployed
2021/12/10 03:00:43 Waiting for app: TestCacheGet to get un-deployed
2021/12/10 03:00:43 App: TestCacheGet got un-deployed
2021/12/10 03:00:43 No eventing-con process running
2021/12/10 03:00:43 Function: TestCacheGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCacheGet deleting in the background"
}
2021/12/10 03:00:45 Context: Bucket flush, response code: 200 dump: 
2021/12/10 03:00:55 Waiting for dst bucket item count to get to: 0 curr count: 20
2021/12/10 03:00:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 03:00:57 Context: Bucket flush, response code: 200 dump: 
2021/12/10 03:01:07 Waiting for dst bucket item count to get to: 0 curr count: 20
2021/12/10 03:01:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCacheGet (95.05s)
=== RUN   TestCacheRYOW
2021/12/10 03:01:07 Deploying app: TestCacheRYOW. Item count src bucket: 0 dst bucket: 0
2021/12/10 03:01:08 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCacheRYOW' in metakv",
  "warnings": null
 }
}
2021/12/10 03:01:13 Waiting for app: TestCacheRYOW to get deployed
2021/12/10 03:01:18 Waiting for app: TestCacheRYOW to get deployed
2021/12/10 03:01:23 Waiting for app: TestCacheRYOW to get deployed
2021/12/10 03:01:28 Waiting for app: TestCacheRYOW to get deployed
2021/12/10 03:01:28 App: TestCacheRYOW got deployed
2021/12/10 03:01:28 Starting bucket ops to source bucket
2021/12/10 03:01:33 Waiting for dst bucket item count to get to: 50 curr count: 0
2021/12/10 03:01:38 Waiting for dst bucket item count to get to: 50 curr count: 25
2021/12/10 03:01:38 src & dst bucket item count matched up. src bucket count: 50 dst bucket count: 50
2021/12/10 03:01:38 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 03:01:38 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 03:01:43 Waiting for app: TestCacheRYOW to get un-deployed
2021/12/10 03:01:49 Waiting for app: TestCacheRYOW to get un-deployed
2021/12/10 03:01:54 Waiting for app: TestCacheRYOW to get un-deployed
2021/12/10 03:01:59 Waiting for app: TestCacheRYOW to get un-deployed
2021/12/10 03:01:59 App: TestCacheRYOW got un-deployed
2021/12/10 03:01:59 No eventing-con process running
2021/12/10 03:01:59 Function: TestCacheRYOW delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCacheRYOW deleting in the background"
}
2021/12/10 03:02:00 Context: Bucket flush, response code: 200 dump: 
2021/12/10 03:02:10 Waiting for dst bucket item count to get to: 0 curr count: 38
2021/12/10 03:02:15 Waiting for dst bucket item count to get to: 0 curr count: 38
2021/12/10 03:02:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 03:02:17 Context: Bucket flush, response code: 200 dump: 
2021/12/10 03:02:27 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/10 03:02:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCacheRYOW (80.15s)
=== RUN   TestCacheExpiry
2021/12/10 03:02:27 Deploying app: TestCacheExpiry. Item count src bucket: 0 dst bucket: 0
2021/12/10 03:02:28 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCacheExpiry' in metakv",
  "warnings": null
 }
}
2021/12/10 03:02:33 Waiting for app: TestCacheExpiry to get deployed
2021/12/10 03:02:38 Waiting for app: TestCacheExpiry to get deployed
2021/12/10 03:02:43 Waiting for app: TestCacheExpiry to get deployed
2021/12/10 03:02:48 Waiting for app: TestCacheExpiry to get deployed
2021/12/10 03:02:48 App: TestCacheExpiry got deployed
2021/12/10 03:02:48 Starting bucket ops to source bucket
2021/12/10 03:02:54 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/10 03:02:59 Waiting for dst bucket item count to get to: 100 curr count: 3
2021/12/10 03:03:04 Waiting for dst bucket item count to get to: 100 curr count: 42
2021/12/10 03:03:09 Waiting for dst bucket item count to get to: 100 curr count: 86
2021/12/10 03:03:14 Waiting for dst bucket item count to get to: 100 curr count: 97
2021/12/10 03:03:14 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/10 03:03:14 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 03:03:14 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 03:03:19 Waiting for app: TestCacheExpiry to get un-deployed
2021/12/10 03:03:24 Waiting for app: TestCacheExpiry to get un-deployed
2021/12/10 03:03:29 Waiting for app: TestCacheExpiry to get un-deployed
2021/12/10 03:03:34 Waiting for app: TestCacheExpiry to get un-deployed
2021/12/10 03:03:34 App: TestCacheExpiry got un-deployed
2021/12/10 03:03:34 No eventing-con process running
2021/12/10 03:03:34 Function: TestCacheExpiry delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCacheExpiry deleting in the background"
}
2021/12/10 03:03:36 Context: Bucket flush, response code: 200 dump: 
2021/12/10 03:03:46 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/10 03:03:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 03:03:47 Context: Bucket flush, response code: 200 dump: 
2021/12/10 03:03:57 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/10 03:03:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCacheExpiry (90.22s)
=== RUN   TestCacheExpiryCustom
2021/12/10 03:03:57 Deploying app: TestCacheExpiryCustom. Item count src bucket: 0 dst bucket: 0
2021/12/10 03:03:59 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCacheExpiryCustom' in metakv",
  "warnings": null
 }
}
2021/12/10 03:04:04 Waiting for app: TestCacheExpiryCustom to get deployed
2021/12/10 03:04:09 Waiting for app: TestCacheExpiryCustom to get deployed
2021/12/10 03:04:14 Waiting for app: TestCacheExpiryCustom to get deployed
2021/12/10 03:04:19 Waiting for app: TestCacheExpiryCustom to get deployed
2021/12/10 03:04:19 App: TestCacheExpiryCustom got deployed
2021/12/10 03:04:19 Starting bucket ops to source bucket
2021/12/10 03:04:24 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/10 03:04:29 Waiting for dst bucket item count to get to: 100 curr count: 3
2021/12/10 03:04:34 Waiting for dst bucket item count to get to: 100 curr count: 42
2021/12/10 03:04:39 Waiting for dst bucket item count to get to: 100 curr count: 86
2021/12/10 03:04:44 Waiting for dst bucket item count to get to: 100 curr count: 97
2021/12/10 03:04:44 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/10 03:04:44 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 03:04:44 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 03:04:49 Waiting for app: TestCacheExpiryCustom to get un-deployed
2021/12/10 03:04:54 Waiting for app: TestCacheExpiryCustom to get un-deployed
2021/12/10 03:04:59 Waiting for app: TestCacheExpiryCustom to get un-deployed
2021/12/10 03:05:04 Waiting for app: TestCacheExpiryCustom to get un-deployed
2021/12/10 03:05:04 App: TestCacheExpiryCustom got un-deployed
2021/12/10 03:05:04 No eventing-con process running
2021/12/10 03:05:04 Function: TestCacheExpiryCustom delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCacheExpiryCustom deleting in the background"
}
2021/12/10 03:05:06 Context: Bucket flush, response code: 200 dump: 
2021/12/10 03:05:16 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/10 03:05:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 03:05:17 Context: Bucket flush, response code: 200 dump: 
2021/12/10 03:05:27 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/10 03:05:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCacheExpiryCustom (90.16s)
=== RUN   TestCacheOverflowCustom
2021/12/10 03:05:27 Deploying app: TestCacheOverflowCustom. Item count src bucket: 0 dst bucket: 0
2021/12/10 03:05:29 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCacheOverflowCustom' in metakv",
  "warnings": null
 }
}
2021/12/10 03:05:34 Waiting for app: TestCacheOverflowCustom to get deployed
2021/12/10 03:05:39 Waiting for app: TestCacheOverflowCustom to get deployed
2021/12/10 03:05:44 Waiting for app: TestCacheOverflowCustom to get deployed
2021/12/10 03:05:49 Waiting for app: TestCacheOverflowCustom to get deployed
2021/12/10 03:05:49 App: TestCacheOverflowCustom got deployed
2021/12/10 03:05:49 Starting bucket ops to source bucket
2021/12/10 03:05:54 Waiting for dst bucket item count to get to: 35 curr count: 0
2021/12/10 03:05:59 Waiting for dst bucket item count to get to: 35 curr count: 0
2021/12/10 03:06:04 Waiting for dst bucket item count to get to: 35 curr count: 9
2021/12/10 03:06:09 Waiting for dst bucket item count to get to: 35 curr count: 20
2021/12/10 03:06:14 Waiting for dst bucket item count to get to: 35 curr count: 28
2021/12/10 03:06:19 Waiting for dst bucket item count to get to: 35 curr count: 34
2021/12/10 03:06:24 Waiting for dst bucket item count to get to: 35 curr count: 34
2021/12/10 03:06:24 src & dst bucket item count matched up. src bucket count: 35 dst bucket count: 35
2021/12/10 03:06:24 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 03:06:24 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 03:06:29 Waiting for app: TestCacheOverflowCustom to get un-deployed
2021/12/10 03:06:34 Waiting for app: TestCacheOverflowCustom to get un-deployed
2021/12/10 03:06:39 Waiting for app: TestCacheOverflowCustom to get un-deployed
2021/12/10 03:06:44 Waiting for app: TestCacheOverflowCustom to get un-deployed
2021/12/10 03:06:44 App: TestCacheOverflowCustom got un-deployed
2021/12/10 03:06:44 No eventing-con process running
2021/12/10 03:06:44 Function: TestCacheOverflowCustom delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCacheOverflowCustom deleting in the background"
}
2021/12/10 03:06:46 Context: Bucket flush, response code: 200 dump: 
2021/12/10 03:06:56 Waiting for dst bucket item count to get to: 0 curr count: 5
2021/12/10 03:06:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 03:06:58 Context: Bucket flush, response code: 200 dump: 
2021/12/10 03:07:08 Waiting for dst bucket item count to get to: 0 curr count: 18
2021/12/10 03:07:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCacheOverflowCustom (100.18s)
=== RUN   TestStrictMode
2021/12/10 03:07:08 Deploying app: TestStrictMode. Item count src bucket: 0 dst bucket: 0
2021/12/10 03:07:09 Context: Post to main store, response code: 422 dump: {
 "name": "ERR_HANDLER_COMPILATION",
 "code": 27,
 "description": "Function compilation failed",
 "attributes": null,
 "runtime_info": {
  "code": 27,
  "info": {
   "area": "handlerCode",
   "column_number": 63,
   "compile_success": false,
   "description": "        destination[meta.id] = 00000000000000000000000000000101;\n                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nSyntaxError: Octal literals are not allowed in strict mode.\n",
   "index": 132,
   "language": "JavaScript",
   "line_number": 4
  }
 }
}
--- PASS: TestStrictMode (1.06s)
=== RUN   TestDataTypes
2021/12/10 03:07:09 Deploying app: TestDataTypes. Item count src bucket: 0 dst bucket: 0
2021/12/10 03:07:10 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDataTypes' in metakv",
  "warnings": null
 }
}
2021/12/10 03:07:15 Waiting for app: TestDataTypes to get deployed
2021/12/10 03:07:20 Waiting for app: TestDataTypes to get deployed
2021/12/10 03:07:25 Waiting for app: TestDataTypes to get deployed
2021/12/10 03:07:30 Waiting for app: TestDataTypes to get deployed
2021/12/10 03:07:30 App: TestDataTypes got deployed
2021/12/10 03:07:30 Starting bucket ops to source bucket
2021/12/10 03:07:35 Waiting for dst bucket item count to get to: 18 curr count: 0
2021/12/10 03:07:40 Waiting for dst bucket item count to get to: 18 curr count: 0
2021/12/10 03:07:45 Waiting for dst bucket item count to get to: 18 curr count: 8
2021/12/10 03:07:45 src & dst bucket item count matched up. src bucket count: 18 dst bucket count: 18
2021/12/10 03:07:45 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 03:07:45 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 03:07:50 Waiting for app: TestDataTypes to get un-deployed
2021/12/10 03:07:55 Waiting for app: TestDataTypes to get un-deployed
2021/12/10 03:08:00 Waiting for app: TestDataTypes to get un-deployed
2021/12/10 03:08:05 Waiting for app: TestDataTypes to get un-deployed
2021/12/10 03:08:05 App: TestDataTypes got un-deployed
2021/12/10 03:08:05 No eventing-con process running
2021/12/10 03:08:05 Function: TestDataTypes delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDataTypes deleting in the background"
}
2021/12/10 03:08:07 Context: Bucket flush, response code: 200 dump: 
2021/12/10 03:08:17 Waiting for dst bucket item count to get to: 0 curr count: 1
2021/12/10 03:08:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 03:08:19 Context: Bucket flush, response code: 200 dump: 
2021/12/10 03:08:29 Waiting for dst bucket item count to get to: 0 curr count: 17
2021/12/10 03:08:34 Waiting for dst bucket item count to get to: 0 curr count: 7
2021/12/10 03:08:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDataTypes (84.96s)
=== RUN   TestEnoentGet
2021/12/10 03:08:34 Deploying app: TestEnoentGet. Item count src bucket: 0 dst bucket: 0
2021/12/10 03:08:35 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentGet' in metakv",
  "warnings": null
 }
}
2021/12/10 03:08:40 Waiting for app: TestEnoentGet to get deployed
2021/12/10 03:08:45 Waiting for app: TestEnoentGet to get deployed
2021/12/10 03:08:50 Waiting for app: TestEnoentGet to get deployed
2021/12/10 03:08:55 Waiting for app: TestEnoentGet to get deployed
2021/12/10 03:08:55 App: TestEnoentGet got deployed
2021/12/10 03:08:55 Starting bucket ops to source bucket
2021/12/10 03:09:00 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/10 03:09:05 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/10 03:09:05 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/10 03:09:05 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 03:09:05 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 03:09:10 Waiting for app: TestEnoentGet to get un-deployed
2021/12/10 03:09:15 Waiting for app: TestEnoentGet to get un-deployed
2021/12/10 03:09:20 Waiting for app: TestEnoentGet to get un-deployed
2021/12/10 03:09:25 Waiting for app: TestEnoentGet to get un-deployed
2021/12/10 03:09:25 App: TestEnoentGet got un-deployed
2021/12/10 03:09:25 No eventing-con process running
2021/12/10 03:09:25 Function: TestEnoentGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentGet deleting in the background"
}
2021/12/10 03:09:27 Context: Bucket flush, response code: 200 dump: 
2021/12/10 03:09:37 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/10 03:09:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 03:09:39 Context: Bucket flush, response code: 200 dump: 
2021/12/10 03:09:49 Waiting for dst bucket item count to get to: 0 curr count: 93
2021/12/10 03:09:54 Waiting for dst bucket item count to get to: 0 curr count: 43
2021/12/10 03:09:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 03:09:54 Deploying app: TestEnoentGet. Item count src bucket: 0 dst bucket: 0
2021/12/10 03:09:55 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentGet' in metakv",
  "warnings": null
 }
}
2021/12/10 03:10:00 Waiting for app: TestEnoentGet to get deployed
2021/12/10 03:10:05 Waiting for app: TestEnoentGet to get deployed
2021/12/10 03:10:10 Waiting for app: TestEnoentGet to get deployed
2021/12/10 03:10:15 Waiting for app: TestEnoentGet to get deployed
2021/12/10 03:10:15 App: TestEnoentGet got deployed
2021/12/10 03:10:15 Starting bucket ops to source bucket
2021/12/10 03:10:20 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/10 03:10:25 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/10 03:10:25 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/10 03:10:25 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 03:10:25 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 03:10:30 Waiting for app: TestEnoentGet to get un-deployed
2021/12/10 03:10:35 Waiting for app: TestEnoentGet to get un-deployed
2021/12/10 03:10:40 Waiting for app: TestEnoentGet to get un-deployed
2021/12/10 03:10:45 Waiting for app: TestEnoentGet to get un-deployed
2021/12/10 03:10:45 App: TestEnoentGet got un-deployed
2021/12/10 03:10:45 No eventing-con process running
2021/12/10 03:10:45 Function: TestEnoentGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentGet deleting in the background"
}
2021/12/10 03:10:47 Context: Bucket flush, response code: 200 dump: 
2021/12/10 03:10:57 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/10 03:10:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 03:10:59 Context: Bucket flush, response code: 200 dump: 
2021/12/10 03:11:09 Waiting for dst bucket item count to get to: 0 curr count: 96
2021/12/10 03:11:14 Waiting for dst bucket item count to get to: 0 curr count: 46
2021/12/10 03:11:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEnoentGet (160.10s)
=== RUN   TestEnoentDelete
2021/12/10 03:11:14 Deploying app: TestEnoentDelete. Item count src bucket: 0 dst bucket: 0
2021/12/10 03:11:15 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentDelete' in metakv",
  "warnings": null
 }
}
2021/12/10 03:11:20 Waiting for app: TestEnoentDelete to get deployed
2021/12/10 03:11:25 Waiting for app: TestEnoentDelete to get deployed
2021/12/10 03:11:30 Waiting for app: TestEnoentDelete to get deployed
2021/12/10 03:11:35 Waiting for app: TestEnoentDelete to get deployed
2021/12/10 03:11:35 App: TestEnoentDelete got deployed
2021/12/10 03:11:35 Starting bucket ops to source bucket
2021/12/10 03:11:40 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/10 03:11:45 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/10 03:11:45 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/10 03:11:45 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 03:11:45 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 03:11:50 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/10 03:11:55 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/10 03:12:00 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/10 03:12:05 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/10 03:12:05 App: TestEnoentDelete got un-deployed
2021/12/10 03:12:05 No eventing-con process running
2021/12/10 03:12:05 Function: TestEnoentDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentDelete deleting in the background"
}
2021/12/10 03:12:07 Context: Bucket flush, response code: 200 dump: 
2021/12/10 03:12:17 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/10 03:12:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 03:12:19 Context: Bucket flush, response code: 200 dump: 
2021/12/10 03:12:29 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/10 03:12:34 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/10 03:12:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 03:12:34 Deploying app: TestEnoentDelete. Item count src bucket: 0 dst bucket: 0
2021/12/10 03:12:35 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentDelete' in metakv",
  "warnings": null
 }
}
2021/12/10 03:12:40 Waiting for app: TestEnoentDelete to get deployed
2021/12/10 03:12:45 Waiting for app: TestEnoentDelete to get deployed
2021/12/10 03:12:50 Waiting for app: TestEnoentDelete to get deployed
2021/12/10 03:12:55 Waiting for app: TestEnoentDelete to get deployed
2021/12/10 03:12:55 App: TestEnoentDelete got deployed
2021/12/10 03:12:55 Starting bucket ops to source bucket
2021/12/10 03:13:00 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/10 03:13:00 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/10 03:13:00 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 03:13:00 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 03:13:05 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/10 03:13:10 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/10 03:13:15 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/10 03:13:20 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/10 03:13:21 App: TestEnoentDelete got un-deployed
2021/12/10 03:13:21 No eventing-con process running
2021/12/10 03:13:21 Function: TestEnoentDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentDelete deleting in the background"
}
2021/12/10 03:13:22 Context: Bucket flush, response code: 200 dump: 
2021/12/10 03:13:32 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/10 03:13:37 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/10 03:13:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 03:13:39 Context: Bucket flush, response code: 200 dump: 
2021/12/10 03:13:49 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/10 03:13:54 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/10 03:13:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEnoentDelete (160.28s)
=== RUN   TestError
2021/12/10 03:14:04 Waiting for app: TestError to get un-deployed
2021/12/10 03:14:04 App: TestError got un-deployed
2021/12/10 03:14:04 No eventing-con process running
2021/12/10 03:14:04 Function: TestError delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestError not found"
 }
}
2021/12/10 03:14:06 Context: Bucket flush, response code: 200 dump: 
2021/12/10 03:14:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 03:14:13 Context: Bucket flush, response code: 200 dump: 
2021/12/10 03:14:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 03:14:18 Deploying app: TestError. Item count src bucket: 0 dst bucket: 0
2021/12/10 03:14:19 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestError' in metakv",
  "warnings": null
 }
}
2021/12/10 03:14:24 Waiting for app: TestError to get deployed
2021/12/10 03:14:29 Waiting for app: TestError to get deployed
2021/12/10 03:14:34 Waiting for app: TestError to get deployed
2021/12/10 03:14:39 Waiting for app: TestError to get deployed
2021/12/10 03:14:39 App: TestError got deployed
2021/12/10 03:14:39 Starting bucket ops to source bucket
2021/12/10 03:14:44 Waiting for dst bucket item count to get to: 300 curr count: 0
2021/12/10 03:14:49 Waiting for dst bucket item count to get to: 300 curr count: 144
2021/12/10 03:14:49 src & dst bucket item count matched up. src bucket count: 300 dst bucket count: 300
2021/12/10 03:14:49 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 03:14:49 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 03:14:54 Waiting for app: TestError to get un-deployed
2021/12/10 03:14:59 Waiting for app: TestError to get un-deployed
2021/12/10 03:15:04 Waiting for app: TestError to get un-deployed
2021/12/10 03:15:09 Waiting for app: TestError to get un-deployed
2021/12/10 03:15:09 App: TestError got un-deployed
2021/12/10 03:15:09 No eventing-con process running
2021/12/10 03:15:09 Function: TestError delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestError deleting in the background"
}
2021/12/10 03:15:11 Context: Bucket flush, response code: 200 dump: 
2021/12/10 03:15:21 Waiting for dst bucket item count to get to: 0 curr count: 98
2021/12/10 03:15:26 Waiting for dst bucket item count to get to: 0 curr count: 98
2021/12/10 03:15:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 03:15:28 Context: Bucket flush, response code: 200 dump: 
2021/12/10 03:15:38 Waiting for dst bucket item count to get to: 0 curr count: 147
2021/12/10 03:15:43 Waiting for dst bucket item count to get to: 0 curr count: 3
2021/12/10 03:15:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestError (108.62s)
=== RUN   TestCRLF
2021/12/10 03:15:53 Waiting for app: TestCRLF to get un-deployed
2021/12/10 03:15:53 App: TestCRLF got un-deployed
2021/12/10 03:15:53 No eventing-con process running
2021/12/10 03:15:53 Function: TestCRLF delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestCRLF not found"
 }
}
2021/12/10 03:15:54 Context: Bucket flush, response code: 200 dump: 
2021/12/10 03:15:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 03:16:01 Context: Bucket flush, response code: 200 dump: 
2021/12/10 03:16:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 03:16:06 Deploying app: TestCRLF. Item count src bucket: 0 dst bucket: 0
2021/12/10 03:16:07 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCRLF' in metakv",
  "warnings": null
 }
}
2021/12/10 03:16:12 Waiting for app: TestCRLF to get deployed
2021/12/10 03:16:17 Waiting for app: TestCRLF to get deployed
2021/12/10 03:16:22 Waiting for app: TestCRLF to get deployed
2021/12/10 03:16:27 Waiting for app: TestCRLF to get deployed
2021/12/10 03:16:27 App: TestCRLF got deployed
2021/12/10 03:16:27 Starting bucket ops to source bucket
2021/12/10 03:16:33 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/10 03:16:38 Waiting for dst bucket item count to get to: 100 curr count: 55
2021/12/10 03:16:43 Waiting for dst bucket item count to get to: 100 curr count: 55
2021/12/10 03:16:43 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/10 03:16:43 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 03:16:43 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 03:16:48 Waiting for app: TestCRLF to get un-deployed
2021/12/10 03:16:53 Waiting for app: TestCRLF to get un-deployed
2021/12/10 03:16:58 Waiting for app: TestCRLF to get un-deployed
2021/12/10 03:17:03 Waiting for app: TestCRLF to get un-deployed
2021/12/10 03:17:03 App: TestCRLF got un-deployed
2021/12/10 03:17:03 No eventing-con process running
2021/12/10 03:17:03 Function: TestCRLF delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCRLF deleting in the background"
}
2021/12/10 03:17:04 Context: Bucket flush, response code: 200 dump: 
2021/12/10 03:17:14 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/10 03:17:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 03:17:16 Context: Bucket flush, response code: 200 dump: 
2021/12/10 03:17:26 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/10 03:17:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCRLF (103.56s)
=== RUN   TestImportExport
2021/12/10 03:17:36 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/10 03:17:36 App: TestImportExport_n1ql_insert_on_update got un-deployed
2021/12/10 03:17:36 No eventing-con process running
2021/12/10 03:17:36 Function: TestImportExport_n1ql_insert_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestImportExport_n1ql_insert_on_update not found"
 }
}
2021/12/10 03:17:38 Context: Bucket flush, response code: 200 dump: 
2021/12/10 03:17:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 03:17:45 Context: Bucket flush, response code: 200 dump: 
2021/12/10 03:17:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 03:17:55 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/10 03:17:55 App: TestImportExport_bucket_op_on_update got un-deployed
2021/12/10 03:17:55 No eventing-con process running
2021/12/10 03:17:55 Function: TestImportExport_bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestImportExport_bucket_op_on_update not found"
 }
}
2021/12/10 03:17:57 Context: Bucket flush, response code: 200 dump: 
2021/12/10 03:18:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 03:18:03 Context: Bucket flush, response code: 200 dump: 
2021/12/10 03:18:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 03:18:08 Deploying app: TestImportExport_n1ql_insert_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/10 03:18:10 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestImportExport_n1ql_insert_on_update' in metakv",
  "warnings": null
 }
}
2021/12/10 03:18:10 Deploying app: TestImportExport_bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/10 03:18:11 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestImportExport_bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/10 03:18:16 Waiting for app: TestImportExport_n1ql_insert_on_update to get deployed
2021/12/10 03:18:21 Waiting for app: TestImportExport_n1ql_insert_on_update to get deployed
2021/12/10 03:18:26 Waiting for app: TestImportExport_n1ql_insert_on_update to get deployed
2021/12/10 03:18:31 Waiting for app: TestImportExport_n1ql_insert_on_update to get deployed
2021/12/10 03:18:31 App: TestImportExport_n1ql_insert_on_update got deployed
2021/12/10 03:18:36 Waiting for app: TestImportExport_bucket_op_on_update to get deployed
2021/12/10 03:18:41 Waiting for app: TestImportExport_bucket_op_on_update to get deployed
2021/12/10 03:18:46 Waiting for app: TestImportExport_bucket_op_on_update to get deployed
2021/12/10 03:18:46 App: TestImportExport_bucket_op_on_update got deployed
2021/12/10 03:18:46 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 03:18:51 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/10 03:18:56 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/10 03:19:01 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/10 03:19:06 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/10 03:19:06 App: TestImportExport_n1ql_insert_on_update got un-deployed
2021/12/10 03:19:06 No eventing-con process running
2021/12/10 03:19:06 Function: TestImportExport_n1ql_insert_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestImportExport_n1ql_insert_on_update deleting in the background"
}
2021/12/10 03:19:06 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 03:19:11 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/10 03:19:16 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/10 03:19:21 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/10 03:19:26 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/10 03:19:26 App: TestImportExport_bucket_op_on_update got un-deployed
2021/12/10 03:19:26 No eventing-con process running
2021/12/10 03:19:26 Function: TestImportExport_bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestImportExport_bucket_op_on_update deleting in the background"
}
2021/12/10 03:19:44 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/10 03:19:44 App: TestImportExport_bucket_op_on_update got un-deployed
2021/12/10 03:19:44 No eventing-con process running
2021/12/10 03:19:44 Function: TestImportExport_bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestImportExport_bucket_op_on_update deleting in the background"
}
2021/12/10 03:19:45 Context: Bucket flush, response code: 200 dump: 
2021/12/10 03:19:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 03:19:52 Context: Bucket flush, response code: 200 dump: 
2021/12/10 03:19:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 03:20:02 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/10 03:20:02 App: TestImportExport_n1ql_insert_on_update got un-deployed
2021/12/10 03:20:02 No eventing-con process running
2021/12/10 03:20:02 Function: TestImportExport_n1ql_insert_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestImportExport_n1ql_insert_on_update deleting in the background"
}
2021/12/10 03:20:04 Context: Bucket flush, response code: 200 dump: 
2021/12/10 03:20:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 03:20:11 Context: Bucket flush, response code: 200 dump: 
2021/12/10 03:20:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestImportExport (169.49s)
=== RUN   TestDeployUndeployLoopNonDefaultSettings
2021/12/10 03:20:26 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/10 03:20:26 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/10 03:20:26 No eventing-con process running
2021/12/10 03:20:26 Function: TestDeployUndeployLoopNonDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDeployUndeployLoopNonDefaultSettings not found"
 }
}
2021/12/10 03:20:28 Context: Bucket flush, response code: 200 dump: 
2021/12/10 03:20:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 03:20:34 Context: Bucket flush, response code: 200 dump: 
2021/12/10 03:20:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 03:20:39 Deploying app: TestDeployUndeployLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/10 03:20:40 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/10 03:20:40 Starting bucket ops to source bucket
2021/12/10 03:20:48 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 03:20:53 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 03:20:58 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 03:21:04 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 03:21:04 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/10 03:21:09 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get deployed
2021/12/10 03:21:09 App: TestDeployUndeployLoopNonDefaultSettings got deployed
2021/12/10 03:21:09 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 03:21:09 Undeploying app: bucket_op_on_update
2021/12/10 03:21:09 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 03:21:14 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/10 03:21:19 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/10 03:21:24 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/10 03:21:29 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/10 03:21:29 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/10 03:21:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 03:21:29 No eventing-con process running
2021/12/10 03:21:30 Context: Bucket flush, response code: 200 dump: 
2021/12/10 03:21:38 Context: Bucket flush, response code: 200 dump: 
2021/12/10 03:22:13 Deploying app: TestDeployUndeployLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/10 03:22:14 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/10 03:22:14 Starting bucket ops to source bucket
2021/12/10 03:22:22 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 03:22:27 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 03:22:32 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 03:22:37 Waiting for dst bucket item count to get to: 5000 curr count: 2044
2021/12/10 03:22:42 Waiting for dst bucket item count to get to: 5000 curr count: 4544
2021/12/10 03:22:42 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/10 03:22:47 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get deployed
2021/12/10 03:22:47 App: TestDeployUndeployLoopNonDefaultSettings got deployed
2021/12/10 03:22:47 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 03:22:47 Undeploying app: bucket_op_on_update
2021/12/10 03:22:47 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 03:22:52 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/10 03:22:57 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/10 03:23:02 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/10 03:23:07 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/10 03:23:07 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/10 03:23:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 03:23:07 No eventing-con process running
2021/12/10 03:23:09 Context: Bucket flush, response code: 200 dump: 
2021/12/10 03:23:16 Context: Bucket flush, response code: 200 dump: 
2021/12/10 03:23:51 Deploying app: TestDeployUndeployLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/10 03:23:52 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/10 03:23:52 Starting bucket ops to source bucket
2021/12/10 03:24:00 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 03:24:05 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 03:24:10 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 03:24:15 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 03:24:15 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/10 03:24:20 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get deployed
2021/12/10 03:24:20 App: TestDeployUndeployLoopNonDefaultSettings got deployed
2021/12/10 03:24:20 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 03:24:20 Undeploying app: bucket_op_on_update
2021/12/10 03:24:20 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 03:24:25 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/10 03:24:30 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/10 03:24:35 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/10 03:24:40 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/10 03:24:40 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/10 03:24:40 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 03:24:40 No eventing-con process running
2021/12/10 03:24:42 Context: Bucket flush, response code: 200 dump: 
2021/12/10 03:24:49 Context: Bucket flush, response code: 200 dump: 
2021/12/10 03:25:24 Deploying app: TestDeployUndeployLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/10 03:25:25 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/10 03:25:25 Starting bucket ops to source bucket
2021/12/10 03:25:33 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 03:25:38 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 03:25:43 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 03:25:48 Waiting for dst bucket item count to get to: 5000 curr count: 2500
2021/12/10 03:25:53 Waiting for dst bucket item count to get to: 5000 curr count: 2500
2021/12/10 03:25:53 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/10 03:25:58 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get deployed
2021/12/10 03:25:58 App: TestDeployUndeployLoopNonDefaultSettings got deployed
2021/12/10 03:25:58 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 03:25:58 Undeploying app: bucket_op_on_update
2021/12/10 03:25:58 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 03:26:03 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/10 03:26:08 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/10 03:26:13 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/10 03:26:18 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/10 03:26:18 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/10 03:26:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 03:26:18 No eventing-con process running
2021/12/10 03:26:20 Context: Bucket flush, response code: 200 dump: 
2021/12/10 03:26:27 Context: Bucket flush, response code: 200 dump: 
2021/12/10 03:27:02 Deploying app: TestDeployUndeployLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/10 03:27:04 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/10 03:27:04 Starting bucket ops to source bucket
2021/12/10 03:27:12 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 03:27:17 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 03:27:22 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 03:27:27 Waiting for dst bucket item count to get to: 5000 curr count: 2445
2021/12/10 03:27:32 Waiting for dst bucket item count to get to: 5000 curr count: 4945
2021/12/10 03:27:32 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/10 03:27:37 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get deployed
2021/12/10 03:27:37 App: TestDeployUndeployLoopNonDefaultSettings got deployed
2021/12/10 03:27:37 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 03:27:37 Undeploying app: bucket_op_on_update
2021/12/10 03:27:37 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 03:27:42 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/10 03:27:47 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/10 03:27:52 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/10 03:27:57 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/10 03:27:57 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/10 03:27:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 03:27:57 No eventing-con process running
2021/12/10 03:27:58 Context: Bucket flush, response code: 200 dump: 
2021/12/10 03:28:06 Context: Bucket flush, response code: 200 dump: 
2021/12/10 03:28:41 Function: TestDeployUndeployLoopNonDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDeployUndeployLoopNonDefaultSettings deleting in the background"
}
--- PASS: TestDeployUndeployLoopNonDefaultSettings (505.05s)
=== RUN   TestOnUpdateN1QLOp
2021/12/10 03:28:51 Waiting for app: TestOnUpdateN1QLOp to get un-deployed
2021/12/10 03:28:51 App: TestOnUpdateN1QLOp got un-deployed
2021/12/10 03:28:51 No eventing-con process running
2021/12/10 03:28:51 Function: TestOnUpdateN1QLOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateN1QLOp not found"
 }
}
2021/12/10 03:28:53 Context: Bucket flush, response code: 200 dump: 
2021/12/10 03:28:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 03:28:59 Context: Bucket flush, response code: 200 dump: 
2021/12/10 03:29:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 03:29:04 Deploying app: TestOnUpdateN1QLOp. Item count src bucket: 0 dst bucket: 0
2021/12/10 03:29:05 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateN1QLOp' in metakv",
  "warnings": null
 }
}
2021/12/10 03:29:05 Starting bucket ops to source bucket
2021/12/10 03:29:13 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 03:29:18 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 03:29:23 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 03:29:28 Waiting for dst bucket item count to get to: 5000 curr count: 1194
2021/12/10 03:29:33 Waiting for dst bucket item count to get to: 5000 curr count: 1194
2021/12/10 03:29:33 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/10 03:29:33 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 03:29:34 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 03:29:39 Waiting for app: TestOnUpdateN1QLOp to get un-deployed
2021/12/10 03:29:44 Waiting for app: TestOnUpdateN1QLOp to get un-deployed
2021/12/10 03:29:49 Waiting for app: TestOnUpdateN1QLOp to get un-deployed
2021/12/10 03:29:54 Waiting for app: TestOnUpdateN1QLOp to get un-deployed
2021/12/10 03:29:54 App: TestOnUpdateN1QLOp got un-deployed
2021/12/10 03:29:54 No eventing-con process running
2021/12/10 03:29:54 Function: TestOnUpdateN1QLOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateN1QLOp deleting in the background"
}
2021/12/10 03:29:55 Context: Bucket flush, response code: 200 dump: 
2021/12/10 03:30:05 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/10 03:30:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 03:30:07 Context: Bucket flush, response code: 200 dump: 
2021/12/10 03:30:17 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/10 03:30:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateN1QLOp (96.42s)
=== RUN   TestOnUpdateBucketOpDefaultSettings
2021/12/10 03:30:27 Waiting for app: TestOnUpdateBucketOpDefaultSettings to get un-deployed
2021/12/10 03:30:27 App: TestOnUpdateBucketOpDefaultSettings got un-deployed
2021/12/10 03:30:27 No eventing-con process running
2021/12/10 03:30:27 Function: TestOnUpdateBucketOpDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateBucketOpDefaultSettings not found"
 }
}
2021/12/10 03:30:29 Context: Bucket flush, response code: 200 dump: 
2021/12/10 03:30:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 03:30:36 Context: Bucket flush, response code: 200 dump: 
2021/12/10 03:30:41 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 03:30:41 Deploying app: TestOnUpdateBucketOpDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/10 03:30:42 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateBucketOpDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/10 03:30:42 Starting bucket ops to source bucket
2021/12/10 03:30:50 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 03:30:55 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 03:31:00 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 03:31:05 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 03:31:05 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/10 03:31:05 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 03:31:05 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 03:31:10 Waiting for app: TestOnUpdateBucketOpDefaultSettings to get un-deployed
2021/12/10 03:31:15 Waiting for app: TestOnUpdateBucketOpDefaultSettings to get un-deployed
2021/12/10 03:31:20 Waiting for app: TestOnUpdateBucketOpDefaultSettings to get un-deployed
2021/12/10 03:31:25 Waiting for app: TestOnUpdateBucketOpDefaultSettings to get un-deployed
2021/12/10 03:31:25 App: TestOnUpdateBucketOpDefaultSettings got un-deployed
2021/12/10 03:31:25 No eventing-con process running
2021/12/10 03:31:25 Function: TestOnUpdateBucketOpDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateBucketOpDefaultSettings deleting in the background"
}
2021/12/10 03:31:27 Context: Bucket flush, response code: 200 dump: 
2021/12/10 03:31:37 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/10 03:31:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 03:31:39 Context: Bucket flush, response code: 200 dump: 
2021/12/10 03:31:49 Waiting for dst bucket item count to get to: 0 curr count: 4496
2021/12/10 03:31:54 Waiting for dst bucket item count to get to: 0 curr count: 1996
2021/12/10 03:31:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateBucketOpDefaultSettings (96.44s)
=== RUN   TestOnUpdateBucketOpNonDefaultSettings
2021/12/10 03:32:04 Waiting for app: TestOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/10 03:32:04 App: TestOnUpdateBucketOpNonDefaultSettings got un-deployed
2021/12/10 03:32:04 No eventing-con process running
2021/12/10 03:32:04 Function: TestOnUpdateBucketOpNonDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateBucketOpNonDefaultSettings not found"
 }
}
2021/12/10 03:32:05 Context: Bucket flush, response code: 200 dump: 
2021/12/10 03:32:10 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 03:32:12 Context: Bucket flush, response code: 200 dump: 
2021/12/10 03:32:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 03:32:17 Deploying app: TestOnUpdateBucketOpNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/10 03:32:18 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateBucketOpNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/10 03:32:18 Starting bucket ops to source bucket
2021/12/10 03:32:26 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 03:32:31 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 03:32:36 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 03:32:41 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 03:32:46 Waiting for dst bucket item count to get to: 5000 curr count: 2500
2021/12/10 03:32:46 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/10 03:32:46 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 03:32:46 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 03:32:51 Waiting for app: TestOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/10 03:32:57 Waiting for app: TestOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/10 03:33:02 Waiting for app: TestOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/10 03:33:07 Waiting for app: TestOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/10 03:33:07 App: TestOnUpdateBucketOpNonDefaultSettings got un-deployed
2021/12/10 03:33:07 No eventing-con process running
2021/12/10 03:33:07 Function: TestOnUpdateBucketOpNonDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateBucketOpNonDefaultSettings deleting in the background"
}
2021/12/10 03:33:08 Context: Bucket flush, response code: 200 dump: 
2021/12/10 03:33:18 Waiting for dst bucket item count to get to: 0 curr count: 4046
2021/12/10 03:33:23 Waiting for dst bucket item count to get to: 0 curr count: 1546
2021/12/10 03:33:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 03:33:25 Context: Bucket flush, response code: 200 dump: 
2021/12/10 03:33:35 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/10 03:33:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateBucketOpNonDefaultSettings (101.61s)
=== RUN   TestOnUpdateBucketOpDefaultSettings10K
2021/12/10 03:33:45 Waiting for app: TestOnUpdateBucketOpDefaultSettings10K to get un-deployed
2021/12/10 03:33:45 App: TestOnUpdateBucketOpDefaultSettings10K got un-deployed
2021/12/10 03:33:45 No eventing-con process running
2021/12/10 03:33:45 Function: TestOnUpdateBucketOpDefaultSettings10K delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateBucketOpDefaultSettings10K not found"
 }
}
2021/12/10 03:33:47 Context: Bucket flush, response code: 200 dump: 
2021/12/10 03:33:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 03:33:54 Context: Bucket flush, response code: 200 dump: 
2021/12/10 03:33:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 03:33:59 Deploying app: TestOnUpdateBucketOpDefaultSettings10K. Item count src bucket: 0 dst bucket: 0
2021/12/10 03:34:00 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateBucketOpDefaultSettings10K' in metakv",
  "warnings": null
 }
}
2021/12/10 03:34:00 Starting bucket ops to source bucket
2021/12/10 03:34:08 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 03:34:13 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 03:34:18 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 03:34:23 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 03:34:28 Waiting for dst bucket item count to get to: 5000 curr count: 2500
2021/12/10 03:34:28 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/10 03:34:28 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 03:34:28 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 03:34:33 Waiting for app: TestOnUpdateBucketOpDefaultSettings10K to get un-deployed
2021/12/10 03:34:38 Waiting for app: TestOnUpdateBucketOpDefaultSettings10K to get un-deployed
2021/12/10 03:34:43 Waiting for app: TestOnUpdateBucketOpDefaultSettings10K to get un-deployed
2021/12/10 03:34:48 Waiting for app: TestOnUpdateBucketOpDefaultSettings10K to get un-deployed
2021/12/10 03:34:48 App: TestOnUpdateBucketOpDefaultSettings10K got un-deployed
2021/12/10 03:34:48 No eventing-con process running
2021/12/10 03:34:48 Function: TestOnUpdateBucketOpDefaultSettings10K delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateBucketOpDefaultSettings10K deleting in the background"
}
2021/12/10 03:34:50 Context: Bucket flush, response code: 200 dump: 
2021/12/10 03:35:00 Waiting for dst bucket item count to get to: 0 curr count: 2801
2021/12/10 03:35:05 Waiting for dst bucket item count to get to: 0 curr count: 2801
2021/12/10 03:35:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 03:35:07 Context: Bucket flush, response code: 200 dump: 
2021/12/10 03:35:17 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/10 03:35:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateBucketOpDefaultSettings10K (101.36s)
=== RUN   TestOnUpdateBucketOpDefaultSettings100K
2021/12/10 03:35:27 Waiting for app: TestOnUpdateBucketOpDefaultSettings100K to get un-deployed
2021/12/10 03:35:27 App: TestOnUpdateBucketOpDefaultSettings100K got un-deployed
2021/12/10 03:35:27 No eventing-con process running
2021/12/10 03:35:27 Function: TestOnUpdateBucketOpDefaultSettings100K delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateBucketOpDefaultSettings100K not found"
 }
}
2021/12/10 03:35:28 Context: Bucket flush, response code: 200 dump: 
2021/12/10 03:35:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 03:35:35 Context: Bucket flush, response code: 200 dump: 
2021/12/10 03:35:40 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 03:35:40 Deploying app: TestOnUpdateBucketOpDefaultSettings100K. Item count src bucket: 0 dst bucket: 0
2021/12/10 03:35:42 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateBucketOpDefaultSettings100K' in metakv",
  "warnings": null
 }
}
2021/12/10 03:35:42 Starting bucket ops to source bucket
2021/12/10 03:35:50 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 03:35:55 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 03:36:00 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 03:36:05 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 03:36:05 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/10 03:36:05 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 03:36:05 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 03:36:10 Waiting for app: TestOnUpdateBucketOpDefaultSettings100K to get un-deployed
2021/12/10 03:36:15 Waiting for app: TestOnUpdateBucketOpDefaultSettings100K to get un-deployed
2021/12/10 03:36:20 Waiting for app: TestOnUpdateBucketOpDefaultSettings100K to get un-deployed
2021/12/10 03:36:25 Waiting for app: TestOnUpdateBucketOpDefaultSettings100K to get un-deployed
2021/12/10 03:36:25 App: TestOnUpdateBucketOpDefaultSettings100K got un-deployed
2021/12/10 03:36:25 No eventing-con process running
2021/12/10 03:36:25 Function: TestOnUpdateBucketOpDefaultSettings100K delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateBucketOpDefaultSettings100K deleting in the background"
}
2021/12/10 03:36:27 Context: Bucket flush, response code: 200 dump: 
2021/12/10 03:36:37 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/10 03:36:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 03:36:39 Context: Bucket flush, response code: 200 dump: 
2021/12/10 03:36:49 Waiting for dst bucket item count to get to: 0 curr count: 4652
2021/12/10 03:36:54 Waiting for dst bucket item count to get to: 0 curr count: 2152
2021/12/10 03:36:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateBucketOpDefaultSettings100K (97.09s)
=== RUN   TestOnDeleteBucketOp
2021/12/10 03:37:04 Waiting for app: TestOnDeleteBucketOp to get un-deployed
2021/12/10 03:37:04 App: TestOnDeleteBucketOp got un-deployed
2021/12/10 03:37:04 No eventing-con process running
2021/12/10 03:37:04 Function: TestOnDeleteBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnDeleteBucketOp not found"
 }
}
2021/12/10 03:37:05 Context: Bucket flush, response code: 200 dump: 
2021/12/10 03:37:10 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 03:37:12 Context: Bucket flush, response code: 200 dump: 
2021/12/10 03:37:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 03:37:17 Deploying app: TestOnDeleteBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/10 03:37:18 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnDeleteBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/10 03:37:18 Starting bucket ops to source bucket
2021/12/10 03:37:29 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 03:37:34 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 03:37:39 Waiting for dst bucket item count to get to: 5000 curr count: 4
2021/12/10 03:37:44 Waiting for dst bucket item count to get to: 5000 curr count: 2508
2021/12/10 03:37:44 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/10 03:37:44 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 03:37:44 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 03:37:49 Waiting for app: TestOnDeleteBucketOp to get un-deployed
2021/12/10 03:37:54 Waiting for app: TestOnDeleteBucketOp to get un-deployed
2021/12/10 03:37:59 Waiting for app: TestOnDeleteBucketOp to get un-deployed
2021/12/10 03:38:04 Waiting for app: TestOnDeleteBucketOp to get un-deployed
2021/12/10 03:38:04 App: TestOnDeleteBucketOp got un-deployed
2021/12/10 03:38:04 No eventing-con process running
2021/12/10 03:38:04 Function: TestOnDeleteBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnDeleteBucketOp deleting in the background"
}
2021/12/10 03:38:06 Context: Bucket flush, response code: 200 dump: 
2021/12/10 03:38:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 03:38:13 Context: Bucket flush, response code: 200 dump: 
2021/12/10 03:38:23 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/10 03:38:28 Waiting for dst bucket item count to get to: 0 curr count: 2496
2021/12/10 03:38:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnDeleteBucketOp (94.77s)
=== RUN   TestOnDeleteBucketOp5K
2021/12/10 03:38:38 Waiting for app: TestOnDeleteBucketOp5K to get un-deployed
2021/12/10 03:38:38 App: TestOnDeleteBucketOp5K got un-deployed
2021/12/10 03:38:38 No eventing-con process running
2021/12/10 03:38:38 Function: TestOnDeleteBucketOp5K delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnDeleteBucketOp5K not found"
 }
}
2021/12/10 03:38:40 Context: Bucket flush, response code: 200 dump: 
2021/12/10 03:38:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 03:38:47 Context: Bucket flush, response code: 200 dump: 
2021/12/10 03:38:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 03:38:52 Deploying app: TestOnDeleteBucketOp5K. Item count src bucket: 0 dst bucket: 0
2021/12/10 03:38:53 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnDeleteBucketOp5K' in metakv",
  "warnings": null
 }
}
2021/12/10 03:38:53 Starting bucket ops to source bucket
2021/12/10 03:39:04 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 03:39:09 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 03:39:14 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 03:39:14 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/10 03:39:14 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 03:39:14 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 03:39:19 Waiting for app: TestOnDeleteBucketOp5K to get un-deployed
2021/12/10 03:39:24 Waiting for app: TestOnDeleteBucketOp5K to get un-deployed
2021/12/10 03:39:29 Waiting for app: TestOnDeleteBucketOp5K to get un-deployed
2021/12/10 03:39:34 Waiting for app: TestOnDeleteBucketOp5K to get un-deployed
2021/12/10 03:39:34 App: TestOnDeleteBucketOp5K got un-deployed
2021/12/10 03:39:34 No eventing-con process running
2021/12/10 03:39:34 Function: TestOnDeleteBucketOp5K delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnDeleteBucketOp5K deleting in the background"
}
2021/12/10 03:39:36 Context: Bucket flush, response code: 200 dump: 
2021/12/10 03:39:41 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 03:39:43 Context: Bucket flush, response code: 200 dump: 
2021/12/10 03:39:53 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/10 03:39:58 Waiting for dst bucket item count to get to: 0 curr count: 2496
2021/12/10 03:39:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnDeleteBucketOp5K (89.64s)
=== RUN   TestTimerBucketOp
2021/12/10 03:40:08 Waiting for app: TestTimerBucketOp to get un-deployed
2021/12/10 03:40:08 App: TestTimerBucketOp got un-deployed
2021/12/10 03:40:08 No eventing-con process running
2021/12/10 03:40:08 Function: TestTimerBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestTimerBucketOp not found"
 }
}
2021/12/10 03:40:10 Context: Bucket flush, response code: 200 dump: 
2021/12/10 03:40:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 03:40:17 Context: Bucket flush, response code: 200 dump: 
2021/12/10 03:40:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 03:40:22 Deploying app: TestTimerBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/10 03:40:23 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestTimerBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/10 03:40:23 Starting bucket ops to source bucket
2021/12/10 03:40:31 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 03:40:36 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 03:40:41 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 03:40:46 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 03:40:51 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 03:40:56 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 03:41:01 Waiting for dst bucket item count to get to: 5000 curr count: 1869
2021/12/10 03:41:06 Waiting for dst bucket item count to get to: 5000 curr count: 1869
2021/12/10 03:41:06 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/10 03:41:06 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 03:41:06 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 03:41:11 Waiting for app: TestTimerBucketOp to get un-deployed
2021/12/10 03:41:16 Waiting for app: TestTimerBucketOp to get un-deployed
2021/12/10 03:41:21 Waiting for app: TestTimerBucketOp to get un-deployed
2021/12/10 03:41:26 Waiting for app: TestTimerBucketOp to get un-deployed
2021/12/10 03:41:26 App: TestTimerBucketOp got un-deployed
2021/12/10 03:41:26 No eventing-con process running
2021/12/10 03:41:26 Function: TestTimerBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestTimerBucketOp deleting in the background"
}
2021/12/10 03:41:28 Context: Bucket flush, response code: 200 dump: 
2021/12/10 03:41:38 Waiting for dst bucket item count to get to: 0 curr count: 2644
2021/12/10 03:41:43 Waiting for dst bucket item count to get to: 0 curr count: 144
2021/12/10 03:41:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 03:41:44 Context: Bucket flush, response code: 200 dump: 
2021/12/10 03:41:54 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/10 03:41:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestTimerBucketOp (116.32s)
=== RUN   TestTimerInPastBucketOp
2021/12/10 03:42:04 Waiting for app: TestTimerInPastBucketOp to get un-deployed
2021/12/10 03:42:04 App: TestTimerInPastBucketOp got un-deployed
2021/12/10 03:42:04 No eventing-con process running
2021/12/10 03:42:04 Function: TestTimerInPastBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestTimerInPastBucketOp not found"
 }
}
2021/12/10 03:42:06 Context: Bucket flush, response code: 200 dump: 
2021/12/10 03:42:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 03:42:13 Context: Bucket flush, response code: 200 dump: 
2021/12/10 03:42:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 03:42:18 Deploying app: TestTimerInPastBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/10 03:42:19 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestTimerInPastBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/10 03:42:19 Starting bucket ops to source bucket
2021/12/10 03:42:27 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 03:42:32 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 03:42:37 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 03:42:42 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 03:42:47 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 03:42:52 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 03:42:57 Waiting for dst bucket item count to get to: 5000 curr count: 1688
2021/12/10 03:43:02 Waiting for dst bucket item count to get to: 5000 curr count: 4188
2021/12/10 03:43:02 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/10 03:43:02 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 03:43:02 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 03:43:07 Waiting for app: TestTimerInPastBucketOp to get un-deployed
2021/12/10 03:43:12 Waiting for app: TestTimerInPastBucketOp to get un-deployed
2021/12/10 03:43:17 Waiting for app: TestTimerInPastBucketOp to get un-deployed
2021/12/10 03:43:22 Waiting for app: TestTimerInPastBucketOp to get un-deployed
2021/12/10 03:43:22 App: TestTimerInPastBucketOp got un-deployed
2021/12/10 03:43:22 No eventing-con process running
2021/12/10 03:43:22 Function: TestTimerInPastBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestTimerInPastBucketOp deleting in the background"
}
2021/12/10 03:43:24 Context: Bucket flush, response code: 200 dump: 
2021/12/10 03:43:34 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/10 03:43:39 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/10 03:43:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 03:43:41 Context: Bucket flush, response code: 200 dump: 
2021/12/10 03:43:51 Waiting for dst bucket item count to get to: 0 curr count: 4833
2021/12/10 03:43:56 Waiting for dst bucket item count to get to: 0 curr count: 4833
2021/12/10 03:43:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestTimerInPastBucketOp (121.45s)
=== RUN   TestDeployUndeployLoopTimer
2021/12/10 03:44:06 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/10 03:44:06 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/10 03:44:06 No eventing-con process running
2021/12/10 03:44:06 Function: TestDeployUndeployLoopTimer delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDeployUndeployLoopTimer not found"
 }
}
2021/12/10 03:44:08 Context: Bucket flush, response code: 200 dump: 
2021/12/10 03:44:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 03:44:14 Context: Bucket flush, response code: 200 dump: 
2021/12/10 03:44:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 03:44:19 Deploying app: TestDeployUndeployLoopTimer. Item count src bucket: 0 dst bucket: 0
2021/12/10 03:44:21 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopTimer' in metakv",
  "warnings": null
 }
}
2021/12/10 03:44:21 Starting bucket ops to source bucket
2021/12/10 03:44:28 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 03:44:33 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 03:44:38 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 03:44:43 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 03:44:48 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 03:44:53 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 03:44:58 Waiting for dst bucket item count to get to: 5000 curr count: 3761
2021/12/10 03:45:03 Waiting for dst bucket item count to get to: 5000 curr count: 3761
2021/12/10 03:45:03 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/10 03:45:08 Waiting for app: TestDeployUndeployLoopTimer to get deployed
2021/12/10 03:45:08 App: TestDeployUndeployLoopTimer got deployed
2021/12/10 03:45:08 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 03:45:09 Undeploying app: bucket_op_with_timer
2021/12/10 03:45:09 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 03:45:14 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/10 03:45:19 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/10 03:45:24 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/10 03:45:29 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/10 03:45:29 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/10 03:45:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 03:45:29 No eventing-con process running
2021/12/10 03:45:30 Context: Bucket flush, response code: 200 dump: 
2021/12/10 03:45:38 Context: Bucket flush, response code: 200 dump: 
2021/12/10 03:46:13 Deploying app: TestDeployUndeployLoopTimer. Item count src bucket: 0 dst bucket: 0
2021/12/10 03:46:14 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopTimer' in metakv",
  "warnings": null
 }
}
2021/12/10 03:46:14 Starting bucket ops to source bucket
2021/12/10 03:46:22 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 03:46:27 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 03:46:32 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 03:46:37 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 03:46:42 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 03:46:47 Waiting for dst bucket item count to get to: 5000 curr count: 338
2021/12/10 03:46:52 Waiting for dst bucket item count to get to: 5000 curr count: 576
2021/12/10 03:46:57 Waiting for dst bucket item count to get to: 5000 curr count: 2738
2021/12/10 03:46:57 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/10 03:47:02 Waiting for app: TestDeployUndeployLoopTimer to get deployed
2021/12/10 03:47:02 App: TestDeployUndeployLoopTimer got deployed
2021/12/10 03:47:02 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 03:47:02 Undeploying app: bucket_op_with_timer
2021/12/10 03:47:02 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 03:47:07 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/10 03:47:12 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/10 03:47:17 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/10 03:47:22 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/10 03:47:22 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/10 03:47:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 03:47:22 No eventing-con process running
2021/12/10 03:47:24 Context: Bucket flush, response code: 200 dump: 
2021/12/10 03:47:31 Context: Bucket flush, response code: 200 dump: 
2021/12/10 03:48:06 Deploying app: TestDeployUndeployLoopTimer. Item count src bucket: 0 dst bucket: 0
2021/12/10 03:48:07 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopTimer' in metakv",
  "warnings": null
 }
}
2021/12/10 03:48:07 Starting bucket ops to source bucket
2021/12/10 03:48:15 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 03:48:20 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 03:48:25 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 03:48:30 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 03:48:35 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 03:48:40 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 03:48:45 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 03:48:45 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/10 03:48:50 Waiting for app: TestDeployUndeployLoopTimer to get deployed
2021/12/10 03:48:50 App: TestDeployUndeployLoopTimer got deployed
2021/12/10 03:48:50 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 03:48:50 Undeploying app: bucket_op_with_timer
2021/12/10 03:48:50 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 03:48:55 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/10 03:49:00 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/10 03:49:05 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/10 03:49:10 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/10 03:49:10 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/10 03:49:10 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 03:49:10 No eventing-con process running
2021/12/10 03:49:12 Context: Bucket flush, response code: 200 dump: 
2021/12/10 03:49:19 Context: Bucket flush, response code: 200 dump: 
2021/12/10 03:49:54 Deploying app: TestDeployUndeployLoopTimer. Item count src bucket: 0 dst bucket: 0
2021/12/10 03:49:55 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopTimer' in metakv",
  "warnings": null
 }
}
2021/12/10 03:49:55 Starting bucket ops to source bucket
2021/12/10 03:50:03 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 03:50:08 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 03:50:13 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 03:50:18 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 03:50:23 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 03:50:28 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 03:50:33 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 03:50:33 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/10 03:50:38 Waiting for app: TestDeployUndeployLoopTimer to get deployed
2021/12/10 03:50:38 App: TestDeployUndeployLoopTimer got deployed
2021/12/10 03:50:38 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 03:50:38 Undeploying app: bucket_op_with_timer
2021/12/10 03:50:38 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 03:50:43 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/10 03:50:48 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/10 03:50:53 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/10 03:50:58 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/10 03:50:58 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/10 03:50:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 03:50:58 No eventing-con process running
2021/12/10 03:51:00 Context: Bucket flush, response code: 200 dump: 
2021/12/10 03:51:07 Context: Bucket flush, response code: 200 dump: 
2021/12/10 03:51:42 Deploying app: TestDeployUndeployLoopTimer. Item count src bucket: 0 dst bucket: 0
2021/12/10 03:51:43 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopTimer' in metakv",
  "warnings": null
 }
}
2021/12/10 03:51:43 Starting bucket ops to source bucket
2021/12/10 03:51:51 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 03:51:56 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 03:52:01 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 03:52:06 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 03:52:11 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 03:52:16 Waiting for dst bucket item count to get to: 5000 curr count: 15
2021/12/10 03:52:21 Waiting for dst bucket item count to get to: 5000 curr count: 17
2021/12/10 03:52:26 Waiting for dst bucket item count to get to: 5000 curr count: 2502
2021/12/10 03:52:26 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/10 03:52:31 Waiting for app: TestDeployUndeployLoopTimer to get deployed
2021/12/10 03:52:32 App: TestDeployUndeployLoopTimer got deployed
2021/12/10 03:52:32 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 03:52:32 Undeploying app: bucket_op_with_timer
2021/12/10 03:52:32 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 03:52:37 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/10 03:52:42 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/10 03:52:47 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/10 03:52:52 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/10 03:52:52 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/10 03:52:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 03:52:52 No eventing-con process running
2021/12/10 03:52:53 Context: Bucket flush, response code: 200 dump: 
2021/12/10 03:53:01 Context: Bucket flush, response code: 200 dump: 
2021/12/10 03:53:36 Function: TestDeployUndeployLoopTimer delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDeployUndeployLoopTimer deleting in the background"
}
--- PASS: TestDeployUndeployLoopTimer (579.80s)
=== RUN   TestMultipleHandlers
2021/12/10 03:53:46 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get un-deployed
2021/12/10 03:53:46 App: TestMultipleHandlers_bucket_op_on_update got un-deployed
2021/12/10 03:53:46 No eventing-con process running
2021/12/10 03:53:46 Function: TestMultipleHandlers_bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMultipleHandlers_bucket_op_on_update not found"
 }
}
2021/12/10 03:53:47 Context: Bucket flush, response code: 200 dump: 
2021/12/10 03:53:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 03:53:54 Context: Bucket flush, response code: 200 dump: 
2021/12/10 03:53:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 03:54:04 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get un-deployed
2021/12/10 03:54:04 App: TestMultipleHandlers_bucket_op_on_delete got un-deployed
2021/12/10 03:54:04 No eventing-con process running
2021/12/10 03:54:04 Function: TestMultipleHandlers_bucket_op_on_delete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMultipleHandlers_bucket_op_on_delete not found"
 }
}
2021/12/10 03:54:06 Context: Bucket flush, response code: 200 dump: 
2021/12/10 03:54:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 03:54:13 Context: Bucket flush, response code: 200 dump: 
2021/12/10 03:54:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 03:54:18 Deploying app: TestMultipleHandlers_bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/10 03:54:19 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMultipleHandlers_bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/10 03:54:19 Deploying app: TestMultipleHandlers_bucket_op_on_delete. Item count src bucket: 0 dst bucket: 0
2021/12/10 03:54:20 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMultipleHandlers_bucket_op_on_delete' in metakv",
  "warnings": null
 }
}
2021/12/10 03:54:25 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get deployed
2021/12/10 03:54:30 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get deployed
2021/12/10 03:54:35 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get deployed
2021/12/10 03:54:35 App: TestMultipleHandlers_bucket_op_on_update got deployed
2021/12/10 03:54:40 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get deployed
2021/12/10 03:54:45 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get deployed
2021/12/10 03:54:50 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get deployed
2021/12/10 03:54:55 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get deployed
2021/12/10 03:54:55 App: TestMultipleHandlers_bucket_op_on_delete got deployed
2021/12/10 03:54:55 Starting bucket ops to source bucket
2021/12/10 03:55:05 Waiting for dst bucket item count to get to: 5000 curr count: 952
2021/12/10 03:55:10 Waiting for dst bucket item count to get to: 5000 curr count: 2984
2021/12/10 03:55:15 Waiting for dst bucket item count to get to: 5000 curr count: 4532
2021/12/10 03:55:15 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/10 03:55:15 Starting bucket ops to source bucket
2021/12/10 03:55:30 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/10 03:55:35 Waiting for dst bucket item count to get to: 10000 curr count: 7504
2021/12/10 03:55:35 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/10 03:55:35 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 03:55:36 Function: TestMultipleHandlers_bucket_op_on_update update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/10 03:55:36 Function: TestMultipleHandlers_bucket_op_on_delete update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/10 03:55:36 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 03:55:41 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get un-deployed
2021/12/10 03:55:46 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get un-deployed
2021/12/10 03:55:51 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get un-deployed
2021/12/10 03:55:56 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get un-deployed
2021/12/10 03:55:56 App: TestMultipleHandlers_bucket_op_on_update got un-deployed
2021/12/10 03:55:56 No eventing-con process running
2021/12/10 03:55:56 Function: TestMultipleHandlers_bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMultipleHandlers_bucket_op_on_update deleting in the background"
}
2021/12/10 03:55:58 Context: Bucket flush, response code: 200 dump: 
2021/12/10 03:56:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 03:56:05 Context: Bucket flush, response code: 200 dump: 
2021/12/10 03:56:15 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/10 03:56:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 03:56:15 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 03:56:20 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get un-deployed
2021/12/10 03:56:25 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get un-deployed
2021/12/10 03:56:30 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get un-deployed
2021/12/10 03:56:35 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get un-deployed
2021/12/10 03:56:35 App: TestMultipleHandlers_bucket_op_on_delete got un-deployed
2021/12/10 03:56:35 No eventing-con process running
2021/12/10 03:56:35 Function: TestMultipleHandlers_bucket_op_on_delete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMultipleHandlers_bucket_op_on_delete deleting in the background"
}
2021/12/10 03:56:37 Context: Bucket flush, response code: 200 dump: 
2021/12/10 03:56:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 03:56:44 Context: Bucket flush, response code: 200 dump: 
2021/12/10 03:56:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestMultipleHandlers (192.87s)
=== RUN   TestPauseResumeLoopDefaultSettings
2021/12/10 03:56:59 Waiting for app: TestPauseResumeLoopDefaultSettings to get un-deployed
2021/12/10 03:56:59 App: TestPauseResumeLoopDefaultSettings got un-deployed
2021/12/10 03:56:59 No eventing-con process running
2021/12/10 03:56:59 Function: TestPauseResumeLoopDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestPauseResumeLoopDefaultSettings not found"
 }
}
2021/12/10 03:57:00 Context: Bucket flush, response code: 200 dump: 
2021/12/10 03:57:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 03:57:07 Context: Bucket flush, response code: 200 dump: 
2021/12/10 03:57:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 03:57:12 Deploying app: TestPauseResumeLoopDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/10 03:57:13 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestPauseResumeLoopDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/10 03:57:13 Starting bucket ops to source bucket
2021/12/10 03:57:21 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 03:57:26 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 03:57:31 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 03:57:36 Waiting for dst bucket item count to get to: 5000 curr count: 2500
2021/12/10 03:57:36 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/10 03:57:36 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 03:57:41 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to deployed
2021/12/10 03:57:41 Function: TestPauseResumeLoopDefaultSettings status changed to deployed
2021/12/10 03:57:41 Pausing app: bucket_op_on_update
2021/12/10 03:57:41 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/10 03:57:46 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to paused
2021/12/10 03:57:46 Function: TestPauseResumeLoopDefaultSettings status changed to paused
2021/12/10 03:57:46 Resuming app: bucket_op_on_update
2021/12/10 03:57:47 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/10 03:57:47 Starting bucket ops to source bucket
2021/12/10 03:57:55 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/10 03:58:00 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/10 03:58:05 Waiting for dst bucket item count to get to: 10000 curr count: 7500
2021/12/10 03:58:05 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/10 03:58:05 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 03:58:10 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to deployed
2021/12/10 03:58:10 Function: TestPauseResumeLoopDefaultSettings status changed to deployed
2021/12/10 03:58:10 Pausing app: bucket_op_on_update
2021/12/10 03:58:11 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/10 03:58:16 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to paused
2021/12/10 03:58:16 Function: TestPauseResumeLoopDefaultSettings status changed to paused
2021/12/10 03:58:16 Resuming app: bucket_op_on_update
2021/12/10 03:58:17 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/10 03:58:17 Starting bucket ops to source bucket
2021/12/10 03:58:25 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/10 03:58:30 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/10 03:58:35 Waiting for dst bucket item count to get to: 15000 curr count: 12500
2021/12/10 03:58:35 src & dst bucket item count matched up. src bucket count: 15000 dst bucket count: 15000
2021/12/10 03:58:35 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 03:58:40 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to deployed
2021/12/10 03:58:40 Function: TestPauseResumeLoopDefaultSettings status changed to deployed
2021/12/10 03:58:40 Pausing app: bucket_op_on_update
2021/12/10 03:58:40 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/10 03:58:45 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to paused
2021/12/10 03:58:45 Function: TestPauseResumeLoopDefaultSettings status changed to paused
2021/12/10 03:58:45 Resuming app: bucket_op_on_update
2021/12/10 03:58:46 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/10 03:58:46 Starting bucket ops to source bucket
2021/12/10 03:58:54 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/10 03:58:59 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/10 03:59:04 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/10 03:59:09 Waiting for dst bucket item count to get to: 20000 curr count: 17500
2021/12/10 03:59:09 src & dst bucket item count matched up. src bucket count: 20000 dst bucket count: 20000
2021/12/10 03:59:09 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 03:59:14 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to deployed
2021/12/10 03:59:14 Function: TestPauseResumeLoopDefaultSettings status changed to deployed
2021/12/10 03:59:14 Pausing app: bucket_op_on_update
2021/12/10 03:59:14 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/10 03:59:19 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to paused
2021/12/10 03:59:19 Function: TestPauseResumeLoopDefaultSettings status changed to paused
2021/12/10 03:59:19 Resuming app: bucket_op_on_update
2021/12/10 03:59:20 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/10 03:59:20 Starting bucket ops to source bucket
2021/12/10 03:59:28 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/10 03:59:33 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/10 03:59:38 Waiting for dst bucket item count to get to: 25000 curr count: 21061
2021/12/10 03:59:43 Waiting for dst bucket item count to get to: 25000 curr count: 23561
2021/12/10 03:59:43 src & dst bucket item count matched up. src bucket count: 25000 dst bucket count: 25000
2021/12/10 03:59:43 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 03:59:48 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to deployed
2021/12/10 03:59:48 Function: TestPauseResumeLoopDefaultSettings status changed to deployed
2021/12/10 03:59:48 Pausing app: bucket_op_on_update
2021/12/10 03:59:48 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/10 03:59:53 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to paused
2021/12/10 03:59:53 Function: TestPauseResumeLoopDefaultSettings status changed to paused
2021/12/10 03:59:53 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 03:59:58 Waiting for app: TestPauseResumeLoopDefaultSettings to get un-deployed
2021/12/10 04:00:03 Waiting for app: TestPauseResumeLoopDefaultSettings to get un-deployed
2021/12/10 04:00:08 Waiting for app: TestPauseResumeLoopDefaultSettings to get un-deployed
2021/12/10 04:00:13 Waiting for app: TestPauseResumeLoopDefaultSettings to get un-deployed
2021/12/10 04:00:13 App: TestPauseResumeLoopDefaultSettings got un-deployed
2021/12/10 04:00:13 No eventing-con process running
2021/12/10 04:00:13 Function: TestPauseResumeLoopDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestPauseResumeLoopDefaultSettings deleting in the background"
}
2021/12/10 04:00:15 Context: Bucket flush, response code: 200 dump: 
2021/12/10 04:00:25 Waiting for dst bucket item count to get to: 0 curr count: 25000
2021/12/10 04:00:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 04:00:27 Context: Bucket flush, response code: 200 dump: 
2021/12/10 04:00:37 Waiting for dst bucket item count to get to: 0 curr count: 25000
2021/12/10 04:00:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestPauseResumeLoopDefaultSettings (228.52s)
=== RUN   TestPauseResumeLoopNonDefaultSettings
2021/12/10 04:00:47 Waiting for app: TestPauseResumeLoopNonDefaultSettings to get un-deployed
2021/12/10 04:00:47 App: TestPauseResumeLoopNonDefaultSettings got un-deployed
2021/12/10 04:00:47 No eventing-con process running
2021/12/10 04:00:47 Function: TestPauseResumeLoopNonDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestPauseResumeLoopNonDefaultSettings not found"
 }
}
2021/12/10 04:00:49 Context: Bucket flush, response code: 200 dump: 
2021/12/10 04:00:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 04:00:56 Context: Bucket flush, response code: 200 dump: 
2021/12/10 04:01:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 04:01:01 Deploying app: TestPauseResumeLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/10 04:01:02 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestPauseResumeLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/10 04:01:02 Starting bucket ops to source bucket
2021/12/10 04:01:10 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 04:01:15 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 04:01:20 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 04:01:25 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 04:01:30 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 04:01:35 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 04:01:40 Waiting for dst bucket item count to get to: 5000 curr count: 3781
2021/12/10 04:01:45 Waiting for dst bucket item count to get to: 5000 curr count: 3781
2021/12/10 04:01:45 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/10 04:01:45 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 04:01:50 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to deployed
2021/12/10 04:01:50 Function: TestPauseResumeLoopNonDefaultSettings status changed to deployed
2021/12/10 04:01:50 Pausing app: bucket_op_with_timer
2021/12/10 04:01:50 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/10 04:01:55 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to paused
2021/12/10 04:01:55 Function: TestPauseResumeLoopNonDefaultSettings status changed to paused
2021/12/10 04:01:55 Resuming app: bucket_op_with_timer
2021/12/10 04:01:56 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/10 04:01:56 Starting bucket ops to source bucket
2021/12/10 04:02:04 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/10 04:02:09 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/10 04:02:14 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/10 04:02:19 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/10 04:02:24 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/10 04:02:29 Waiting for dst bucket item count to get to: 10000 curr count: 8492
2021/12/10 04:02:34 Waiting for dst bucket item count to get to: 10000 curr count: 8492
2021/12/10 04:02:34 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/10 04:02:34 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 04:02:39 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to deployed
2021/12/10 04:02:39 Function: TestPauseResumeLoopNonDefaultSettings status changed to deployed
2021/12/10 04:02:39 Pausing app: bucket_op_with_timer
2021/12/10 04:02:39 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/10 04:02:44 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to paused
2021/12/10 04:02:44 Function: TestPauseResumeLoopNonDefaultSettings status changed to paused
2021/12/10 04:02:44 Resuming app: bucket_op_with_timer
2021/12/10 04:02:45 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/10 04:02:45 Starting bucket ops to source bucket
2021/12/10 04:02:53 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/10 04:02:58 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/10 04:03:03 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/10 04:03:08 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/10 04:03:13 Waiting for dst bucket item count to get to: 15000 curr count: 10996
2021/12/10 04:03:18 Waiting for dst bucket item count to get to: 15000 curr count: 13496
2021/12/10 04:03:18 src & dst bucket item count matched up. src bucket count: 15000 dst bucket count: 15000
2021/12/10 04:03:18 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 04:03:23 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to deployed
2021/12/10 04:03:23 Function: TestPauseResumeLoopNonDefaultSettings status changed to deployed
2021/12/10 04:03:23 Pausing app: bucket_op_with_timer
2021/12/10 04:03:23 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/10 04:03:28 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to paused
2021/12/10 04:03:28 Function: TestPauseResumeLoopNonDefaultSettings status changed to paused
2021/12/10 04:03:28 Resuming app: bucket_op_with_timer
2021/12/10 04:03:30 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/10 04:03:30 Starting bucket ops to source bucket
2021/12/10 04:03:38 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/10 04:03:43 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/10 04:03:48 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/10 04:03:53 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/10 04:03:58 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/10 04:04:03 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/10 04:04:08 Waiting for dst bucket item count to get to: 20000 curr count: 17500
2021/12/10 04:04:08 src & dst bucket item count matched up. src bucket count: 20000 dst bucket count: 20000
2021/12/10 04:04:08 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 04:04:13 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to deployed
2021/12/10 04:04:13 Function: TestPauseResumeLoopNonDefaultSettings status changed to deployed
2021/12/10 04:04:13 Pausing app: bucket_op_with_timer
2021/12/10 04:04:13 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/10 04:04:18 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to paused
2021/12/10 04:04:18 Function: TestPauseResumeLoopNonDefaultSettings status changed to paused
2021/12/10 04:04:18 Resuming app: bucket_op_with_timer
2021/12/10 04:04:19 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/10 04:04:19 Starting bucket ops to source bucket
2021/12/10 04:04:27 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/10 04:04:32 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/10 04:04:37 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/10 04:04:42 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/10 04:04:47 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/10 04:04:52 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/10 04:04:57 Waiting for dst bucket item count to get to: 25000 curr count: 22500
2021/12/10 04:04:57 src & dst bucket item count matched up. src bucket count: 25000 dst bucket count: 25000
2021/12/10 04:04:57 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 04:05:02 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to deployed
2021/12/10 04:05:02 Function: TestPauseResumeLoopNonDefaultSettings status changed to deployed
2021/12/10 04:05:02 Pausing app: bucket_op_with_timer
2021/12/10 04:05:02 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/10 04:05:07 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to paused
2021/12/10 04:05:07 Function: TestPauseResumeLoopNonDefaultSettings status changed to paused
2021/12/10 04:05:07 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 04:05:12 Waiting for app: TestPauseResumeLoopNonDefaultSettings to get un-deployed
2021/12/10 04:05:17 Waiting for app: TestPauseResumeLoopNonDefaultSettings to get un-deployed
2021/12/10 04:05:22 Waiting for app: TestPauseResumeLoopNonDefaultSettings to get un-deployed
2021/12/10 04:05:27 Waiting for app: TestPauseResumeLoopNonDefaultSettings to get un-deployed
2021/12/10 04:05:27 App: TestPauseResumeLoopNonDefaultSettings got un-deployed
2021/12/10 04:05:27 No eventing-con process running
2021/12/10 04:05:27 Function: TestPauseResumeLoopNonDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestPauseResumeLoopNonDefaultSettings deleting in the background"
}
2021/12/10 04:05:29 Context: Bucket flush, response code: 200 dump: 
2021/12/10 04:05:39 Waiting for dst bucket item count to get to: 0 curr count: 25000
2021/12/10 04:05:44 Waiting for dst bucket item count to get to: 0 curr count: 12500
2021/12/10 04:05:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 04:05:46 Context: Bucket flush, response code: 200 dump: 
2021/12/10 04:05:56 Waiting for dst bucket item count to get to: 0 curr count: 25000
2021/12/10 04:05:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestPauseResumeLoopNonDefaultSettings (318.65s)
=== RUN   TestPauseAndResumeWithWorkerCountChange
2021/12/10 04:06:06 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get un-deployed
2021/12/10 04:06:06 App: TestPauseAndResumeWithWorkerCountChange got un-deployed
2021/12/10 04:06:06 No eventing-con process running
2021/12/10 04:06:06 Function: TestPauseAndResumeWithWorkerCountChange delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestPauseAndResumeWithWorkerCountChange not found"
 }
}
2021/12/10 04:06:07 Context: Bucket flush, response code: 200 dump: 
2021/12/10 04:06:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 04:06:14 Context: Bucket flush, response code: 200 dump: 
2021/12/10 04:06:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 04:06:19 Deploying app: TestPauseAndResumeWithWorkerCountChange. Item count src bucket: 0 dst bucket: 0
2021/12/10 04:06:20 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestPauseAndResumeWithWorkerCountChange' in metakv",
  "warnings": null
 }
}
2021/12/10 04:06:25 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get deployed
2021/12/10 04:06:30 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get deployed
2021/12/10 04:06:35 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get deployed
2021/12/10 04:06:40 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get deployed
2021/12/10 04:06:40 App: TestPauseAndResumeWithWorkerCountChange got deployed
2021/12/10 04:06:40 Starting bucket ops to source bucket
2021/12/10 04:06:50 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 04:06:55 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 04:07:00 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 04:07:05 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 04:07:10 Waiting for dst bucket item count to get to: 5000 curr count: 3631
2021/12/10 04:07:15 Waiting for dst bucket item count to get to: 5000 curr count: 3631
2021/12/10 04:07:15 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/10 04:07:15 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 04:07:15 Pausing app: bucket_op_with_timer
2021/12/10 04:07:15 Function: TestPauseAndResumeWithWorkerCountChange update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/10 04:07:20 Waiting for function: TestPauseAndResumeWithWorkerCountChange status to change to paused
2021/12/10 04:07:20 Function: TestPauseAndResumeWithWorkerCountChange status changed to paused
2021/12/10 04:07:20 Resuming app: bucket_op_with_timer
2021/12/10 04:07:21 Function: TestPauseAndResumeWithWorkerCountChange update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:6] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/10 04:07:21 Starting bucket ops to source bucket
2021/12/10 04:07:32 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/10 04:07:37 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/10 04:07:42 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/10 04:07:48 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/10 04:07:53 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/10 04:07:53 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/10 04:07:53 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 04:07:58 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get un-deployed
2021/12/10 04:08:03 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get un-deployed
2021/12/10 04:08:08 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get un-deployed
2021/12/10 04:08:13 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get un-deployed
2021/12/10 04:08:13 App: TestPauseAndResumeWithWorkerCountChange got un-deployed
2021/12/10 04:08:13 No eventing-con process running
2021/12/10 04:08:13 Function: TestPauseAndResumeWithWorkerCountChange delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestPauseAndResumeWithWorkerCountChange deleting in the background"
}
2021/12/10 04:08:14 Context: Bucket flush, response code: 200 dump: 
2021/12/10 04:08:24 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/10 04:08:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 04:08:26 Context: Bucket flush, response code: 200 dump: 
2021/12/10 04:08:36 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/10 04:08:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestPauseAndResumeWithWorkerCountChange (160.52s)
=== RUN   TestPauseResumeWithEventingReb
2021/12/10 04:08:46 Waiting for app: TestPauseResumeWithEventingReb to get un-deployed
2021/12/10 04:08:46 App: TestPauseResumeWithEventingReb got un-deployed
2021/12/10 04:08:46 No eventing-con process running
2021/12/10 04:08:46 Function: TestPauseResumeWithEventingReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestPauseResumeWithEventingReb not found"
 }
}
2021/12/10 04:08:48 Context: Bucket flush, response code: 200 dump: 
2021/12/10 04:08:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 04:08:55 Context: Bucket flush, response code: 200 dump: 
2021/12/10 04:09:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 04:09:00 Deploying app: TestPauseResumeWithEventingReb. Item count src bucket: 0 dst bucket: 0
2021/12/10 04:09:01 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestPauseResumeWithEventingReb' in metakv",
  "warnings": null
 }
}
2021/12/10 04:09:06 Waiting for app: TestPauseResumeWithEventingReb to get deployed
2021/12/10 04:09:11 Waiting for app: TestPauseResumeWithEventingReb to get deployed
2021/12/10 04:09:16 Waiting for app: TestPauseResumeWithEventingReb to get deployed
2021/12/10 04:09:21 Waiting for app: TestPauseResumeWithEventingReb to get deployed
2021/12/10 04:09:21 App: TestPauseResumeWithEventingReb got deployed
2021/12/10 04:09:21 Starting bucket ops to source bucket
2021/12/10 04:09:30 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 04:09:35 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 04:09:40 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 04:09:45 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 04:09:50 Waiting for dst bucket item count to get to: 5000 curr count: 4154
2021/12/10 04:09:55 Waiting for dst bucket item count to get to: 5000 curr count: 4154
2021/12/10 04:09:55 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/10 04:09:55 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 04:09:55 Pausing app: bucket_op_with_timer
2021/12/10 04:09:55 Function: TestPauseResumeWithEventingReb update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/10 04:10:00 Waiting for function: TestPauseResumeWithEventingReb status to change to paused
2021/12/10 04:10:00 Function: TestPauseResumeWithEventingReb status changed to paused
2021/12/10 04:10:00 Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/10 04:10:01 Removing node(s): [] from the cluster
2021/12/10 04:10:06 Rebalance progress: 29.16666666666666
2021/12/10 04:10:11 Rebalance progress: 29.16666666666666
2021/12/10 04:10:16 Rebalance progress: 100
2021/12/10 04:10:16 Resuming app: bucket_op_with_timer
2021/12/10 04:10:17 Function: TestPauseResumeWithEventingReb update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:6] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/10 04:10:17 Starting bucket ops to source bucket
2021/12/10 04:10:29 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/10 04:10:34 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/10 04:10:39 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/10 04:10:44 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/10 04:10:49 Waiting for dst bucket item count to get to: 10000 curr count: 5604
2021/12/10 04:10:54 Waiting for dst bucket item count to get to: 10000 curr count: 5604
2021/12/10 04:10:54 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/10 04:10:54 Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/10 04:10:59 Rebalance progress: 29.16666666666666
2021/12/10 04:11:04 Rebalance progress: 29.16666666666666
2021/12/10 04:11:09 Rebalance progress: 100

VB distribution across eventing nodes and their workers::

2021/12/10 04:11:14 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:54} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/10 04:11:29 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:54} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/10 04:11:44 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:54} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/10 04:11:59 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:54} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/10 04:12:14 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:54} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/10 04:12:29 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:54} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/10 04:12:44 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:54} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/10 04:12:59 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:54} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/10 04:13:14 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:54} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/10 04:13:29 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:54} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/10 04:13:44 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:54} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/10 04:13:59 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:54} {Name:#primary Bucket:hello-world Completion:100}]

DCP Stream statuses:
	stream status: running
	len: 1024
	vb list dump: [0-1023]

vbucket curr owner:
Producer node: 127.0.0.1:9302	Node UUID: 489186af2a220152a232d8970a77c4b3
	workerID: worker_TestPauseResumeWithEventingReb_2
	len: 57
	v: [114-170]
	workerID: worker_TestPauseResumeWithEventingReb_3
	len: 57
	v: [171-227]
	workerID: worker_TestPauseResumeWithEventingReb_4
	len: 57
	v: [228-284]
	workerID: worker_TestPauseResumeWithEventingReb_5
	len: 57
	v: [285-341]
	workerID: worker_TestPauseResumeWithEventingReb_0
	len: 57
	v: [0-56]
	workerID: worker_TestPauseResumeWithEventingReb_1
	len: 57
	v: [57-113]
Producer node: 172.31.2.134:9300	Node UUID: 2c6085b1258ecb09bc4044f4129c5a75
	workerID: worker_TestPauseResumeWithEventingReb_2
	len: 57
	v: [797-853]
	workerID: worker_TestPauseResumeWithEventingReb_3
	len: 57
	v: [854-910]
	workerID: worker_TestPauseResumeWithEventingReb_4
	len: 57
	v: [911-967]
	workerID: worker_TestPauseResumeWithEventingReb_5
	len: 56
	v: [968-1023]
	workerID: worker_TestPauseResumeWithEventingReb_0
	len: 57
	v: [683-739]
	workerID: worker_TestPauseResumeWithEventingReb_1
	len: 57
	v: [740-796]
Producer node: 127.0.0.1:9303	Node UUID: a5e5803ffc3c6d65069f0e8054b5433c
	workerID: worker_TestPauseResumeWithEventingReb_5
	len: 56
	v: [627-682]
	workerID: worker_TestPauseResumeWithEventingReb_0
	len: 57
	v: [342-398]
	workerID: worker_TestPauseResumeWithEventingReb_1
	len: 57
	v: [399-455]
	workerID: worker_TestPauseResumeWithEventingReb_2
	len: 57
	v: [456-512]
	workerID: worker_TestPauseResumeWithEventingReb_3
	len: 57
	v: [513-569]
	workerID: worker_TestPauseResumeWithEventingReb_4
	len: 57
	v: [570-626]


2021/12/10 04:14:14 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 04:14:14 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 04:14:19 clusterSetup.go:397: Waiting for app: TestPauseResumeWithEventingReb to get un-deployed
2021/12/10 04:14:24 clusterSetup.go:397: Waiting for app: TestPauseResumeWithEventingReb to get un-deployed
2021/12/10 04:14:29 clusterSetup.go:397: Waiting for app: TestPauseResumeWithEventingReb to get un-deployed
2021/12/10 04:14:34 clusterSetup.go:397: Waiting for app: TestPauseResumeWithEventingReb to get un-deployed
2021/12/10 04:14:34 clusterSetup.go:405: App: TestPauseResumeWithEventingReb got un-deployed
2021/12/10 04:14:34 eventingSetup.go:46: No eventing-con process running
2021/12/10 04:14:34 eventingSetup.go:418: Function: TestPauseResumeWithEventingReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestPauseResumeWithEventingReb deleting in the background"
}
2021/12/10 04:14:36 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 04:14:46 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/10 04:14:46 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 04:14:48 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 04:14:58 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6438
2021/12/10 04:15:03 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1438
2021/12/10 04:15:03 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestPauseResumeWithEventingReb (386.64s)
=== RUN   TestBucketDeleteAfterPause
2021/12/10 04:15:13 clusterSetup.go:397: Waiting for app: TestBucketDeleteAfterPause to get un-deployed
2021/12/10 04:15:13 clusterSetup.go:405: App: TestBucketDeleteAfterPause got un-deployed
2021/12/10 04:15:13 eventingSetup.go:46: No eventing-con process running
2021/12/10 04:15:13 eventingSetup.go:418: Function: TestBucketDeleteAfterPause delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestBucketDeleteAfterPause not found"
 }
}
2021/12/10 04:15:15 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 04:15:20 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 04:15:21 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 04:15:26 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 04:15:26 eventingSetup.go:125: Deploying app: TestBucketDeleteAfterPause. Item count src bucket: 0 dst bucket: 0
2021/12/10 04:15:28 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestBucketDeleteAfterPause' in metakv",
  "warnings": null
 }
}
2021/12/10 04:15:33 clusterSetup.go:346: Waiting for app: TestBucketDeleteAfterPause to get deployed
2021/12/10 04:15:38 clusterSetup.go:346: Waiting for app: TestBucketDeleteAfterPause to get deployed
2021/12/10 04:15:43 clusterSetup.go:346: Waiting for app: TestBucketDeleteAfterPause to get deployed
2021/12/10 04:15:48 clusterSetup.go:346: Waiting for app: TestBucketDeleteAfterPause to get deployed
2021/12/10 04:15:48 clusterSetup.go:354: App: TestBucketDeleteAfterPause got deployed
2021/12/10 04:15:48 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/10 04:15:57 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 871
2021/12/10 04:16:02 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 871
2021/12/10 04:16:02 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/10 04:16:02 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 04:16:02 handler_test.go:712: Pausing app: bucket_op_on_update
2021/12/10 04:16:03 eventingSetup.go:270: Function: TestBucketDeleteAfterPause update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/10 04:16:08 eventingSetup.go:719: Waiting for function: TestBucketDeleteAfterPause status to change to paused
2021/12/10 04:16:08 eventingSetup.go:727: Function: TestBucketDeleteAfterPause status changed to paused
2021/12/10 04:16:08 handler_test.go:716: Deleting source bucket: default
2021/12/10 04:16:14 eventingSetup.go:719: Waiting for function: TestBucketDeleteAfterPause status to change to undeployed
2021/12/10 04:16:14 eventingSetup.go:727: Function: TestBucketDeleteAfterPause status changed to undeployed
2021/12/10 04:16:19 clusterSetup.go:397: Waiting for app: TestBucketDeleteAfterPause to get un-deployed
2021/12/10 04:16:19 clusterSetup.go:405: App: TestBucketDeleteAfterPause got un-deployed
2021/12/10 04:16:19 eventingSetup.go:46: No eventing-con process running
2021/12/10 04:16:19 eventingSetup.go:418: Function: TestBucketDeleteAfterPause delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestBucketDeleteAfterPause deleting in the background"
}
2021/12/10 04:16:20 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 04:16:25 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 04:16:27 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 04:16:37 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/10 04:16:37 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestBucketDeleteAfterPause (94.37s)
=== RUN   TestChangeFnCodeBetweenPauseResume
2021/12/10 04:16:47 clusterSetup.go:397: Waiting for app: TestChangeFnCodeBetweenPauseResume to get un-deployed
2021/12/10 04:16:47 clusterSetup.go:405: App: TestChangeFnCodeBetweenPauseResume got un-deployed
2021/12/10 04:16:47 eventingSetup.go:46: No eventing-con process running
2021/12/10 04:16:47 eventingSetup.go:418: Function: TestChangeFnCodeBetweenPauseResume delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestChangeFnCodeBetweenPauseResume not found"
 }
}
2021/12/10 04:16:49 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 04:16:54 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 04:16:55 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 04:17:00 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 04:17:00 eventingSetup.go:125: Deploying app: TestChangeFnCodeBetweenPauseResume. Item count src bucket: 0 dst bucket: 0
2021/12/10 04:17:01 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestChangeFnCodeBetweenPauseResume' in metakv",
  "warnings": null
 }
}
2021/12/10 04:17:06 clusterSetup.go:346: Waiting for app: TestChangeFnCodeBetweenPauseResume to get deployed
2021/12/10 04:17:11 clusterSetup.go:346: Waiting for app: TestChangeFnCodeBetweenPauseResume to get deployed
2021/12/10 04:17:16 clusterSetup.go:346: Waiting for app: TestChangeFnCodeBetweenPauseResume to get deployed
2021/12/10 04:17:21 clusterSetup.go:346: Waiting for app: TestChangeFnCodeBetweenPauseResume to get deployed
2021/12/10 04:17:22 clusterSetup.go:354: App: TestChangeFnCodeBetweenPauseResume got deployed
2021/12/10 04:17:22 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/10 04:17:56 handler_test.go:739: Pausing function: TestChangeFnCodeBetweenPauseResume
2021/12/10 04:17:56 eventingSetup.go:270: Function: TestChangeFnCodeBetweenPauseResume update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/10 04:18:01 eventingSetup.go:719: Waiting for function: TestChangeFnCodeBetweenPauseResume status to change to paused
2021/12/10 04:18:01 eventingSetup.go:727: Function: TestChangeFnCodeBetweenPauseResume status changed to paused
2021/12/10 04:21:01 handler_test.go:748: Resuming function: TestChangeFnCodeBetweenPauseResume
2021/12/10 04:21:01 eventingSetup.go:125: Deploying app: TestChangeFnCodeBetweenPauseResume. Item count src bucket: 5000 dst bucket: 0
2021/12/10 04:21:03 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestChangeFnCodeBetweenPauseResume' in metakv",
  "warnings": null
 }
}
2021/12/10 04:21:08 eventingSetup.go:795: Waiting for function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter to get to 5000
2021/12/10 04:21:08 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 04:21:08 eventingSetup.go:799: Function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter got to 0 expected 5000
2021/12/10 04:21:13 eventingSetup.go:795: Waiting for function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter to get to 5000
2021/12/10 04:21:13 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 04:21:13 eventingSetup.go:799: Function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter got to 3055 expected 5000
2021/12/10 04:21:18 eventingSetup.go:795: Waiting for function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter to get to 5000
2021/12/10 04:21:18 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 04:21:18 eventingSetup.go:799: Function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter got to 4452 expected 5000
2021/12/10 04:21:23 eventingSetup.go:795: Waiting for function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter to get to 5000
2021/12/10 04:21:23 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 04:21:23 eventingSetup.go:804: Function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter got to 5000
2021/12/10 04:21:23 handler_test.go:753: Undeploying function: TestChangeFnCodeBetweenPauseResume
2021/12/10 04:21:23 eventingSetup.go:270: Function: TestChangeFnCodeBetweenPauseResume update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/10 04:21:33 clusterSetup.go:397: Waiting for app: TestChangeFnCodeBetweenPauseResume to get un-deployed
2021/12/10 04:21:38 clusterSetup.go:397: Waiting for app: TestChangeFnCodeBetweenPauseResume to get un-deployed
2021/12/10 04:21:43 clusterSetup.go:397: Waiting for app: TestChangeFnCodeBetweenPauseResume to get un-deployed
2021/12/10 04:21:43 clusterSetup.go:405: App: TestChangeFnCodeBetweenPauseResume got un-deployed
2021/12/10 04:21:43 eventingSetup.go:46: No eventing-con process running
2021/12/10 04:21:43 eventingSetup.go:418: Function: TestChangeFnCodeBetweenPauseResume delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestChangeFnCodeBetweenPauseResume deleting in the background"
}
2021/12/10 04:21:44 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 04:21:54 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/10 04:21:59 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/10 04:21:59 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 04:22:01 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 04:22:06 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestChangeFnCodeBetweenPauseResume (328.63s)
=== RUN   TestCommentUnCommentOnDelete
2021/12/10 04:22:16 clusterSetup.go:397: Waiting for app: TestCommentUnCommentOnDelete to get un-deployed
2021/12/10 04:22:16 clusterSetup.go:405: App: TestCommentUnCommentOnDelete got un-deployed
2021/12/10 04:22:16 eventingSetup.go:46: No eventing-con process running
2021/12/10 04:22:16 eventingSetup.go:418: Function: TestCommentUnCommentOnDelete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestCommentUnCommentOnDelete not found"
 }
}
2021/12/10 04:22:17 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 04:22:22 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 04:22:24 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 04:22:29 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 04:22:29 eventingSetup.go:125: Deploying app: TestCommentUnCommentOnDelete. Item count src bucket: 0 dst bucket: 0
2021/12/10 04:22:30 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCommentUnCommentOnDelete' in metakv",
  "warnings": null
 }
}
2021/12/10 04:22:35 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/10 04:22:40 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/10 04:22:45 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/10 04:22:50 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/10 04:22:50 clusterSetup.go:354: App: TestCommentUnCommentOnDelete got deployed
2021/12/10 04:22:50 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/10 04:22:59 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 04:23:04 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 04:23:04 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/10 04:23:04 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 04:23:04 handler_test.go:780: Undeploying app: TestCommentUnCommentOnDelete
2021/12/10 04:23:04 eventingSetup.go:270: Function: TestCommentUnCommentOnDelete update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/10 04:23:34 eventingSetup.go:125: Deploying app: TestCommentUnCommentOnDelete. Item count src bucket: 5000 dst bucket: 5000
2021/12/10 04:23:35 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCommentUnCommentOnDelete' in metakv",
  "warnings": null
 }
}
2021/12/10 04:23:40 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/10 04:23:45 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/10 04:23:50 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/10 04:23:55 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/10 04:23:55 clusterSetup.go:354: App: TestCommentUnCommentOnDelete got deployed
2021/12/10 04:23:55 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/10 04:24:08 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4916
2021/12/10 04:24:13 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4916
2021/12/10 04:24:13 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 04:24:13 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 04:24:13 handler_test.go:799: Undeploying app: TestCommentUnCommentOnDelete
2021/12/10 04:24:13 eventingSetup.go:270: Function: TestCommentUnCommentOnDelete update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/10 04:24:23 clusterSetup.go:397: Waiting for app: TestCommentUnCommentOnDelete to get un-deployed
2021/12/10 04:24:28 clusterSetup.go:397: Waiting for app: TestCommentUnCommentOnDelete to get un-deployed
2021/12/10 04:24:33 clusterSetup.go:397: Waiting for app: TestCommentUnCommentOnDelete to get un-deployed
2021/12/10 04:24:33 clusterSetup.go:405: App: TestCommentUnCommentOnDelete got un-deployed
2021/12/10 04:24:33 eventingSetup.go:46: No eventing-con process running
2021/12/10 04:24:33 eventingSetup.go:418: Function: TestCommentUnCommentOnDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCommentUnCommentOnDelete deleting in the background"
}
2021/12/10 04:24:34 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 04:24:39 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 04:24:41 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 04:24:46 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCommentUnCommentOnDelete (160.14s)
=== RUN   TestCPPWorkerCleanup
2021/12/10 04:24:56 clusterSetup.go:397: Waiting for app: TestCPPWorkerCleanup to get un-deployed
2021/12/10 04:24:56 clusterSetup.go:405: App: TestCPPWorkerCleanup got un-deployed
2021/12/10 04:24:56 eventingSetup.go:46: No eventing-con process running
2021/12/10 04:24:56 eventingSetup.go:418: Function: TestCPPWorkerCleanup delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestCPPWorkerCleanup not found"
 }
}
2021/12/10 04:24:57 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 04:25:02 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 04:25:04 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 04:25:09 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 04:25:09 eventingSetup.go:125: Deploying app: TestCPPWorkerCleanup. Item count src bucket: 0 dst bucket: 0
2021/12/10 04:25:10 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCPPWorkerCleanup' in metakv",
  "warnings": null
 }
}
2021/12/10 04:25:10 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/10 04:25:17 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 04:25:22 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 04:25:27 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 04:25:32 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 04:25:32 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/10 04:25:32 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 04:25:32 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 04:25:37 clusterSetup.go:397: Waiting for app: TestCPPWorkerCleanup to get un-deployed
2021/12/10 04:25:42 clusterSetup.go:397: Waiting for app: TestCPPWorkerCleanup to get un-deployed
2021/12/10 04:25:47 clusterSetup.go:397: Waiting for app: TestCPPWorkerCleanup to get un-deployed
2021/12/10 04:25:52 clusterSetup.go:397: Waiting for app: TestCPPWorkerCleanup to get un-deployed
2021/12/10 04:25:52 clusterSetup.go:405: App: TestCPPWorkerCleanup got un-deployed
2021/12/10 04:25:52 eventingSetup.go:46: No eventing-con process running
2021/12/10 04:25:52 eventingSetup.go:418: Function: TestCPPWorkerCleanup delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCPPWorkerCleanup deleting in the background"
}
2021/12/10 04:25:54 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 04:26:04 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/10 04:26:09 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/10 04:26:09 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 04:26:10 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 04:26:20 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4025
2021/12/10 04:26:25 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4025
2021/12/10 04:26:25 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCPPWorkerCleanup (129.41s)
=== RUN   TestWithUserXattrs
2021/12/10 04:27:05 clusterSetup.go:397: Waiting for app: TestWithUserXattrs to get un-deployed
2021/12/10 04:27:05 clusterSetup.go:405: App: TestWithUserXattrs got un-deployed
2021/12/10 04:27:05 eventingSetup.go:46: No eventing-con process running
2021/12/10 04:27:05 eventingSetup.go:418: Function: TestWithUserXattrs delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestWithUserXattrs not found"
 }
}
2021/12/10 04:27:07 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 04:27:12 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 04:27:13 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 04:27:18 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 04:27:18 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/10 04:27:20 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/10 04:27:22 eventingSetup.go:125: Deploying app: TestWithUserXattrs. Item count src bucket: 0 dst bucket: 0
2021/12/10 04:27:23 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestWithUserXattrs' in metakv",
  "warnings": null
 }
}
2021/12/10 04:27:28 clusterSetup.go:346: Waiting for app: TestWithUserXattrs to get deployed
2021/12/10 04:27:33 clusterSetup.go:346: Waiting for app: TestWithUserXattrs to get deployed
2021/12/10 04:27:38 clusterSetup.go:346: Waiting for app: TestWithUserXattrs to get deployed
2021/12/10 04:27:44 clusterSetup.go:346: Waiting for app: TestWithUserXattrs to get deployed
2021/12/10 04:27:44 clusterSetup.go:354: App: TestWithUserXattrs got deployed
2021/12/10 04:27:44 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/10 04:27:53 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 04:27:58 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 2132
2021/12/10 04:28:03 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 4632
2021/12/10 04:28:03 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/10 04:28:03 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 04:28:03 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 04:28:08 clusterSetup.go:397: Waiting for app: TestWithUserXattrs to get un-deployed
2021/12/10 04:28:13 clusterSetup.go:397: Waiting for app: TestWithUserXattrs to get un-deployed
2021/12/10 04:28:18 clusterSetup.go:397: Waiting for app: TestWithUserXattrs to get un-deployed
2021/12/10 04:28:23 clusterSetup.go:397: Waiting for app: TestWithUserXattrs to get un-deployed
2021/12/10 04:28:23 clusterSetup.go:405: App: TestWithUserXattrs got un-deployed
2021/12/10 04:28:23 eventingSetup.go:46: No eventing-con process running
2021/12/10 04:28:23 eventingSetup.go:418: Function: TestWithUserXattrs delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestWithUserXattrs deleting in the background"
}
2021/12/10 04:28:24 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 04:28:34 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/10 04:28:39 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/10 04:28:39 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 04:28:41 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 04:28:51 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4907
2021/12/10 04:28:56 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4907
2021/12/10 04:28:56 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestWithUserXattrs (120.53s)
=== RUN   TestEventProcessingPostBucketFlush
2021/12/10 04:29:06 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/10 04:29:06 clusterSetup.go:405: App: TestEventProcessingPostBucketFlush got un-deployed
2021/12/10 04:29:06 eventingSetup.go:46: No eventing-con process running
2021/12/10 04:29:06 eventingSetup.go:418: Function: TestEventProcessingPostBucketFlush delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventProcessingPostBucketFlush not found"
 }
}
2021/12/10 04:29:07 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 04:29:12 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 04:29:14 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 04:29:19 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 04:29:19 eventingSetup.go:125: Deploying app: TestEventProcessingPostBucketFlush. Item count src bucket: 0 dst bucket: 0
2021/12/10 04:29:20 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventProcessingPostBucketFlush' in metakv",
  "warnings": null
 }
}
2021/12/10 04:29:25 clusterSetup.go:346: Waiting for app: TestEventProcessingPostBucketFlush to get deployed
2021/12/10 04:29:30 clusterSetup.go:346: Waiting for app: TestEventProcessingPostBucketFlush to get deployed
2021/12/10 04:29:35 clusterSetup.go:346: Waiting for app: TestEventProcessingPostBucketFlush to get deployed
2021/12/10 04:29:40 clusterSetup.go:346: Waiting for app: TestEventProcessingPostBucketFlush to get deployed
2021/12/10 04:29:40 clusterSetup.go:354: App: TestEventProcessingPostBucketFlush got deployed
2021/12/10 04:29:40 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/10 04:29:49 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 04:29:54 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 04:29:54 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/10 04:29:54 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 04:29:58 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 04:30:08 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/10 04:30:08 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 04:30:10 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 04:30:20 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3103
2021/12/10 04:30:25 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3103
2021/12/10 04:30:25 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 04:30:25 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/10 04:30:50 eventingSetup.go:441: Waiting for dst bucket item count to get to: 25000 curr count: 16086
2021/12/10 04:30:55 eventingSetup.go:441: Waiting for dst bucket item count to get to: 25000 curr count: 16086
2021/12/10 04:30:55 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 25000 dst bucket count: 25000
2021/12/10 04:30:55 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 04:30:55 eventingSetup.go:270: Function: TestEventProcessingPostBucketFlush update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/10 04:31:00 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/10 04:31:05 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/10 04:31:10 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/10 04:31:15 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/10 04:31:15 clusterSetup.go:405: App: TestEventProcessingPostBucketFlush got un-deployed
2021/12/10 04:31:15 eventingSetup.go:46: No eventing-con process running
2021/12/10 04:31:20 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/10 04:31:20 clusterSetup.go:405: App: TestEventProcessingPostBucketFlush got un-deployed
2021/12/10 04:31:20 eventingSetup.go:46: No eventing-con process running
2021/12/10 04:31:20 eventingSetup.go:418: Function: TestEventProcessingPostBucketFlush delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventProcessingPostBucketFlush deleting in the background"
}
2021/12/10 04:31:22 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 04:31:32 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 25000
2021/12/10 04:31:37 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 25000
2021/12/10 04:31:37 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 04:31:38 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 04:31:48 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 21470
2021/12/10 04:31:53 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 8970
2021/12/10 04:31:53 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventProcessingPostBucketFlush (177.54s)
=== RUN   TestMetaBucketDelete
2021/12/10 04:32:03 clusterSetup.go:397: Waiting for app: TestMetaBucketDelete to get un-deployed
2021/12/10 04:32:04 clusterSetup.go:405: App: TestMetaBucketDelete got un-deployed
2021/12/10 04:32:04 eventingSetup.go:46: No eventing-con process running
2021/12/10 04:32:04 eventingSetup.go:418: Function: TestMetaBucketDelete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMetaBucketDelete not found"
 }
}
2021/12/10 04:32:05 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 04:32:10 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 04:32:12 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 04:32:17 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 04:32:17 eventingSetup.go:125: Deploying app: TestMetaBucketDelete. Item count src bucket: 0 dst bucket: 0
2021/12/10 04:32:18 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMetaBucketDelete' in metakv",
  "warnings": null
 }
}
2021/12/10 04:32:23 clusterSetup.go:346: Waiting for app: TestMetaBucketDelete to get deployed
2021/12/10 04:32:28 clusterSetup.go:346: Waiting for app: TestMetaBucketDelete to get deployed
2021/12/10 04:32:33 clusterSetup.go:346: Waiting for app: TestMetaBucketDelete to get deployed
2021/12/10 04:32:38 clusterSetup.go:346: Waiting for app: TestMetaBucketDelete to get deployed
2021/12/10 04:32:38 clusterSetup.go:354: App: TestMetaBucketDelete got deployed
2021/12/10 04:32:38 handler_test.go:902: Deleting metadata bucket: eventing
2021/12/10 04:32:39 handler_test.go:904: Deleted metadata bucket: eventing
2021/12/10 04:32:44 clusterSetup.go:397: Waiting for app: TestMetaBucketDelete to get un-deployed
2021/12/10 04:32:44 clusterSetup.go:405: App: TestMetaBucketDelete got un-deployed
2021/12/10 04:32:59 clusterSetup.go:397: Waiting for app: TestMetaBucketDelete to get un-deployed
2021/12/10 04:32:59 clusterSetup.go:405: App: TestMetaBucketDelete got un-deployed
2021/12/10 04:32:59 eventingSetup.go:46: No eventing-con process running
2021/12/10 04:32:59 eventingSetup.go:418: Function: TestMetaBucketDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMetaBucketDelete deleting in the background"
}
2021/12/10 04:33:00 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 04:33:05 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 04:33:07 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 04:33:12 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestMetaBucketDelete (78.45s)
=== RUN   TestMetaBucketDeleteWithBootstrap
2021/12/10 04:33:22 clusterSetup.go:397: Waiting for app: TestMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/10 04:33:22 clusterSetup.go:405: App: TestMetaBucketDeleteWithBootstrap got un-deployed
2021/12/10 04:33:22 eventingSetup.go:46: No eventing-con process running
2021/12/10 04:33:22 eventingSetup.go:418: Function: TestMetaBucketDeleteWithBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMetaBucketDeleteWithBootstrap not found"
 }
}
2021/12/10 04:33:23 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 04:33:28 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 04:33:30 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 04:33:35 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 04:33:35 eventingSetup.go:125: Deploying app: TestMetaBucketDeleteWithBootstrap. Item count src bucket: 0 dst bucket: 0
2021/12/10 04:33:36 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMetaBucketDeleteWithBootstrap' in metakv",
  "warnings": null
 }
}
2021/12/10 04:33:55 handler_test.go:923: Deleting metadata bucket: eventing
2021/12/10 04:33:56 handler_test.go:925: Deleted metadata bucket: eventing
2021/12/10 04:34:11 clusterSetup.go:397: Waiting for app: TestMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/10 04:34:11 clusterSetup.go:405: App: TestMetaBucketDeleteWithBootstrap got un-deployed
2021/12/10 04:34:26 clusterSetup.go:397: Waiting for app: TestMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/10 04:34:26 clusterSetup.go:405: App: TestMetaBucketDeleteWithBootstrap got un-deployed
2021/12/10 04:34:26 eventingSetup.go:46: No eventing-con process running
2021/12/10 04:34:26 eventingSetup.go:418: Function: TestMetaBucketDeleteWithBootstrap delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMetaBucketDeleteWithBootstrap deleting in the background"
}
2021/12/10 04:34:27 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 04:34:32 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 04:34:34 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 04:34:39 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestMetaBucketDeleteWithBootstrap (86.71s)
=== RUN   TestSourceBucketDelete
2021/12/10 04:34:49 clusterSetup.go:397: Waiting for app: TestSourceBucketDelete to get un-deployed
2021/12/10 04:34:49 clusterSetup.go:405: App: TestSourceBucketDelete got un-deployed
2021/12/10 04:34:49 eventingSetup.go:46: No eventing-con process running
2021/12/10 04:34:49 eventingSetup.go:418: Function: TestSourceBucketDelete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestSourceBucketDelete not found"
 }
}
2021/12/10 04:34:50 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 04:34:55 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 04:34:57 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 04:35:02 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 04:35:02 eventingSetup.go:125: Deploying app: TestSourceBucketDelete. Item count src bucket: 0 dst bucket: 0
2021/12/10 04:35:03 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestSourceBucketDelete' in metakv",
  "warnings": null
 }
}
2021/12/10 04:35:08 clusterSetup.go:346: Waiting for app: TestSourceBucketDelete to get deployed
2021/12/10 04:35:13 clusterSetup.go:346: Waiting for app: TestSourceBucketDelete to get deployed
2021/12/10 04:35:18 clusterSetup.go:346: Waiting for app: TestSourceBucketDelete to get deployed
2021/12/10 04:35:23 clusterSetup.go:346: Waiting for app: TestSourceBucketDelete to get deployed
2021/12/10 04:35:23 clusterSetup.go:354: App: TestSourceBucketDelete got deployed
2021/12/10 04:35:23 handler_test.go:945: Deleting source bucket: default
2021/12/10 04:35:24 handler_test.go:947: Deleted source bucket: default
2021/12/10 04:35:30 clusterSetup.go:397: Waiting for app: TestSourceBucketDelete to get un-deployed
2021/12/10 04:35:30 clusterSetup.go:405: App: TestSourceBucketDelete got un-deployed
2021/12/10 04:35:45 clusterSetup.go:397: Waiting for app: TestSourceBucketDelete to get un-deployed
2021/12/10 04:35:45 clusterSetup.go:405: App: TestSourceBucketDelete got un-deployed
2021/12/10 04:35:45 eventingSetup.go:46: No eventing-con process running
2021/12/10 04:35:45 eventingSetup.go:418: Function: TestSourceBucketDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestSourceBucketDelete deleting in the background"
}
2021/12/10 04:35:46 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 04:35:51 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 04:35:53 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 04:35:58 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestSourceBucketDelete (79.18s)
=== RUN   TestSourceBucketDeleteWithBootstrap
2021/12/10 04:36:08 clusterSetup.go:397: Waiting for app: TestSourceBucketDeleteWithBootstrap to get un-deployed
2021/12/10 04:36:08 clusterSetup.go:405: App: TestSourceBucketDeleteWithBootstrap got un-deployed
2021/12/10 04:36:08 eventingSetup.go:46: No eventing-con process running
2021/12/10 04:36:08 eventingSetup.go:418: Function: TestSourceBucketDeleteWithBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestSourceBucketDeleteWithBootstrap not found"
 }
}
2021/12/10 04:36:09 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 04:36:14 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 04:36:16 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 04:36:21 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 04:36:21 eventingSetup.go:125: Deploying app: TestSourceBucketDeleteWithBootstrap. Item count src bucket: 0 dst bucket: 0
2021/12/10 04:36:22 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestSourceBucketDeleteWithBootstrap' in metakv",
  "warnings": null
 }
}
2021/12/10 04:36:41 handler_test.go:966: Deleting source bucket: default
2021/12/10 04:36:42 handler_test.go:968: Deleted source bucket: default
2021/12/10 04:36:57 clusterSetup.go:397: Waiting for app: TestSourceBucketDeleteWithBootstrap to get un-deployed
2021/12/10 04:36:57 clusterSetup.go:405: App: TestSourceBucketDeleteWithBootstrap got un-deployed
2021/12/10 04:37:12 clusterSetup.go:397: Waiting for app: TestSourceBucketDeleteWithBootstrap to get un-deployed
2021/12/10 04:37:12 clusterSetup.go:405: App: TestSourceBucketDeleteWithBootstrap got un-deployed
2021/12/10 04:37:12 eventingSetup.go:46: No eventing-con process running
2021/12/10 04:37:12 eventingSetup.go:418: Function: TestSourceBucketDeleteWithBootstrap delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestSourceBucketDeleteWithBootstrap deleting in the background"
}
2021/12/10 04:37:14 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 04:37:19 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 04:37:20 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 04:37:25 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestSourceBucketDeleteWithBootstrap (87.58s)
=== RUN   TestSourceAndMetaBucketDeleteWithBootstrap
2021/12/10 04:37:35 clusterSetup.go:397: Waiting for app: TestSourceAndMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/10 04:37:35 clusterSetup.go:405: App: TestSourceAndMetaBucketDeleteWithBootstrap got un-deployed
2021/12/10 04:37:35 eventingSetup.go:46: No eventing-con process running
2021/12/10 04:37:35 eventingSetup.go:418: Function: TestSourceAndMetaBucketDeleteWithBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestSourceAndMetaBucketDeleteWithBootstrap not found"
 }
}
2021/12/10 04:37:37 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 04:37:42 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 04:37:43 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 04:37:48 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 04:37:48 eventingSetup.go:125: Deploying app: TestSourceAndMetaBucketDeleteWithBootstrap. Item count src bucket: 0 dst bucket: 0
2021/12/10 04:37:50 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestSourceAndMetaBucketDeleteWithBootstrap' in metakv",
  "warnings": null
 }
}
2021/12/10 04:38:08 handler_test.go:988: Deleting source bucket: default
2021/12/10 04:38:10 handler_test.go:990: Deleted source bucket: default
2021/12/10 04:38:10 handler_test.go:992: Deleting metadata bucket: eventing
2021/12/10 04:38:11 handler_test.go:994: Deleted metadata bucket: eventing
2021/12/10 04:38:26 clusterSetup.go:397: Waiting for app: TestSourceAndMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/10 04:38:26 clusterSetup.go:405: App: TestSourceAndMetaBucketDeleteWithBootstrap got un-deployed
2021/12/10 04:38:42 clusterSetup.go:397: Waiting for app: TestSourceAndMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/10 04:38:42 clusterSetup.go:405: App: TestSourceAndMetaBucketDeleteWithBootstrap got un-deployed
2021/12/10 04:38:42 eventingSetup.go:46: No eventing-con process running
2021/12/10 04:38:42 eventingSetup.go:418: Function: TestSourceAndMetaBucketDeleteWithBootstrap delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestSourceAndMetaBucketDeleteWithBootstrap deleting in the background"
}
2021/12/10 04:38:43 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 04:38:48 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 04:38:50 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 04:38:55 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestSourceAndMetaBucketDeleteWithBootstrap (89.25s)
=== RUN   TestUndeployDuringBootstrap
2021/12/10 04:39:05 clusterSetup.go:397: Waiting for app: TestUndeployDuringBootstrap to get un-deployed
2021/12/10 04:39:05 clusterSetup.go:405: App: TestUndeployDuringBootstrap got un-deployed
2021/12/10 04:39:05 eventingSetup.go:46: No eventing-con process running
2021/12/10 04:39:05 eventingSetup.go:418: Function: TestUndeployDuringBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestUndeployDuringBootstrap not found"
 }
}
2021/12/10 04:39:06 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 04:39:11 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 04:39:13 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 04:39:18 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 04:39:18 eventingSetup.go:125: Deploying app: TestUndeployDuringBootstrap. Item count src bucket: 0 dst bucket: 0
2021/12/10 04:39:19 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestUndeployDuringBootstrap' in metakv",
  "warnings": null
 }
}
2021/12/10 04:39:19 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/10 04:39:31 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 04:39:31 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 04:39:31 clusterSetup.go:375: Apps undergoing bootstrap: true
2021/12/10 04:39:31 clusterSetup.go:378: App: TestUndeployDuringBootstrap started bootstrap
2021/12/10 04:39:31 clusterSetup.go:375: Apps undergoing bootstrap: true
2021/12/10 04:39:36 clusterSetup.go:367: No apps undergoing bootstrap
2021/12/10 04:39:41 clusterSetup.go:397: Waiting for app: TestUndeployDuringBootstrap to get un-deployed
2021/12/10 04:39:46 clusterSetup.go:397: Waiting for app: TestUndeployDuringBootstrap to get un-deployed
2021/12/10 04:39:51 clusterSetup.go:397: Waiting for app: TestUndeployDuringBootstrap to get un-deployed
2021/12/10 04:39:51 clusterSetup.go:405: App: TestUndeployDuringBootstrap got un-deployed
2021/12/10 04:39:51 eventingSetup.go:46: No eventing-con process running
2021/12/10 04:40:16 clusterSetup.go:397: Waiting for app: TestUndeployDuringBootstrap to get un-deployed
2021/12/10 04:40:16 clusterSetup.go:405: App: TestUndeployDuringBootstrap got un-deployed
2021/12/10 04:40:16 eventingSetup.go:46: No eventing-con process running
2021/12/10 04:40:16 eventingSetup.go:418: Function: TestUndeployDuringBootstrap delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestUndeployDuringBootstrap deleting in the background"
}
2021/12/10 04:40:17 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 04:40:27 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/10 04:40:27 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 04:40:29 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 04:40:39 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3883
2021/12/10 04:40:44 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1383
2021/12/10 04:40:44 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestUndeployDuringBootstrap (109.29s)
=== RUN   TestDeleteBeforeUndeploy
2021/12/10 04:40:49 eventingSetup.go:125: Deploying app: TestDeleteBeforeUndeploy. Item count src bucket: 0 dst bucket: 0
2021/12/10 04:40:50 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeleteBeforeUndeploy' in metakv",
  "warnings": null
 }
}
2021/12/10 04:40:55 clusterSetup.go:346: Waiting for app: TestDeleteBeforeUndeploy to get deployed
2021/12/10 04:41:00 clusterSetup.go:346: Waiting for app: TestDeleteBeforeUndeploy to get deployed
2021/12/10 04:41:05 clusterSetup.go:346: Waiting for app: TestDeleteBeforeUndeploy to get deployed
2021/12/10 04:41:10 clusterSetup.go:346: Waiting for app: TestDeleteBeforeUndeploy to get deployed
2021/12/10 04:41:10 clusterSetup.go:354: App: TestDeleteBeforeUndeploy got deployed
2021/12/10 04:41:10 eventingSetup.go:418: Function: TestDeleteBeforeUndeploy delete from main store request response code: 422 dump: {
 "name": "ERR_APP_NOT_UNDEPLOYED",
 "code": 22,
 "description": "Function hasn't undeployed",
 "attributes": null,
 "runtime_info": {
  "code": 22,
  "info": "Function: TestDeleteBeforeUndeploy skipping delete request from primary store, as it hasn't been undeployed"
 }
}
2021/12/10 04:41:10 eventingSetup.go:270: Function: TestDeleteBeforeUndeploy update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/10 04:41:10 eventingSetup.go:418: Function: TestDeleteBeforeUndeploy delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDeleteBeforeUndeploy deleting in the background"
}
2021/12/10 04:41:15 clusterSetup.go:397: Waiting for app: TestDeleteBeforeUndeploy to get un-deployed
2021/12/10 04:41:15 clusterSetup.go:405: App: TestDeleteBeforeUndeploy got un-deployed
2021/12/10 04:41:15 eventingSetup.go:46: No eventing-con process running
2021/12/10 04:41:15 eventingSetup.go:418: Function: TestDeleteBeforeUndeploy delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDeleteBeforeUndeploy not found"
 }
}
2021/12/10 04:41:17 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 04:41:22 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 04:41:24 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 04:41:29 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDeleteBeforeUndeploy (44.59s)
=== RUN   TestUndeployWhenTimersAreFired
2021/12/10 04:41:39 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/10 04:41:39 clusterSetup.go:405: App: TestUndeployWhenTimersAreFired got un-deployed
2021/12/10 04:41:39 eventingSetup.go:46: No eventing-con process running
2021/12/10 04:41:39 eventingSetup.go:418: Function: TestUndeployWhenTimersAreFired delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestUndeployWhenTimersAreFired not found"
 }
}
2021/12/10 04:41:40 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 04:41:45 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 04:41:47 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 04:41:52 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 04:41:52 eventingSetup.go:125: Deploying app: TestUndeployWhenTimersAreFired. Item count src bucket: 0 dst bucket: 0
2021/12/10 04:41:53 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestUndeployWhenTimersAreFired' in metakv",
  "warnings": null
 }
}
2021/12/10 04:41:58 clusterSetup.go:346: Waiting for app: TestUndeployWhenTimersAreFired to get deployed
2021/12/10 04:42:03 clusterSetup.go:346: Waiting for app: TestUndeployWhenTimersAreFired to get deployed
2021/12/10 04:42:08 clusterSetup.go:346: Waiting for app: TestUndeployWhenTimersAreFired to get deployed
2021/12/10 04:42:13 clusterSetup.go:346: Waiting for app: TestUndeployWhenTimersAreFired to get deployed
2021/12/10 04:42:13 clusterSetup.go:354: App: TestUndeployWhenTimersAreFired got deployed
2021/12/10 04:42:13 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/10 04:42:43 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 04:42:48 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/10 04:42:53 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/10 04:42:58 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/10 04:43:03 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/10 04:43:03 clusterSetup.go:405: App: TestUndeployWhenTimersAreFired got un-deployed
2021/12/10 04:43:03 eventingSetup.go:46: No eventing-con process running
2021/12/10 04:44:48 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/10 04:44:48 clusterSetup.go:405: App: TestUndeployWhenTimersAreFired got un-deployed
2021/12/10 04:44:48 eventingSetup.go:46: No eventing-con process running
2021/12/10 04:44:48 eventingSetup.go:418: Function: TestUndeployWhenTimersAreFired delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestUndeployWhenTimersAreFired deleting in the background"
}
2021/12/10 04:44:50 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 04:45:00 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/10 04:45:05 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/10 04:45:05 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 04:45:06 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 04:45:16 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/10 04:45:16 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestUndeployWhenTimersAreFired (227.88s)
=== RUN   TestUndeployWithKVFailover
2021/12/10 04:45:26 clusterSetup.go:397: Waiting for app: TestUndeployWithKVFailover to get un-deployed
2021/12/10 04:45:26 clusterSetup.go:405: App: TestUndeployWithKVFailover got un-deployed
2021/12/10 04:45:26 eventingSetup.go:46: No eventing-con process running
2021/12/10 04:45:26 eventingSetup.go:418: Function: TestUndeployWithKVFailover delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestUndeployWithKVFailover not found"
 }
}
2021/12/10 04:45:28 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 04:45:33 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 04:45:35 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 04:45:40 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 04:45:40 eventingSetup.go:125: Deploying app: TestUndeployWithKVFailover. Item count src bucket: 0 dst bucket: 0
2021/12/10 04:45:41 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestUndeployWithKVFailover' in metakv",
  "warnings": null
 }
}
2021/12/10 04:45:41 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/10 04:45:48 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 04:45:53 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 04:45:58 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 04:46:03 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 04:46:08 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 04:46:13 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 04:46:18 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 2471
2021/12/10 04:46:23 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 2471
2021/12/10 04:46:23 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/10 04:46:23 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: kv to the cluster
2021/12/10 04:46:23 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/10 04:46:28 Rebalance progress: 29.16666666666666
2021/12/10 04:46:33 Rebalance progress: 29.16666666666666
2021/12/10 04:46:38 Rebalance progress: 100
2021/12/10 04:46:38 Failing over: [127.0.0.1:19003]
2021/12/10 04:46:48 Function: TestUndeployWithKVFailover update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/10 04:47:48 Removing node(s): [] from the cluster
2021/12/10 04:47:53 Rebalance progress: 29.16666666666666
2021/12/10 04:47:58 Rebalance progress: 29.16666666666666
2021/12/10 04:48:03 Rebalance progress: 100
2021/12/10 04:48:08 Waiting for app: TestUndeployWithKVFailover to get un-deployed
2021/12/10 04:48:08 App: TestUndeployWithKVFailover got un-deployed
2021/12/10 04:48:08 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 04:48:13 Waiting for app: TestUndeployWithKVFailover to get un-deployed
2021/12/10 04:48:13 App: TestUndeployWithKVFailover got un-deployed
2021/12/10 04:48:13 No eventing-con process running
2021/12/10 04:48:13 Function: TestUndeployWithKVFailover delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestUndeployWithKVFailover deleting in the background"
}
2021/12/10 04:48:15 Context: Bucket flush, response code: 200 dump: 
2021/12/10 04:48:25 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/10 04:48:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 04:48:26 Context: Bucket flush, response code: 200 dump: 
2021/12/10 04:48:36 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/10 04:48:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestUndeployWithKVFailover (199.95s)
=== RUN   TestBucketFlushWhileFnDeployed
2021/12/10 04:48:46 Waiting for app: TestBucketFlushWhileFnDeployed to get un-deployed
2021/12/10 04:48:46 App: TestBucketFlushWhileFnDeployed got un-deployed
2021/12/10 04:48:46 No eventing-con process running
2021/12/10 04:48:46 Function: TestBucketFlushWhileFnDeployed delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestBucketFlushWhileFnDeployed not found"
 }
}
2021/12/10 04:48:48 Context: Bucket flush, response code: 200 dump: 
2021/12/10 04:48:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 04:48:54 Context: Bucket flush, response code: 200 dump: 
2021/12/10 04:48:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 04:48:59 Deploying app: TestBucketFlushWhileFnDeployed. Item count src bucket: 0 dst bucket: 0
2021/12/10 04:49:01 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestBucketFlushWhileFnDeployed' in metakv",
  "warnings": null
 }
}
2021/12/10 04:49:06 Waiting for app: TestBucketFlushWhileFnDeployed to get deployed
2021/12/10 04:49:11 Waiting for app: TestBucketFlushWhileFnDeployed to get deployed
2021/12/10 04:49:16 Waiting for app: TestBucketFlushWhileFnDeployed to get deployed
2021/12/10 04:49:21 Waiting for app: TestBucketFlushWhileFnDeployed to get deployed
2021/12/10 04:49:21 App: TestBucketFlushWhileFnDeployed got deployed
2021/12/10 04:49:21 Starting bucket ops to source bucket
2021/12/10 04:49:41 Waiting for dst bucket item count to get to: 20000 curr count: 9933
2021/12/10 04:49:41 src & dst bucket item count matched up. src bucket count: 20000 dst bucket count: 20000
2021/12/10 04:49:41 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 04:49:44 Context: Bucket flush, response code: 200 dump: 
2021/12/10 04:49:54 Waiting for dst bucket item count to get to: 0 curr count: 20000
2021/12/10 04:49:59 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/10 04:49:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 04:50:00 Context: Bucket flush, response code: 200 dump: 
2021/12/10 04:50:10 Waiting for dst bucket item count to get to: 0 curr count: 15418
2021/12/10 04:50:15 Waiting for dst bucket item count to get to: 0 curr count: 15418
2021/12/10 04:50:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 04:50:15 Starting bucket ops to source bucket
2021/12/10 04:50:28 Waiting for dst bucket item count to get to: 10000 curr count: 3714
2021/12/10 04:50:33 Waiting for dst bucket item count to get to: 10000 curr count: 3714
2021/12/10 04:50:33 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/10 04:50:33 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 04:50:33 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 04:50:38 Waiting for app: TestBucketFlushWhileFnDeployed to get un-deployed
2021/12/10 04:50:43 Waiting for app: TestBucketFlushWhileFnDeployed to get un-deployed
2021/12/10 04:50:48 Waiting for app: TestBucketFlushWhileFnDeployed to get un-deployed
2021/12/10 04:50:53 Waiting for app: TestBucketFlushWhileFnDeployed to get un-deployed
2021/12/10 04:50:53 App: TestBucketFlushWhileFnDeployed got un-deployed
2021/12/10 04:50:53 No eventing-con process running
2021/12/10 04:50:53 Function: TestBucketFlushWhileFnDeployed delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestBucketFlushWhileFnDeployed deleting in the background"
}
2021/12/10 04:50:54 Context: Bucket flush, response code: 200 dump: 
2021/12/10 04:51:04 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/10 04:51:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 04:51:06 Context: Bucket flush, response code: 200 dump: 
2021/12/10 04:51:16 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/10 04:51:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestBucketFlushWhileFnDeployed (159.88s)
=== RUN   TestTimerOverwrite
2021/12/10 04:51:26 Waiting for app: TestTimerOverwrite to get un-deployed
2021/12/10 04:51:26 App: TestTimerOverwrite got un-deployed
2021/12/10 04:51:26 No eventing-con process running
2021/12/10 04:51:26 Function: TestTimerOverwrite delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestTimerOverwrite not found"
 }
}
2021/12/10 04:51:28 Context: Bucket flush, response code: 200 dump: 
2021/12/10 04:51:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 04:51:34 Context: Bucket flush, response code: 200 dump: 
2021/12/10 04:51:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 04:51:39 Deploying app: TestTimerOverwrite. Item count src bucket: 0 dst bucket: 0
2021/12/10 04:51:41 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestTimerOverwrite' in metakv",
  "warnings": null
 }
}
2021/12/10 04:51:46 Waiting for app: TestTimerOverwrite to get deployed
2021/12/10 04:51:51 Waiting for app: TestTimerOverwrite to get deployed
2021/12/10 04:51:56 Waiting for app: TestTimerOverwrite to get deployed
2021/12/10 04:52:01 Waiting for app: TestTimerOverwrite to get deployed
2021/12/10 04:52:01 App: TestTimerOverwrite got deployed
2021/12/10 04:52:11 Starting bucket ops to source bucket
2021/12/10 04:52:19 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 04:52:24 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 04:52:29 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 04:52:34 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 04:52:39 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 04:52:44 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 04:52:49 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 04:52:54 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 04:52:59 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 04:53:04 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 04:53:09 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 04:53:14 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 04:53:14 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/10 04:53:25 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 04:53:25 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 04:53:30 Waiting for app: TestTimerOverwrite to get un-deployed
2021/12/10 04:53:35 Waiting for app: TestTimerOverwrite to get un-deployed
2021/12/10 04:53:40 Waiting for app: TestTimerOverwrite to get un-deployed
2021/12/10 04:53:45 Waiting for app: TestTimerOverwrite to get un-deployed
2021/12/10 04:53:45 App: TestTimerOverwrite got un-deployed
2021/12/10 04:53:45 No eventing-con process running
2021/12/10 04:53:45 Function: TestTimerOverwrite delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestTimerOverwrite deleting in the background"
}
2021/12/10 04:53:46 Context: Bucket flush, response code: 200 dump: 
2021/12/10 04:53:56 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/10 04:53:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 04:53:58 Context: Bucket flush, response code: 200 dump: 
2021/12/10 04:54:08 Waiting for dst bucket item count to get to: 0 curr count: 2682
2021/12/10 04:54:13 Waiting for dst bucket item count to get to: 0 curr count: 177
2021/12/10 04:54:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestTimerOverwrite (176.42s)
=== RUN   TestUndeployBackdoorDuringBootstrap
2021/12/10 04:54:18 Adding node: https://127.0.0.1:19003 with role: kv to the cluster
2021/12/10 04:54:18 Removing node(s): [] from the cluster
2021/12/10 04:54:23 Rebalance progress: 100
2021/12/10 04:54:23 Failing over: [127.0.0.1:19003]
2021/12/10 04:54:28 Waiting for app: TestUndeployBackdoorDuringBootstrap to get un-deployed
2021/12/10 04:54:28 App: TestUndeployBackdoorDuringBootstrap got un-deployed
2021/12/10 04:54:28 No eventing-con process running
2021/12/10 04:54:28 Function: TestUndeployBackdoorDuringBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestUndeployBackdoorDuringBootstrap not found"
 }
}
2021/12/10 04:54:29 Context: Bucket flush, response code: 200 dump: 
2021/12/10 04:54:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 04:54:36 Context: Bucket flush, response code: 200 dump: 
2021/12/10 04:54:41 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 04:54:41 Deploying app: TestUndeployBackdoorDuringBootstrap. Item count src bucket: 0 dst bucket: 0
2021/12/10 04:54:42 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestUndeployBackdoorDuringBootstrap' in metakv",
  "warnings": null
 }
}
2021/12/10 04:54:42 Starting bucket ops to source bucket
2021/12/10 04:54:52 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 04:54:52 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 04:54:52 Context: Post to set retry counter, response code: 200 dump: 
2021/12/10 04:54:52 set retry counter, response &{[] }
2021/12/10 04:55:57 Waiting for app: TestUndeployBackdoorDuringBootstrap to get un-deployed
2021/12/10 04:55:57 App: TestUndeployBackdoorDuringBootstrap got un-deployed
2021/12/10 04:55:57 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 04:55:57 Function: TestUndeployBackdoorDuringBootstrap delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestUndeployBackdoorDuringBootstrap deleting in the background"
}
2021/12/10 04:56:02 Waiting for app: TestUndeployBackdoorDuringBootstrap to get un-deployed
2021/12/10 04:56:02 App: TestUndeployBackdoorDuringBootstrap got un-deployed
2021/12/10 04:56:03 No eventing-con process running
2021/12/10 04:56:03 Function: TestUndeployBackdoorDuringBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestUndeployBackdoorDuringBootstrap not found"
 }
}
2021/12/10 04:56:04 Context: Bucket flush, response code: 200 dump: 
2021/12/10 04:56:14 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/10 04:56:19 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/10 04:56:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 04:56:20 Context: Bucket flush, response code: 200 dump: 
2021/12/10 04:56:30 Waiting for dst bucket item count to get to: 0 curr count: 2908
2021/12/10 04:56:35 Waiting for dst bucket item count to get to: 0 curr count: 2908
2021/12/10 04:56:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 04:56:35 Removing node(s): [] from the cluster
2021/12/10 04:56:41 Rebalance progress: 100
--- PASS: TestUndeployBackdoorDuringBootstrap (147.85s)
=== RUN   TestOnUpdateSrcMutation
2021/12/10 04:56:51 Waiting for app: TestOnUpdateSrcMutation to get un-deployed
2021/12/10 04:56:51 App: TestOnUpdateSrcMutation got un-deployed
2021/12/10 04:56:51 No eventing-con process running
2021/12/10 04:56:51 Function: TestOnUpdateSrcMutation delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateSrcMutation not found"
 }
}
2021/12/10 04:56:52 Context: Bucket flush, response code: 200 dump: 
2021/12/10 04:56:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 04:56:59 Context: Bucket flush, response code: 200 dump: 
2021/12/10 04:57:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 04:57:04 Deploying app: TestOnUpdateSrcMutation. Item count src bucket: 0 dst bucket: 0
2021/12/10 04:57:05 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateSrcMutation' in metakv",
  "warnings": null
 }
}
2021/12/10 04:57:05 Starting bucket ops to source bucket
2021/12/10 04:57:32 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/10 04:57:32 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 04:57:32 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 04:57:37 Waiting for app: TestOnUpdateSrcMutation to get un-deployed
2021/12/10 04:57:42 Waiting for app: TestOnUpdateSrcMutation to get un-deployed
2021/12/10 04:57:47 Waiting for app: TestOnUpdateSrcMutation to get un-deployed
2021/12/10 04:57:52 Waiting for app: TestOnUpdateSrcMutation to get un-deployed
2021/12/10 04:57:52 App: TestOnUpdateSrcMutation got un-deployed
2021/12/10 04:57:52 No eventing-con process running
2021/12/10 04:57:52 Function: TestOnUpdateSrcMutation delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateSrcMutation deleting in the background"
}
2021/12/10 04:57:53 Context: Bucket flush, response code: 200 dump: 
2021/12/10 04:58:03 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/10 04:58:08 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/10 04:58:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 04:58:10 Context: Bucket flush, response code: 200 dump: 
2021/12/10 04:58:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateSrcMutation (94.42s)
=== RUN   TestOnDeleteSrcMutation
2021/12/10 04:58:25 Waiting for app: TestOnDeleteSrcMutation to get un-deployed
2021/12/10 04:58:25 App: TestOnDeleteSrcMutation got un-deployed
2021/12/10 04:58:25 No eventing-con process running
2021/12/10 04:58:25 Function: TestOnDeleteSrcMutation delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnDeleteSrcMutation not found"
 }
}
2021/12/10 04:58:26 Context: Bucket flush, response code: 200 dump: 
2021/12/10 04:58:31 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 04:58:33 Context: Bucket flush, response code: 200 dump: 
2021/12/10 04:58:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 04:58:38 Deploying app: TestOnDeleteSrcMutation. Item count src bucket: 0 dst bucket: 0
2021/12/10 04:58:39 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnDeleteSrcMutation' in metakv",
  "warnings": null
 }
}
2021/12/10 04:58:39 Starting bucket ops to source bucket
2021/12/10 04:59:08 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/10 04:59:08 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 04:59:08 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 04:59:13 Waiting for app: TestOnDeleteSrcMutation to get un-deployed
2021/12/10 04:59:18 Waiting for app: TestOnDeleteSrcMutation to get un-deployed
2021/12/10 04:59:23 Waiting for app: TestOnDeleteSrcMutation to get un-deployed
2021/12/10 04:59:28 Waiting for app: TestOnDeleteSrcMutation to get un-deployed
2021/12/10 04:59:28 App: TestOnDeleteSrcMutation got un-deployed
2021/12/10 04:59:28 No eventing-con process running
2021/12/10 04:59:28 Function: TestOnDeleteSrcMutation delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnDeleteSrcMutation deleting in the background"
}
2021/12/10 04:59:29 Context: Bucket flush, response code: 200 dump: 
2021/12/10 04:59:39 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/10 04:59:44 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/10 04:59:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 04:59:46 Context: Bucket flush, response code: 200 dump: 
2021/12/10 04:59:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnDeleteSrcMutation (96.14s)
=== RUN   TestOnUpdateSrcMutationWithTimer
2021/12/10 05:00:01 Waiting for app: TestOnUpdateSrcMutationWithTimer to get un-deployed
2021/12/10 05:00:01 App: TestOnUpdateSrcMutationWithTimer got un-deployed
2021/12/10 05:00:01 No eventing-con process running
2021/12/10 05:00:01 Function: TestOnUpdateSrcMutationWithTimer delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateSrcMutationWithTimer not found"
 }
}
2021/12/10 05:00:02 Context: Bucket flush, response code: 200 dump: 
2021/12/10 05:00:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 05:00:09 Context: Bucket flush, response code: 200 dump: 
2021/12/10 05:00:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 05:00:14 Deploying app: TestOnUpdateSrcMutationWithTimer. Item count src bucket: 0 dst bucket: 0
2021/12/10 05:00:15 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateSrcMutationWithTimer' in metakv",
  "warnings": null
 }
}
2021/12/10 05:00:15 Starting bucket ops to source bucket
2021/12/10 05:00:52 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/10 05:00:52 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 05:00:52 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 05:00:57 Waiting for app: TestOnUpdateSrcMutationWithTimer to get un-deployed
2021/12/10 05:01:02 Waiting for app: TestOnUpdateSrcMutationWithTimer to get un-deployed
2021/12/10 05:01:07 Waiting for app: TestOnUpdateSrcMutationWithTimer to get un-deployed
2021/12/10 05:01:12 Waiting for app: TestOnUpdateSrcMutationWithTimer to get un-deployed
2021/12/10 05:01:12 App: TestOnUpdateSrcMutationWithTimer got un-deployed
2021/12/10 05:01:12 No eventing-con process running
2021/12/10 05:01:13 Function: TestOnUpdateSrcMutationWithTimer delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateSrcMutationWithTimer deleting in the background"
}
2021/12/10 05:01:14 Context: Bucket flush, response code: 200 dump: 
2021/12/10 05:01:24 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/10 05:01:29 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/10 05:01:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 05:01:30 Context: Bucket flush, response code: 200 dump: 
2021/12/10 05:01:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateSrcMutationWithTimer (104.40s)
=== RUN   TestOnDeleteSrcMutationsWithTimer
2021/12/10 05:01:45 Waiting for app: TestOnDeleteSrcMutationsWithTimer to get un-deployed
2021/12/10 05:01:45 App: TestOnDeleteSrcMutationsWithTimer got un-deployed
2021/12/10 05:01:46 No eventing-con process running
2021/12/10 05:01:46 Function: TestOnDeleteSrcMutationsWithTimer delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnDeleteSrcMutationsWithTimer not found"
 }
}
2021/12/10 05:01:47 Context: Bucket flush, response code: 200 dump: 
2021/12/10 05:01:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 05:01:54 Context: Bucket flush, response code: 200 dump: 
2021/12/10 05:01:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 05:01:59 Deploying app: TestOnDeleteSrcMutationsWithTimer. Item count src bucket: 0 dst bucket: 0
2021/12/10 05:02:00 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnDeleteSrcMutationsWithTimer' in metakv",
  "warnings": null
 }
}
2021/12/10 05:02:00 Starting bucket ops to source bucket
2021/12/10 05:02:43 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/10 05:02:43 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 05:02:44 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 05:02:49 Waiting for app: TestOnDeleteSrcMutationsWithTimer to get un-deployed
2021/12/10 05:02:54 Waiting for app: TestOnDeleteSrcMutationsWithTimer to get un-deployed
2021/12/10 05:02:59 Waiting for app: TestOnDeleteSrcMutationsWithTimer to get un-deployed
2021/12/10 05:03:04 Waiting for app: TestOnDeleteSrcMutationsWithTimer to get un-deployed
2021/12/10 05:03:04 App: TestOnDeleteSrcMutationsWithTimer got un-deployed
2021/12/10 05:03:04 No eventing-con process running
2021/12/10 05:03:04 Function: TestOnDeleteSrcMutationsWithTimer delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnDeleteSrcMutationsWithTimer deleting in the background"
}
2021/12/10 05:03:05 Context: Bucket flush, response code: 200 dump: 
2021/12/10 05:03:15 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/10 05:03:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 05:03:17 Context: Bucket flush, response code: 200 dump: 
2021/12/10 05:03:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnDeleteSrcMutationsWithTimer (106.21s)
=== RUN   TestInterHandlerRecursion
2021/12/10 05:03:32 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/10 05:03:32 App: TestInterHandlerRecursion_source_bucket_update_op got un-deployed
2021/12/10 05:03:32 No eventing-con process running
2021/12/10 05:03:32 Function: TestInterHandlerRecursion_source_bucket_update_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestInterHandlerRecursion_source_bucket_update_op not found"
 }
}
2021/12/10 05:03:33 Context: Bucket flush, response code: 200 dump: 
2021/12/10 05:03:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 05:03:40 Context: Bucket flush, response code: 200 dump: 
2021/12/10 05:03:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 05:03:50 Waiting for app: TestInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/10 05:03:50 App: TestInterHandlerRecursion_src_bucket_op_on_delete got un-deployed
2021/12/10 05:03:50 No eventing-con process running
2021/12/10 05:03:50 Function: TestInterHandlerRecursion_src_bucket_op_on_delete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestInterHandlerRecursion_src_bucket_op_on_delete not found"
 }
}
2021/12/10 05:03:51 Context: Bucket flush, response code: 200 dump: 
2021/12/10 05:03:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 05:03:58 Context: Bucket flush, response code: 200 dump: 
2021/12/10 05:04:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 05:04:03 Deploying app: TestInterHandlerRecursion_source_bucket_update_op. Item count src bucket: 0 dst bucket: 0
2021/12/10 05:04:04 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterHandlerRecursion_source_bucket_update_op' in metakv",
  "warnings": null
 }
}
2021/12/10 05:04:04 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterHandlerRecursion_source_bucket_update_op' in metakv",
  "warnings": null
 }
} err 
2021/12/10 05:04:09 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/10 05:04:14 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/10 05:04:19 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/10 05:04:24 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/10 05:04:24 App: TestInterHandlerRecursion_source_bucket_update_op got deployed
2021/12/10 05:04:24 Deploying app: TestInterHandlerRecursion_src_bucket_op_on_delete. Item count src bucket: 0 dst bucket: 0
2021/12/10 05:04:24 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_INTER_FUNCTION_RECURSION",
 "code": 50,
 "description": "Inter function recursion error, only one function is allowed to do source bucket mutation/delete on a bucket",
 "attributes": null,
 "runtime_info": {
  "code": 50,
  "info": "Inter handler recursion error"
 }
}
2021/12/10 05:04:24 response body {
 "name": "ERR_INTER_FUNCTION_RECURSION",
 "code": 50,
 "description": "Inter function recursion error, only one function is allowed to do source bucket mutation/delete on a bucket",
 "attributes": null,
 "runtime_info": {
  "code": 50,
  "info": "Inter handler recursion error"
 }
} err 
2021/12/10 05:04:24 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 05:04:29 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/10 05:04:34 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/10 05:04:39 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/10 05:04:44 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/10 05:04:44 App: TestInterHandlerRecursion_source_bucket_update_op got un-deployed
2021/12/10 05:04:44 No eventing-con process running
2021/12/10 05:04:44 Function: TestInterHandlerRecursion_source_bucket_update_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestInterHandlerRecursion_source_bucket_update_op deleting in the background"
}
2021/12/10 05:04:45 Context: Bucket flush, response code: 200 dump: 
2021/12/10 05:04:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 05:04:52 Context: Bucket flush, response code: 200 dump: 
2021/12/10 05:04:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 05:05:02 Waiting for app: TestInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/10 05:05:02 App: TestInterHandlerRecursion_src_bucket_op_on_delete got un-deployed
2021/12/10 05:05:02 No eventing-con process running
2021/12/10 05:05:02 Function: TestInterHandlerRecursion_src_bucket_op_on_delete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestInterHandlerRecursion_src_bucket_op_on_delete not found"
 }
}
2021/12/10 05:05:04 Context: Bucket flush, response code: 200 dump: 
2021/12/10 05:05:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 05:05:10 Context: Bucket flush, response code: 200 dump: 
2021/12/10 05:05:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestInterHandlerRecursion (113.55s)
=== RUN   TestInterBucketRecursion
2021/12/10 05:05:20 Deploying app: TestInterBucketRecursion_function1. Item count src bucket: 0 dst bucket: 0
2021/12/10 05:05:21 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterBucketRecursion_function1' in metakv",
  "warnings": null
 }
}
2021/12/10 05:05:21 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterBucketRecursion_function1' in metakv",
  "warnings": null
 }
} err 
2021/12/10 05:05:21 Deploying app: TestInterBucketRecursion_function2. Item count src bucket: 0 dst bucket: 0
2021/12/10 05:05:21 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_INTER_BUCKET_RECURSION",
 "code": 51,
 "description": "Inter bucket recursion error, deployment of current handler will cause inter bucket recursion",
 "attributes": null,
 "runtime_info": {
  "code": 51,
  "info": "Inter bucket recursion error; function: TestInterBucketRecursion_function2 causes a cycle involving functions: [TestInterBucketRecursion_function1], hence deployment is disallowed"
 }
}
2021/12/10 05:05:26 Waiting for app: TestInterBucketRecursion_function1 to get deployed
2021/12/10 05:05:31 Waiting for app: TestInterBucketRecursion_function1 to get deployed
2021/12/10 05:05:36 Waiting for app: TestInterBucketRecursion_function1 to get deployed
2021/12/10 05:05:41 Waiting for app: TestInterBucketRecursion_function1 to get deployed
2021/12/10 05:05:41 App: TestInterBucketRecursion_function1 got deployed
2021/12/10 05:05:42 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 05:05:47 Waiting for app: TestInterBucketRecursion_function1 to get un-deployed
2021/12/10 05:05:52 Waiting for app: TestInterBucketRecursion_function1 to get un-deployed
2021/12/10 05:05:57 Waiting for app: TestInterBucketRecursion_function1 to get un-deployed
2021/12/10 05:06:02 Waiting for app: TestInterBucketRecursion_function1 to get un-deployed
2021/12/10 05:06:02 App: TestInterBucketRecursion_function1 got un-deployed
2021/12/10 05:06:02 No eventing-con process running
2021/12/10 05:06:02 Function: TestInterBucketRecursion_function1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestInterBucketRecursion_function1 deleting in the background"
}
2021/12/10 05:06:03 Context: Bucket flush, response code: 200 dump: 
2021/12/10 05:06:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 05:06:10 Context: Bucket flush, response code: 200 dump: 
2021/12/10 05:06:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 05:06:20 Waiting for app: TestInterBucketRecursion_function2 to get un-deployed
2021/12/10 05:06:20 App: TestInterBucketRecursion_function2 got un-deployed
2021/12/10 05:06:20 No eventing-con process running
2021/12/10 05:06:20 Function: TestInterBucketRecursion_function2 delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestInterBucketRecursion_function2 not found"
 }
}
2021/12/10 05:06:21 Context: Bucket flush, response code: 200 dump: 
2021/12/10 05:06:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 05:06:28 Context: Bucket flush, response code: 200 dump: 
2021/12/10 05:06:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestInterBucketRecursion (77.46s)
=== RUN   TestLargeHandler
2021/12/10 05:06:33 Changed force_compress value to false
2021/12/10 05:06:33 Deploying app: TestLargeHandler_function. Item count src bucket: 0 dst bucket: 0
2021/12/10 05:06:33 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_APPCODE_SIZE",
 "code": 39,
 "description": "Function Code size is more than the configured limit",
 "attributes": null,
 "runtime_info": {
  "code": 39,
  "info": "Function: TestLargeHandler_function handler Code size is more than 131072. Code Size: 131768"
 }
}
2021/12/10 05:06:33 Changed force_compress value to true
2021/12/10 05:06:33 Deploying app: TestLargeHandler_function. Item count src bucket: 0 dst bucket: 0
2021/12/10 05:06:35 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestLargeHandler_function' in metakv",
  "warnings": null
 }
}
2021/12/10 05:06:40 Waiting for app: TestLargeHandler_function to get deployed
2021/12/10 05:06:45 Waiting for app: TestLargeHandler_function to get deployed
2021/12/10 05:06:50 Waiting for app: TestLargeHandler_function to get deployed
2021/12/10 05:06:55 Waiting for app: TestLargeHandler_function to get deployed
2021/12/10 05:06:55 App: TestLargeHandler_function got deployed
2021/12/10 05:06:55 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 05:07:00 Waiting for app: TestLargeHandler_function to get un-deployed
2021/12/10 05:07:05 Waiting for app: TestLargeHandler_function to get un-deployed
2021/12/10 05:07:10 Waiting for app: TestLargeHandler_function to get un-deployed
2021/12/10 05:07:15 Waiting for app: TestLargeHandler_function to get un-deployed
2021/12/10 05:07:15 App: TestLargeHandler_function got un-deployed
2021/12/10 05:07:15 No eventing-con process running
2021/12/10 05:07:15 Function: TestLargeHandler_function delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestLargeHandler_function deleting in the background"
}
2021/12/10 05:07:16 Context: Bucket flush, response code: 200 dump: 
2021/12/10 05:07:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 05:07:23 Context: Bucket flush, response code: 200 dump: 
2021/12/10 05:07:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestLargeHandler (55.37s)
=== RUN   TestAllowInterHandlerRecursion
2021/12/10 05:07:38 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/10 05:07:38 App: TestAllowInterHandlerRecursion_source_bucket_update_op got un-deployed
2021/12/10 05:07:38 No eventing-con process running
2021/12/10 05:07:38 Function: TestAllowInterHandlerRecursion_source_bucket_update_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestAllowInterHandlerRecursion_source_bucket_update_op not found"
 }
}
2021/12/10 05:07:39 Context: Bucket flush, response code: 200 dump: 
2021/12/10 05:07:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 05:07:46 Context: Bucket flush, response code: 200 dump: 
2021/12/10 05:07:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 05:07:56 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/10 05:07:56 App: TestAllowInterHandlerRecursion_src_bucket_op_on_delete got un-deployed
2021/12/10 05:07:56 No eventing-con process running
2021/12/10 05:07:56 Function: TestAllowInterHandlerRecursion_src_bucket_op_on_delete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestAllowInterHandlerRecursion_src_bucket_op_on_delete not found"
 }
}
2021/12/10 05:07:57 Context: Bucket flush, response code: 200 dump: 
2021/12/10 05:08:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 05:08:04 Context: Bucket flush, response code: 200 dump: 
2021/12/10 05:08:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 05:08:09 Deploying app: TestAllowInterHandlerRecursion_source_bucket_update_op. Item count src bucket: 0 dst bucket: 0
2021/12/10 05:08:10 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterHandlerRecursion_source_bucket_update_op' in metakv",
  "warnings": null
 }
}
2021/12/10 05:08:10 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterHandlerRecursion_source_bucket_update_op' in metakv",
  "warnings": null
 }
} err 
2021/12/10 05:08:15 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/10 05:08:20 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/10 05:08:25 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/10 05:08:30 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/10 05:08:30 App: TestAllowInterHandlerRecursion_source_bucket_update_op got deployed
2021/12/10 05:08:30 Deploying app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete. Item count src bucket: 0 dst bucket: 0
2021/12/10 05:08:32 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterHandlerRecursion_src_bucket_op_on_delete' in metakv",
  "warnings": null
 }
}
2021/12/10 05:08:32 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterHandlerRecursion_src_bucket_op_on_delete' in metakv",
  "warnings": null
 }
} err 
2021/12/10 05:08:37 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get deployed
2021/12/10 05:08:42 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get deployed
2021/12/10 05:08:47 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get deployed
2021/12/10 05:08:52 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get deployed
2021/12/10 05:08:52 App: TestAllowInterHandlerRecursion_src_bucket_op_on_delete got deployed
2021/12/10 05:08:52 TestAllowInterHandlerRecursion_src_bucket_op_on_delete is deployed
2021/12/10 05:08:52 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 05:08:57 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/10 05:09:02 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/10 05:09:07 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/10 05:09:12 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/10 05:09:12 App: TestAllowInterHandlerRecursion_source_bucket_update_op got un-deployed
2021/12/10 05:09:12 No eventing-con process running
2021/12/10 05:09:12 Function: TestAllowInterHandlerRecursion_source_bucket_update_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAllowInterHandlerRecursion_source_bucket_update_op deleting in the background"
}
2021/12/10 05:09:14 Context: Bucket flush, response code: 200 dump: 
2021/12/10 05:09:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 05:09:21 Context: Bucket flush, response code: 200 dump: 
2021/12/10 05:09:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 05:09:26 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 05:09:31 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/10 05:09:36 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/10 05:09:41 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/10 05:09:46 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/10 05:09:46 App: TestAllowInterHandlerRecursion_src_bucket_op_on_delete got un-deployed
2021/12/10 05:09:46 No eventing-con process running
2021/12/10 05:09:46 Function: TestAllowInterHandlerRecursion_src_bucket_op_on_delete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAllowInterHandlerRecursion_src_bucket_op_on_delete deleting in the background"
}
2021/12/10 05:09:47 Context: Bucket flush, response code: 200 dump: 
2021/12/10 05:09:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 05:09:54 Context: Bucket flush, response code: 200 dump: 
2021/12/10 05:09:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAllowInterHandlerRecursion (151.05s)
=== RUN   TestAllowInterBucketRecursion
2021/12/10 05:10:04 Deploying app: TestAllowInterBucketRecursion_function1. Item count src bucket: 0 dst bucket: 0
2021/12/10 05:10:05 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterBucketRecursion_function1' in metakv",
  "warnings": null
 }
}
2021/12/10 05:10:05 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterBucketRecursion_function1' in metakv",
  "warnings": null
 }
} err 
2021/12/10 05:10:10 Waiting for app: TestAllowInterBucketRecursion_function1 to get deployed
2021/12/10 05:10:15 Waiting for app: TestAllowInterBucketRecursion_function1 to get deployed
2021/12/10 05:10:20 Waiting for app: TestAllowInterBucketRecursion_function1 to get deployed
2021/12/10 05:10:25 Waiting for app: TestAllowInterBucketRecursion_function1 to get deployed
2021/12/10 05:10:25 App: TestAllowInterBucketRecursion_function1 got deployed
2021/12/10 05:10:25 Deploying app: TestAllowInterBucketRecursion_function2. Item count src bucket: 0 dst bucket: 0
2021/12/10 05:10:26 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterBucketRecursion_function2' in metakv",
  "warnings": null
 }
}
2021/12/10 05:10:26 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterBucketRecursion_function2' in metakv",
  "warnings": null
 }
} err 
2021/12/10 05:10:31 Waiting for app: TestAllowInterBucketRecursion_function2 to get deployed
2021/12/10 05:10:36 Waiting for app: TestAllowInterBucketRecursion_function2 to get deployed
2021/12/10 05:10:41 Waiting for app: TestAllowInterBucketRecursion_function2 to get deployed
2021/12/10 05:10:47 Waiting for app: TestAllowInterBucketRecursion_function2 to get deployed
2021/12/10 05:10:47 App: TestAllowInterBucketRecursion_function2 got deployed
2021/12/10 05:10:47 TestAllowInterBucketRecursion_function2 is deployed
2021/12/10 05:10:47 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 05:10:52 Waiting for app: TestAllowInterBucketRecursion_function1 to get un-deployed
2021/12/10 05:10:57 Waiting for app: TestAllowInterBucketRecursion_function1 to get un-deployed
2021/12/10 05:11:02 Waiting for app: TestAllowInterBucketRecursion_function1 to get un-deployed
2021/12/10 05:11:07 Waiting for app: TestAllowInterBucketRecursion_function1 to get un-deployed
2021/12/10 05:11:07 App: TestAllowInterBucketRecursion_function1 got un-deployed
2021/12/10 05:11:07 No eventing-con process running
2021/12/10 05:11:07 Function: TestAllowInterBucketRecursion_function1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAllowInterBucketRecursion_function1 deleting in the background"
}
2021/12/10 05:11:08 Context: Bucket flush, response code: 200 dump: 
2021/12/10 05:11:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 05:11:15 Context: Bucket flush, response code: 200 dump: 
2021/12/10 05:11:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 05:11:21 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 05:11:26 Waiting for app: TestAllowInterBucketRecursion_function2 to get un-deployed
2021/12/10 05:11:31 Waiting for app: TestAllowInterBucketRecursion_function2 to get un-deployed
2021/12/10 05:11:36 Waiting for app: TestAllowInterBucketRecursion_function2 to get un-deployed
2021/12/10 05:11:41 Waiting for app: TestAllowInterBucketRecursion_function2 to get un-deployed
2021/12/10 05:11:41 App: TestAllowInterBucketRecursion_function2 got un-deployed
2021/12/10 05:11:41 No eventing-con process running
2021/12/10 05:11:41 Function: TestAllowInterBucketRecursion_function2 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAllowInterBucketRecursion_function2 deleting in the background"
}
2021/12/10 05:11:42 Context: Bucket flush, response code: 200 dump: 
2021/12/10 05:11:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 05:11:49 Context: Bucket flush, response code: 200 dump: 
2021/12/10 05:11:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAllowInterBucketRecursion (114.57s)
=== RUN   TestInterBucketRecursion2
2021/12/10 05:12:00 Context: Bucket flush, response code: 200 dump: 
2021/12/10 05:12:06 Context: Bucket flush, response code: 200 dump: 
2021/12/10 05:12:12 Context: Bucket flush, response code: 200 dump: 
2021/12/10 05:12:18 Context: Bucket flush, response code: 200 dump: 
2021/12/10 05:12:23 Deploying app: TestInterBucketRecursion2_function1. Item count src bucket: 0 dst bucket: 0
2021/12/10 05:12:24 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterBucketRecursion2_function1' in metakv",
  "warnings": null
 }
}
2021/12/10 05:12:29 Waiting for app: TestInterBucketRecursion2_function1 to get deployed
2021/12/10 05:12:34 Waiting for app: TestInterBucketRecursion2_function1 to get deployed
2021/12/10 05:12:39 Waiting for app: TestInterBucketRecursion2_function1 to get deployed
2021/12/10 05:12:44 Waiting for app: TestInterBucketRecursion2_function1 to get deployed
2021/12/10 05:12:44 App: TestInterBucketRecursion2_function1 got deployed
2021/12/10 05:12:44 Deploying app: TestInterBucketRecursion2_function2. Item count src bucket: 0 dst bucket: 0
2021/12/10 05:12:45 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterBucketRecursion2_function2' in metakv",
  "warnings": null
 }
}
2021/12/10 05:12:50 Waiting for app: TestInterBucketRecursion2_function2 to get deployed
2021/12/10 05:12:55 Waiting for app: TestInterBucketRecursion2_function2 to get deployed
2021/12/10 05:13:00 Waiting for app: TestInterBucketRecursion2_function2 to get deployed
2021/12/10 05:13:05 Waiting for app: TestInterBucketRecursion2_function2 to get deployed
2021/12/10 05:13:05 App: TestInterBucketRecursion2_function2 got deployed
2021/12/10 05:13:05 Deploying app: TestInterBucketRecursion2_function3. Item count src bucket: 0 dst bucket: 0
2021/12/10 05:13:06 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterBucketRecursion2_function3' in metakv",
  "warnings": null
 }
}
2021/12/10 05:13:11 Waiting for app: TestInterBucketRecursion2_function3 to get deployed
2021/12/10 05:13:17 Waiting for app: TestInterBucketRecursion2_function3 to get deployed
2021/12/10 05:13:22 Waiting for app: TestInterBucketRecursion2_function3 to get deployed
2021/12/10 05:13:27 Waiting for app: TestInterBucketRecursion2_function3 to get deployed
2021/12/10 05:13:27 App: TestInterBucketRecursion2_function3 got deployed
2021/12/10 05:13:27 Deploying app: TestInterBucketRecursion2_function4. Item count src bucket: 0 dst bucket: 0
2021/12/10 05:13:27 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_INTER_BUCKET_RECURSION",
 "code": 51,
 "description": "Inter bucket recursion error, deployment of current handler will cause inter bucket recursion",
 "attributes": null,
 "runtime_info": {
  "code": 51,
  "info": "Inter bucket recursion error; function: TestInterBucketRecursion2_function4 causes a cycle involving functions: [TestInterBucketRecursion2_function1 TestInterBucketRecursion2_function2 TestInterBucketRecursion2_function3], hence deployment is disallowed"
 }
}
2021/12/10 05:13:27 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 05:13:32 Waiting for app: TestInterBucketRecursion2_function1 to get un-deployed
2021/12/10 05:13:37 Waiting for app: TestInterBucketRecursion2_function1 to get un-deployed
2021/12/10 05:13:42 Waiting for app: TestInterBucketRecursion2_function1 to get un-deployed
2021/12/10 05:13:47 Waiting for app: TestInterBucketRecursion2_function1 to get un-deployed
2021/12/10 05:13:47 App: TestInterBucketRecursion2_function1 got un-deployed
2021/12/10 05:13:47 No eventing-con process running
2021/12/10 05:13:47 Function: TestInterBucketRecursion2_function1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestInterBucketRecursion2_function1 deleting in the background"
}
2021/12/10 05:13:48 Context: Bucket flush, response code: 200 dump: 
2021/12/10 05:13:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 05:13:55 Context: Bucket flush, response code: 200 dump: 
2021/12/10 05:14:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 05:14:00 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 05:14:05 Waiting for app: TestInterBucketRecursion2_function2 to get un-deployed
2021/12/10 05:14:10 Waiting for app: TestInterBucketRecursion2_function2 to get un-deployed
2021/12/10 05:14:15 Waiting for app: TestInterBucketRecursion2_function2 to get un-deployed
2021/12/10 05:14:20 Waiting for app: TestInterBucketRecursion2_function2 to get un-deployed
2021/12/10 05:14:20 App: TestInterBucketRecursion2_function2 got un-deployed
2021/12/10 05:14:20 No eventing-con process running
2021/12/10 05:14:20 Function: TestInterBucketRecursion2_function2 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestInterBucketRecursion2_function2 deleting in the background"
}
2021/12/10 05:14:21 Context: Bucket flush, response code: 200 dump: 
2021/12/10 05:14:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 05:14:28 Context: Bucket flush, response code: 200 dump: 
2021/12/10 05:14:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 05:14:33 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 05:14:38 Waiting for app: TestInterBucketRecursion2_function3 to get un-deployed
2021/12/10 05:14:43 Waiting for app: TestInterBucketRecursion2_function3 to get un-deployed
2021/12/10 05:14:48 Waiting for app: TestInterBucketRecursion2_function3 to get un-deployed
2021/12/10 05:14:53 Waiting for app: TestInterBucketRecursion2_function3 to get un-deployed
2021/12/10 05:14:53 App: TestInterBucketRecursion2_function3 got un-deployed
2021/12/10 05:14:53 No eventing-con process running
2021/12/10 05:14:54 Function: TestInterBucketRecursion2_function3 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestInterBucketRecursion2_function3 deleting in the background"
}
2021/12/10 05:14:55 Context: Bucket flush, response code: 200 dump: 
2021/12/10 05:15:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 05:15:02 Context: Bucket flush, response code: 200 dump: 
2021/12/10 05:15:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 05:15:12 Waiting for app: TestInterBucketRecursion2_function4 to get un-deployed
2021/12/10 05:15:12 App: TestInterBucketRecursion2_function4 got un-deployed
2021/12/10 05:15:12 No eventing-con process running
2021/12/10 05:15:12 Function: TestInterBucketRecursion2_function4 delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestInterBucketRecursion2_function4 not found"
 }
}
2021/12/10 05:15:13 Context: Bucket flush, response code: 200 dump: 
2021/12/10 05:15:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 05:15:20 Context: Bucket flush, response code: 200 dump: 
2021/12/10 05:15:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestInterBucketRecursion2 (213.19s)
=== RUN   TestBucketDeleteWithRebOut
2021/12/10 05:15:32 Removing node(s): [127.0.0.1:19001 127.0.0.1:19002 127.0.0.1:19003] from the cluster
2021/12/10 05:15:37 Rebalance progress: 100

VB distribution across eventing nodes and their workers::

2021/12/10 05:15:40 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:49} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/10 05:15:55 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:49} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/10 05:16:10 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:49} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/10 05:16:25 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:49} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/10 05:16:40 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:49} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/10 05:16:55 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:49} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/10 05:17:10 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:49} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/10 05:17:25 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:49} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/10 05:17:40 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:49} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/10 05:17:55 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:49} {Name:#primary Bucket:hello-world Completion:100}]

DCP Stream statuses:

vbucket curr owner:


2021/12/10 05:18:10 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 05:18:10 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/10 05:18:10 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/10 05:18:15 Rebalance progress: 100

VB distribution across eventing nodes and their workers::


DCP Stream statuses:

vbucket curr owner:


2021/12/10 05:18:16 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 05:18:16 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/10 05:18:16 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/10 05:18:21 Rebalance progress: 100

VB distribution across eventing nodes and their workers::


DCP Stream statuses:

vbucket curr owner:


2021/12/10 05:18:21 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 05:18:26 clusterSetup.go:397: Waiting for app: TestBucketDeleteWithRebOut to get un-deployed
2021/12/10 05:18:26 clusterSetup.go:405: App: TestBucketDeleteWithRebOut got un-deployed
2021/12/10 05:18:26 eventingSetup.go:46: No eventing-con process running
2021/12/10 05:18:26 eventingSetup.go:418: Function: TestBucketDeleteWithRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestBucketDeleteWithRebOut not found"
 }
}
2021/12/10 05:18:27 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 05:18:32 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 05:18:34 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 05:18:39 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 05:18:44 eventingSetup.go:125: Deploying app: TestBucketDeleteWithRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/10 05:18:45 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestBucketDeleteWithRebOut' in metakv",
  "warnings": null
 }
}
2021/12/10 05:18:50 clusterSetup.go:346: Waiting for app: TestBucketDeleteWithRebOut to get deployed
2021/12/10 05:18:55 clusterSetup.go:346: Waiting for app: TestBucketDeleteWithRebOut to get deployed
2021/12/10 05:19:00 clusterSetup.go:346: Waiting for app: TestBucketDeleteWithRebOut to get deployed
2021/12/10 05:19:05 clusterSetup.go:346: Waiting for app: TestBucketDeleteWithRebOut to get deployed
2021/12/10 05:19:05 clusterSetup.go:354: App: TestBucketDeleteWithRebOut got deployed

VB distribution across eventing nodes and their workers::


DCP Stream statuses:
	stream status: running
	len: 1024
	vb list dump: [0-1023]

vbucket curr owner:
Producer node: 127.0.0.1:9302	Node UUID: 489186af2a220152a232d8970a77c4b3
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [0-113]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [114-227]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 114
	v: [228-341]
Producer node: 172.31.2.134:9300	Node UUID: 2c6085b1258ecb09bc4044f4129c5a75
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 113
	v: [911-1023]
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [683-796]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [797-910]
Producer node: 127.0.0.1:9303	Node UUID: a5e5803ffc3c6d65069f0e8054b5433c
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [342-455]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [456-569]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 113
	v: [570-682]


2021/12/10 05:19:05 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 05:19:05 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/10 05:19:10 Rebalance progress: 29.16666666666666
2021/12/10 05:19:15 Rebalance progress: 29.16666666666666
2021/12/10 05:19:20 Rebalance progress: 100

VB distribution across eventing nodes and their workers::


DCP Stream statuses:
	stream status: running
	len: 1024
	vb list dump: [0-1023]

vbucket curr owner:
Producer node: 127.0.0.1:9302	Node UUID: 489186af2a220152a232d8970a77c4b3
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 114
	v: [228-341]
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [0-113]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [114-227]
Producer node: 172.31.2.134:9300	Node UUID: 2c6085b1258ecb09bc4044f4129c5a75
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 113
	v: [911-1023]
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [683-796]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [797-910]
Producer node: 127.0.0.1:9303	Node UUID: a5e5803ffc3c6d65069f0e8054b5433c
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [342-455]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [456-569]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 113
	v: [570-682]


2021/12/10 05:19:20 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 05:19:20 handler_test.go:1674: Deleting source bucket: default
2021/12/10 05:19:22 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/10 05:19:22 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/10 05:19:27 Rebalance progress: 29.16666666666666
2021/12/10 05:19:32 Rebalance progress: 29.16666666666666
2021/12/10 05:19:37 Rebalance progress: 100

VB distribution across eventing nodes and their workers::


DCP Stream statuses:
	stream status: running
	len: 902
	vb list dump: [0-4, 6-10, 12-22, 24-27, 29-36, 38-41, 43-46, 48-61, 63, 65, 67-70, 72-80, 82-101, 103-109, 111-125, 127-132, 135-138, 140-146, 149-150, 152-168, 170-176, 178-179, 181-183, 185-192, 194-205, 207-209, 211-213, 215-237, 239-260, 262-276, 279-281, 283-284, 286-288, 290-299, 303-325, 327-328, 331-341, 343-345, 348-366, 369-390, 392-394, 397-405, 407-428, 431-433, 435-440, 442-461, 464-465, 467-477, 479, 482-483, 485, 488-490, 492, 494-496, 498-499, 502-505, 507-573, 575, 577-583, 585-597, 599-609, 611-626, 628-635, 637-639, 641-657, 659-660, 662-664, 666, 668-671, 673-687, 689, 691-709, 711-712, 714-722, 724, 726-745, 747-751, 754, 756-772, 774-775, 777-790, 792-811, 813-814, 816-820, 823-824, 826-827, 829-831, 833-839, 843-862, 864-877, 879-887, 889-890, 892-914, 916-937, 939, 941-942, 944-946, 949, 951-959, 961-987, 989-998, 1000-1001, 1003-1022]
	stream status: stopped
	len: 122
	vb list dump: [5, 11, 23, 28, 37, 42, 47, 62, 64, 66, 71, 81, 102, 110, 126, 133-134, 139, 147-148, 151, 169, 177, 180, 184, 193, 206, 210, 214, 238, 261, 277-278, 282, 285, 289, 300-302, 326, 329-330, 342, 346-347, 367-368, 391, 395-396, 406, 429-430, 434, 441, 462-463, 466, 478, 480-481, 484, 486-487, 491, 493, 497, 500-501, 506, 574, 576, 584, 598, 610, 627, 636, 640, 658, 661, 665, 667, 672, 688, 690, 710, 713, 723, 725, 746, 752-753, 755, 773, 776, 791, 812, 815, 821-822, 825, 828, 832, 840-842, 863, 878, 888, 891, 915, 938, 940, 943, 947-948, 950, 960, 988, 999, 1002, 1023]

vbucket curr owner:
Producer node: 172.31.2.134:9300	Node UUID: 2c6085b1258ecb09bc4044f4129c5a75
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 101
	v: [911-914, 916-937, 939, 941-942, 944-946, 949, 951-959, 961-987, 989-998, 1000-1001, 1003-1022]
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 101
	v: [683-687, 689, 691-709, 711-712, 714-722, 724, 726-745, 747-751, 754, 756-772, 774-775, 777-790, 792-796]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 100
	v: [797-811, 813-814, 816-820, 823-824, 826-827, 829-831, 833-839, 843-862, 864-877, 879-887, 889-890, 892-910]
Producer node: 127.0.0.1:9303	Node UUID: a5e5803ffc3c6d65069f0e8054b5433c
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 101
	v: [343-345, 348-366, 369-390, 392-394, 397-405, 407-428, 431-433, 435-440, 442-455]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 99
	v: [456-461, 464-465, 467-477, 479, 482-483, 485, 488-490, 492, 494-496, 498-499, 502-505, 507-569]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 100
	v: [570-573, 575, 577-583, 585-597, 599-609, 611-626, 628-635, 637-639, 641-657, 659-660, 662-664, 666, 668-671, 673-682]
Producer node: 127.0.0.1:9302	Node UUID: 489186af2a220152a232d8970a77c4b3
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 101
	v: [228-237, 239-260, 262-276, 279-281, 283-284, 286-288, 290-299, 303-325, 327-328, 331-341]
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 100
	v: [0-4, 6-10, 12-22, 24-27, 29-36, 38-41, 43-46, 48-61, 63, 65, 67-70, 72-80, 82-101, 103-109, 111-113]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 99
	v: [114-125, 127-132, 135-138, 140-146, 149-150, 152-168, 170-176, 178-179, 181-183, 185-192, 194-205, 207-209, 211-213, 215-227]


2021/12/10 05:19:37 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 05:19:42 eventingSetup.go:719: Waiting for function: TestBucketDeleteWithRebOut status to change to undeployed
2021/12/10 05:19:42 eventingSetup.go:727: Function: TestBucketDeleteWithRebOut status changed to undeployed
2021/12/10 05:19:48 clusterSetup.go:397: Waiting for app: TestBucketDeleteWithRebOut to get un-deployed
2021/12/10 05:19:48 clusterSetup.go:405: App: TestBucketDeleteWithRebOut got un-deployed
2021/12/10 05:19:48 eventingSetup.go:46: No eventing-con process running
2021/12/10 05:19:48 eventingSetup.go:418: Function: TestBucketDeleteWithRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestBucketDeleteWithRebOut deleting in the background"
}
2021/12/10 05:19:49 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 05:19:54 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 05:19:56 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 05:20:01 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 05:20:01 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv,index,n1ql to the cluster
2021/12/10 05:20:01 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/10 05:20:06 Rebalance progress: 100
--- PASS: TestBucketDeleteWithRebOut (281.58s)
=== RUN   TestN1QLRecursion
2021/12/10 05:20:08 Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/10 05:20:09 Error in adding nodes : , response: ["Prepare join failed. Node is already part of cluster."]
2021/12/10 05:20:09 Removing node(s): [] from the cluster
2021/12/10 05:20:14 Rebalance progress: 100
2021/12/10 05:20:20 Context: Bucket flush, response code: 200 dump: 
2021/12/10 05:20:26 Context: Bucket flush, response code: 200 dump: 
2021/12/10 05:20:32 Context: Bucket flush, response code: 200 dump: 
2021/12/10 05:20:37 Deploying app: TestN1QLRecursion_function1. Item count src bucket: 0 dst bucket: 0
2021/12/10 05:20:38 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLRecursion_function1' in metakv",
  "warnings": null
 }
}
2021/12/10 05:20:38 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLRecursion_function1' in metakv",
  "warnings": null
 }
} err 
2021/12/10 05:20:43 Waiting for app: TestN1QLRecursion_function1 to get deployed
2021/12/10 05:20:48 Waiting for app: TestN1QLRecursion_function1 to get deployed
2021/12/10 05:20:53 Waiting for app: TestN1QLRecursion_function1 to get deployed
2021/12/10 05:20:58 Waiting for app: TestN1QLRecursion_function1 to get deployed
2021/12/10 05:20:58 App: TestN1QLRecursion_function1 got deployed
2021/12/10 05:20:58 TestN1QLRecursion_function1 is deployed
2021/12/10 05:20:58 Deploying app: TestN1QLRecursion_function3. Item count src bucket: 0 dst bucket: 0
2021/12/10 05:20:59 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLRecursion_function3' in metakv",
  "warnings": null
 }
}
2021/12/10 05:20:59 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLRecursion_function3' in metakv",
  "warnings": null
 }
} err 
2021/12/10 05:21:04 Waiting for app: TestN1QLRecursion_function3 to get deployed
2021/12/10 05:21:10 Waiting for app: TestN1QLRecursion_function3 to get deployed
2021/12/10 05:21:15 Waiting for app: TestN1QLRecursion_function3 to get deployed
2021/12/10 05:21:20 Waiting for app: TestN1QLRecursion_function3 to get deployed
2021/12/10 05:21:20 App: TestN1QLRecursion_function3 got deployed
2021/12/10 05:21:20 TestN1QLRecursion_function3 is deployed
2021/12/10 05:21:20 Deploying app: TestN1QLRecursion_function2. Item count src bucket: 0 dst bucket: 0
2021/12/10 05:21:20 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_INTER_BUCKET_RECURSION",
 "code": 51,
 "description": "Inter bucket recursion error, deployment of current handler will cause inter bucket recursion",
 "attributes": null,
 "runtime_info": {
  "code": 51,
  "info": "Inter bucket recursion error; function: TestN1QLRecursion_function2 causes a cycle involving functions: [TestN1QLRecursion_function3 TestN1QLRecursion_function1], hence deployment is disallowed"
 }
}
2021/12/10 05:21:20 response body {
 "name": "ERR_INTER_BUCKET_RECURSION",
 "code": 51,
 "description": "Inter bucket recursion error, deployment of current handler will cause inter bucket recursion",
 "attributes": null,
 "runtime_info": {
  "code": 51,
  "info": "Inter bucket recursion error; function: TestN1QLRecursion_function2 causes a cycle involving functions: [TestN1QLRecursion_function3 TestN1QLRecursion_function1], hence deployment is disallowed"
 }
} err 
2021/12/10 05:21:20 Success: TestN1QLRecursion_function2 is not deployed
2021/12/10 05:21:20 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 05:21:25 Waiting for app: TestN1QLRecursion_function1 to get un-deployed
2021/12/10 05:21:30 Waiting for app: TestN1QLRecursion_function1 to get un-deployed
2021/12/10 05:21:35 Waiting for app: TestN1QLRecursion_function1 to get un-deployed
2021/12/10 05:21:40 Waiting for app: TestN1QLRecursion_function1 to get un-deployed
2021/12/10 05:21:40 App: TestN1QLRecursion_function1 got un-deployed
2021/12/10 05:21:40 No eventing-con process running
2021/12/10 05:21:40 Function: TestN1QLRecursion_function1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLRecursion_function1 deleting in the background"
}
2021/12/10 05:21:41 Context: Bucket flush, response code: 200 dump: 
2021/12/10 05:21:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 05:21:48 Context: Bucket flush, response code: 200 dump: 
2021/12/10 05:21:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 05:21:53 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 05:21:58 Waiting for app: TestN1QLRecursion_function3 to get un-deployed
2021/12/10 05:22:03 Waiting for app: TestN1QLRecursion_function3 to get un-deployed
2021/12/10 05:22:08 Waiting for app: TestN1QLRecursion_function3 to get un-deployed
2021/12/10 05:22:13 Waiting for app: TestN1QLRecursion_function3 to get un-deployed
2021/12/10 05:22:13 App: TestN1QLRecursion_function3 got un-deployed
2021/12/10 05:22:13 No eventing-con process running
2021/12/10 05:22:13 Function: TestN1QLRecursion_function3 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLRecursion_function3 deleting in the background"
}
2021/12/10 05:22:15 Context: Bucket flush, response code: 200 dump: 
2021/12/10 05:22:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 05:22:22 Context: Bucket flush, response code: 200 dump: 
2021/12/10 05:22:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 05:22:29 Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/10 05:22:34 Rebalance progress: 100
--- PASS: TestN1QLRecursion (145.17s)
=== RUN   TestN1QLAllowRecursion
2021/12/10 05:22:34 Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/10 05:22:34 Error in adding nodes : , response: ["Prepare join failed. Node is already part of cluster."]
2021/12/10 05:22:34 Removing node(s): [] from the cluster
2021/12/10 05:22:39 Rebalance progress: 100
2021/12/10 05:22:45 Context: Bucket flush, response code: 200 dump: 
2021/12/10 05:22:51 Context: Bucket flush, response code: 200 dump: 
2021/12/10 05:22:56 Context: Bucket flush, response code: 200 dump: 
2021/12/10 05:23:01 Deploying app: TestN1QLAllowRecursion_function1. Item count src bucket: 0 dst bucket: 0
2021/12/10 05:23:03 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function1' in metakv",
  "warnings": null
 }
}
2021/12/10 05:23:03 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function1' in metakv",
  "warnings": null
 }
} err 
2021/12/10 05:23:08 Waiting for app: TestN1QLAllowRecursion_function1 to get deployed
2021/12/10 05:23:13 Waiting for app: TestN1QLAllowRecursion_function1 to get deployed
2021/12/10 05:23:18 Waiting for app: TestN1QLAllowRecursion_function1 to get deployed
2021/12/10 05:23:23 Waiting for app: TestN1QLAllowRecursion_function1 to get deployed
2021/12/10 05:23:23 App: TestN1QLAllowRecursion_function1 got deployed
2021/12/10 05:23:23 TestN1QLAllowRecursion_function1 is deployed
2021/12/10 05:23:23 Deploying app: TestN1QLAllowRecursion_function3. Item count src bucket: 0 dst bucket: 0
2021/12/10 05:23:24 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function3' in metakv",
  "warnings": null
 }
}
2021/12/10 05:23:24 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function3' in metakv",
  "warnings": null
 }
} err 
2021/12/10 05:23:29 Waiting for app: TestN1QLAllowRecursion_function3 to get deployed
2021/12/10 05:23:34 Waiting for app: TestN1QLAllowRecursion_function3 to get deployed
2021/12/10 05:23:39 Waiting for app: TestN1QLAllowRecursion_function3 to get deployed
2021/12/10 05:23:44 Waiting for app: TestN1QLAllowRecursion_function3 to get deployed
2021/12/10 05:23:44 App: TestN1QLAllowRecursion_function3 got deployed
2021/12/10 05:23:44 TestN1QLAllowRecursion_function3 is deployed
2021/12/10 05:23:44 Deploying app: TestN1QLAllowRecursion_function2. Item count src bucket: 0 dst bucket: 0
2021/12/10 05:23:45 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function2' in metakv",
  "warnings": null
 }
}
2021/12/10 05:23:45 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function2' in metakv",
  "warnings": null
 }
} err 
2021/12/10 05:23:50 Waiting for app: TestN1QLAllowRecursion_function2 to get deployed
2021/12/10 05:23:55 Waiting for app: TestN1QLAllowRecursion_function2 to get deployed
2021/12/10 05:24:00 Waiting for app: TestN1QLAllowRecursion_function2 to get deployed
2021/12/10 05:24:05 Waiting for app: TestN1QLAllowRecursion_function2 to get deployed
2021/12/10 05:24:05 App: TestN1QLAllowRecursion_function2 got deployed
2021/12/10 05:24:05 Success: TestN1QLAllowRecursion_function2 is deployed after allow_interbucket_recursion
2021/12/10 05:24:05 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 05:24:10 Waiting for app: TestN1QLAllowRecursion_function1 to get un-deployed
2021/12/10 05:24:15 Waiting for app: TestN1QLAllowRecursion_function1 to get un-deployed
2021/12/10 05:24:20 Waiting for app: TestN1QLAllowRecursion_function1 to get un-deployed
2021/12/10 05:24:25 Waiting for app: TestN1QLAllowRecursion_function1 to get un-deployed
2021/12/10 05:24:25 App: TestN1QLAllowRecursion_function1 got un-deployed
2021/12/10 05:24:25 No eventing-con process running
2021/12/10 05:24:25 Function: TestN1QLAllowRecursion_function1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLAllowRecursion_function1 deleting in the background"
}
2021/12/10 05:24:27 Context: Bucket flush, response code: 200 dump: 
2021/12/10 05:24:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 05:24:34 Context: Bucket flush, response code: 200 dump: 
2021/12/10 05:24:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 05:24:39 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 05:24:44 Waiting for app: TestN1QLAllowRecursion_function2 to get un-deployed
2021/12/10 05:24:49 Waiting for app: TestN1QLAllowRecursion_function2 to get un-deployed
2021/12/10 05:24:54 Waiting for app: TestN1QLAllowRecursion_function2 to get un-deployed
2021/12/10 05:24:59 Waiting for app: TestN1QLAllowRecursion_function2 to get un-deployed
2021/12/10 05:24:59 App: TestN1QLAllowRecursion_function2 got un-deployed
2021/12/10 05:24:59 No eventing-con process running
2021/12/10 05:24:59 Function: TestN1QLAllowRecursion_function2 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLAllowRecursion_function2 deleting in the background"
}
2021/12/10 05:25:01 Context: Bucket flush, response code: 200 dump: 
2021/12/10 05:25:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 05:25:08 Context: Bucket flush, response code: 200 dump: 
2021/12/10 05:25:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 05:25:13 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 05:25:18 Waiting for app: TestN1QLAllowRecursion_function3 to get un-deployed
2021/12/10 05:25:23 Waiting for app: TestN1QLAllowRecursion_function3 to get un-deployed
2021/12/10 05:25:28 Waiting for app: TestN1QLAllowRecursion_function3 to get un-deployed
2021/12/10 05:25:33 Waiting for app: TestN1QLAllowRecursion_function3 to get un-deployed
2021/12/10 05:25:33 App: TestN1QLAllowRecursion_function3 got un-deployed
2021/12/10 05:25:33 No eventing-con process running
2021/12/10 05:25:33 Function: TestN1QLAllowRecursion_function3 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLAllowRecursion_function3 deleting in the background"
}
2021/12/10 05:25:35 Context: Bucket flush, response code: 200 dump: 
2021/12/10 05:25:40 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 05:25:42 Context: Bucket flush, response code: 200 dump: 
2021/12/10 05:25:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 05:25:48 Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/10 05:25:53 Rebalance progress: 100
--- PASS: TestN1QLAllowRecursion (199.75s)
=== RUN   TestOnDeleteExpiryBucketOp
2021/12/10 05:25:53 Sleeping for some time
2021/12/10 05:26:03 Waiting for app: TestOnDeleteExpiryBucketOp to get un-deployed
2021/12/10 05:26:03 App: TestOnDeleteExpiryBucketOp got un-deployed
2021/12/10 05:26:03 No eventing-con process running
2021/12/10 05:26:03 Function: TestOnDeleteExpiryBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnDeleteExpiryBucketOp not found"
 }
}
2021/12/10 05:26:05 Context: Bucket flush, response code: 200 dump: 
2021/12/10 05:26:10 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 05:26:12 Context: Bucket flush, response code: 200 dump: 
2021/12/10 05:26:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 05:26:17 Deploying app: TestOnDeleteExpiryBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/10 05:26:18 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnDeleteExpiryBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/10 05:26:18 Starting bucket ops to source bucket
2021/12/10 05:27:07 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 05:27:12 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 05:27:17 Waiting for dst bucket item count to get to: 5000 curr count: 2500
2021/12/10 05:27:17 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/10 05:27:17 Starting bucket ops to source bucket
2021/12/10 05:27:18 Starting bucket ops to source bucket
2021/12/10 05:28:24 Waiting for dst bucket item count to get to: 7000 curr count: 5000
2021/12/10 05:28:29 Waiting for dst bucket item count to get to: 7000 curr count: 5999
2021/12/10 05:28:34 Waiting for dst bucket item count to get to: 7000 curr count: 6999
2021/12/10 05:28:34 src & dst bucket item count matched up. src bucket count: 7000 dst bucket count: 7000
2021/12/10 05:28:34 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 05:28:34 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 05:28:39 Waiting for app: TestOnDeleteExpiryBucketOp to get un-deployed
2021/12/10 05:28:44 Waiting for app: TestOnDeleteExpiryBucketOp to get un-deployed
2021/12/10 05:28:49 Waiting for app: TestOnDeleteExpiryBucketOp to get un-deployed
2021/12/10 05:28:54 Waiting for app: TestOnDeleteExpiryBucketOp to get un-deployed
2021/12/10 05:28:54 App: TestOnDeleteExpiryBucketOp got un-deployed
2021/12/10 05:28:54 No eventing-con process running
2021/12/10 05:28:55 Function: TestOnDeleteExpiryBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnDeleteExpiryBucketOp deleting in the background"
}
2021/12/10 05:28:56 Context: Bucket flush, response code: 200 dump: 
2021/12/10 05:29:06 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/10 05:29:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 05:29:08 Context: Bucket flush, response code: 200 dump: 
2021/12/10 05:29:18 Waiting for dst bucket item count to get to: 0 curr count: 4951
2021/12/10 05:29:23 Waiting for dst bucket item count to get to: 0 curr count: 1451
2021/12/10 05:29:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnDeleteExpiryBucketOp (209.68s)
=== RUN   TestTimerOverWriteSameReference
2021/12/10 05:29:28 Waiting for app: TestTimerOverWriteSameReference to get un-deployed
2021/12/10 05:29:28 App: TestTimerOverWriteSameReference got un-deployed
2021/12/10 05:29:28 No eventing-con process running
2021/12/10 05:29:28 Function: TestTimerOverWriteSameReference delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestTimerOverWriteSameReference not found"
 }
}
2021/12/10 05:29:30 Context: Bucket flush, response code: 200 dump: 
2021/12/10 05:29:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 05:29:37 Context: Bucket flush, response code: 200 dump: 
2021/12/10 05:29:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 05:29:42 Deploying app: TestTimerOverWriteSameReference. Item count src bucket: 0 dst bucket: 0
2021/12/10 05:29:43 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestTimerOverWriteSameReference' in metakv",
  "warnings": null
 }
}
2021/12/10 05:29:48 Waiting for app: TestTimerOverWriteSameReference to get deployed
2021/12/10 05:29:53 Waiting for app: TestTimerOverWriteSameReference to get deployed
2021/12/10 05:29:58 Waiting for app: TestTimerOverWriteSameReference to get deployed
2021/12/10 05:30:03 Waiting for app: TestTimerOverWriteSameReference to get deployed
2021/12/10 05:30:03 App: TestTimerOverWriteSameReference got deployed
2021/12/10 05:30:13 Starting bucket ops to source bucket
2021/12/10 05:31:14 src & dst bucket item count matched up. src bucket count: 1 dst bucket count: 1
2021/12/10 05:31:24 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 05:31:24 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 05:31:29 Waiting for app: TestTimerOverWriteSameReference to get un-deployed
2021/12/10 05:31:34 Waiting for app: TestTimerOverWriteSameReference to get un-deployed
2021/12/10 05:31:39 Waiting for app: TestTimerOverWriteSameReference to get un-deployed
2021/12/10 05:31:44 Waiting for app: TestTimerOverWriteSameReference to get un-deployed
2021/12/10 05:31:44 App: TestTimerOverWriteSameReference got un-deployed
2021/12/10 05:31:44 No eventing-con process running
2021/12/10 05:31:44 Function: TestTimerOverWriteSameReference delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestTimerOverWriteSameReference deleting in the background"
}
2021/12/10 05:31:46 Context: Bucket flush, response code: 200 dump: 
2021/12/10 05:31:56 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/10 05:31:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 05:31:58 Context: Bucket flush, response code: 200 dump: 
2021/12/10 05:32:08 Waiting for dst bucket item count to get to: 0 curr count: 1
2021/12/10 05:32:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestTimerOverWriteSameReference (164.76s)
=== RUN   TestCancelTimerBucketop
2021/12/10 05:32:13 Waiting for app: TestCancelTimerBucketop to get un-deployed
2021/12/10 05:32:13 App: TestCancelTimerBucketop got un-deployed
2021/12/10 05:32:13 No eventing-con process running
2021/12/10 05:32:13 Function: TestCancelTimerBucketop delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestCancelTimerBucketop not found"
 }
}
2021/12/10 05:32:15 Context: Bucket flush, response code: 200 dump: 
2021/12/10 05:32:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 05:32:21 Context: Bucket flush, response code: 200 dump: 
2021/12/10 05:32:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 05:32:26 Deploying app: TestCancelTimerBucketop. Item count src bucket: 0 dst bucket: 0
2021/12/10 05:32:28 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCancelTimerBucketop' in metakv",
  "warnings": null
 }
}
2021/12/10 05:32:33 Waiting for app: TestCancelTimerBucketop to get deployed
2021/12/10 05:32:38 Waiting for app: TestCancelTimerBucketop to get deployed
2021/12/10 05:32:43 Waiting for app: TestCancelTimerBucketop to get deployed
2021/12/10 05:32:48 Waiting for app: TestCancelTimerBucketop to get deployed
2021/12/10 05:32:48 App: TestCancelTimerBucketop got deployed
2021/12/10 05:32:48 Starting bucket ops to source bucket
2021/12/10 05:32:55 Starting bucket ops to source bucket
2021/12/10 05:33:57 src & dst bucket item count matched up. src bucket count: 1000 dst bucket count: 1000
2021/12/10 05:33:57 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 05:33:57 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 05:34:02 Waiting for app: TestCancelTimerBucketop to get un-deployed
2021/12/10 05:34:07 Waiting for app: TestCancelTimerBucketop to get un-deployed
2021/12/10 05:34:12 Waiting for app: TestCancelTimerBucketop to get un-deployed
2021/12/10 05:34:17 Waiting for app: TestCancelTimerBucketop to get un-deployed
2021/12/10 05:34:17 App: TestCancelTimerBucketop got un-deployed
2021/12/10 05:34:17 No eventing-con process running
2021/12/10 05:34:17 Function: TestCancelTimerBucketop delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCancelTimerBucketop deleting in the background"
}
2021/12/10 05:34:19 Context: Bucket flush, response code: 200 dump: 
2021/12/10 05:34:29 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/10 05:34:34 Waiting for dst bucket item count to get to: 0 curr count: 500
2021/12/10 05:34:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 05:34:36 Context: Bucket flush, response code: 200 dump: 
2021/12/10 05:34:46 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/10 05:34:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCancelTimerBucketop (158.07s)
=== RUN   TestJSExpiryDate
2021/12/10 05:34:51 Starting bucket ops to source bucket
2021/12/10 05:35:04 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/10 05:35:04 Deploying app: TestJSExpiryDate. Item count src bucket: 5000 dst bucket: 0
2021/12/10 05:35:05 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestJSExpiryDate' in metakv",
  "warnings": null
 }
}
2021/12/10 05:35:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 05:35:35 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 05:35:35 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 05:35:40 Waiting for app: TestJSExpiryDate to get un-deployed
2021/12/10 05:35:45 Waiting for app: TestJSExpiryDate to get un-deployed
2021/12/10 05:35:50 Waiting for app: TestJSExpiryDate to get un-deployed
2021/12/10 05:35:55 Waiting for app: TestJSExpiryDate to get un-deployed
2021/12/10 05:35:55 App: TestJSExpiryDate got un-deployed
2021/12/10 05:35:55 No eventing-con process running
2021/12/10 05:35:55 Function: TestJSExpiryDate delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestJSExpiryDate deleting in the background"
}
2021/12/10 05:35:57 Context: Bucket flush, response code: 200 dump: 
2021/12/10 05:36:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 05:36:05 Context: Bucket flush, response code: 200 dump: 
2021/12/10 05:36:10 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestJSExpiryDate (83.73s)
=== RUN   TestBinaryDoc
2021/12/10 05:36:15 Starting bucket ops to source bucket
2021/12/10 05:36:15 Deploying app: TestBinaryDoc. Item count src bucket: 0 dst bucket: 0
2021/12/10 05:36:16 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestBinaryDoc' in metakv",
  "warnings": null
 }
}
2021/12/10 05:36:16 Starting bucket ops to source bucket
2021/12/10 05:36:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 05:36:16 Deleting source bucket docs now
2021/12/10 05:36:16 Starting bucket ops to source bucket
2021/12/10 05:36:53 Waiting for dst bucket item count to get to: 32 curr count: 21
2021/12/10 05:36:58 Waiting for dst bucket item count to get to: 32 curr count: 31
2021/12/10 05:37:03 Waiting for dst bucket item count to get to: 32 curr count: 31
2021/12/10 05:37:03 src & dst bucket item count matched up. src bucket count: 32 dst bucket count: 32
2021/12/10 05:37:03 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 05:37:03 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 05:37:08 Waiting for app: TestBinaryDoc to get un-deployed
2021/12/10 05:37:13 Waiting for app: TestBinaryDoc to get un-deployed
2021/12/10 05:37:18 Waiting for app: TestBinaryDoc to get un-deployed
2021/12/10 05:37:23 Waiting for app: TestBinaryDoc to get un-deployed
2021/12/10 05:37:23 App: TestBinaryDoc got un-deployed
2021/12/10 05:37:23 No eventing-con process running
2021/12/10 05:37:23 Function: TestBinaryDoc delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestBinaryDoc deleting in the background"
}
2021/12/10 05:37:25 Context: Bucket flush, response code: 200 dump: 
2021/12/10 05:37:35 Waiting for dst bucket item count to get to: 0 curr count: 32
2021/12/10 05:37:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 05:37:37 Context: Bucket flush, response code: 200 dump: 
2021/12/10 05:37:47 Waiting for dst bucket item count to get to: 0 curr count: 32
2021/12/10 05:37:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestBinaryDoc (97.35s)
=== RUN   TestConstantBindings
2021/12/10 05:37:47 Deploying app: TestConstantBindings. Item count src bucket: 0 dst bucket: 0
2021/12/10 05:37:48 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestConstantBindings' in metakv",
  "warnings": null
 }
}
2021/12/10 05:37:48 Starting bucket ops to source bucket
2021/12/10 05:37:54 Waiting for dst bucket item count to get to: 6000 curr count: 0
2021/12/10 05:37:59 Waiting for dst bucket item count to get to: 6000 curr count: 0
2021/12/10 05:38:04 Waiting for dst bucket item count to get to: 6000 curr count: 0
2021/12/10 05:38:09 Waiting for dst bucket item count to get to: 6000 curr count: 117
2021/12/10 05:38:14 Waiting for dst bucket item count to get to: 6000 curr count: 3117
2021/12/10 05:38:14 src & dst bucket item count matched up. src bucket count: 6000 dst bucket count: 6000
2021/12/10 05:38:14 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 05:38:14 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 05:38:19 Waiting for app: TestConstantBindings to get un-deployed
2021/12/10 05:38:24 Waiting for app: TestConstantBindings to get un-deployed
2021/12/10 05:38:29 Waiting for app: TestConstantBindings to get un-deployed
2021/12/10 05:38:34 Waiting for app: TestConstantBindings to get un-deployed
2021/12/10 05:38:34 App: TestConstantBindings got un-deployed
2021/12/10 05:38:34 No eventing-con process running
2021/12/10 05:38:34 Function: TestConstantBindings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestConstantBindings deleting in the background"
}
2021/12/10 05:38:36 Context: Bucket flush, response code: 200 dump: 
2021/12/10 05:38:46 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/10 05:38:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 05:38:47 Context: Bucket flush, response code: 200 dump: 
2021/12/10 05:38:57 Waiting for dst bucket item count to get to: 0 curr count: 3000
2021/12/10 05:38:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestConstantBindings (70.50s)
=== RUN   TestCodeInsight
2021/12/10 05:39:02 Waiting for app: TestCodeInsight to get un-deployed
2021/12/10 05:39:02 App: TestCodeInsight got un-deployed
2021/12/10 05:39:03 No eventing-con process running
2021/12/10 05:39:03 Function: TestCodeInsight delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestCodeInsight not found"
 }
}
2021/12/10 05:39:04 Context: Bucket flush, response code: 200 dump: 
2021/12/10 05:39:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 05:39:11 Context: Bucket flush, response code: 200 dump: 
2021/12/10 05:39:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 05:39:16 Starting bucket ops to source bucket
2021/12/10 05:39:17 Deploying app: TestCodeInsight. Item count src bucket: 0 dst bucket: 0
2021/12/10 05:39:18 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCodeInsight' in metakv",
  "warnings": null
 }
}
2021/12/10 05:39:23 Waiting for app: TestCodeInsight to get deployed
2021/12/10 05:39:28 Waiting for app: TestCodeInsight to get deployed
2021/12/10 05:39:33 Waiting for app: TestCodeInsight to get deployed
2021/12/10 05:39:38 Waiting for app: TestCodeInsight to get deployed
2021/12/10 05:39:38 App: TestCodeInsight got deployed
2021/12/10 05:39:43 Waiting for dst bucket item count to get to: 1000 curr count: 2
2021/12/10 05:39:43 src & dst bucket item count matched up. src bucket count: 1000 dst bucket count: 1000
2021/12/10 05:39:43 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 05:39:48 Waiting for app: TestCodeInsight to get un-deployed
2021/12/10 05:39:53 Waiting for app: TestCodeInsight to get un-deployed
2021/12/10 05:39:58 Waiting for app: TestCodeInsight to get un-deployed
2021/12/10 05:40:03 Waiting for app: TestCodeInsight to get un-deployed
2021/12/10 05:40:03 App: TestCodeInsight got un-deployed
2021/12/10 05:40:03 No eventing-con process running
2021/12/10 05:40:03 Function: TestCodeInsight delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCodeInsight deleting in the background"
}
2021/12/10 05:40:05 Context: Bucket flush, response code: 200 dump: 
2021/12/10 05:40:15 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/10 05:40:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 05:40:17 Context: Bucket flush, response code: 200 dump: 
2021/12/10 05:40:27 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/10 05:40:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCodeInsight (89.38s)
=== RUN   TestParserTransform
--- PASS: TestParserTransform (0.05s)
=== RUN   TestParserDetect
--- PASS: TestParserDetect (0.00s)
PASS
ok  	github.com/couchbase/eventing/tests/functional_tests	12627.762s

Functional tests took 3 hours, 30 minutes and 31 seconds.

N1ql tests

2021/12/10 05:40:38 Started n1ql
Post "http://127.0.0.1:9000/pools/default": read tcp 127.0.0.1:49646->127.0.0.1:9000: read: connection reset by peer
Quota setup Post "http://127.0.0.1:9000/pools/default": read tcp 127.0.0.1:49646->127.0.0.1:9000: read: connection reset by peer
2021/12/10 05:40:48 Adding node: https://127.0.0.1:19001 with role: kv,index,n1ql to the cluster
2021/12/10 05:40:48 Error in adding nodes : , response: ["Prepare join failed. Node is already part of cluster."]
2021/12/10 05:40:48 Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/10 05:40:48 Removing node(s): [] from the cluster
2021/12/10 05:40:53 Rebalance progress: 100
=== RUN   TestRecursiveMutationN1QL
2021/12/10 05:40:58 Waiting for app: TestRecursiveMutationN1QL to get un-deployed
2021/12/10 05:40:58 App: TestRecursiveMutationN1QL got un-deployed
2021/12/10 05:40:58 No eventing-con process running
2021/12/10 05:40:58 Function: TestRecursiveMutationN1QL delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestRecursiveMutationN1QL not found"
 }
}
2021/12/10 05:41:00 Context: Bucket flush, response code: 200 dump: 
2021/12/10 05:41:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 05:41:07 Context: Bucket flush, response code: 200 dump: 
2021/12/10 05:41:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 05:41:12 Deploying app: TestRecursiveMutationN1QL. Item count src bucket: 0 dst bucket: 0
2021/12/10 05:41:12 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_INTER_BUCKET_RECURSION",
 "code": 51,
 "description": "Inter bucket recursion error, deployment of current handler will cause inter bucket recursion",
 "attributes": null,
 "runtime_info": {
  "code": 51,
  "info": "Function: TestRecursiveMutationN1QL N1QL DML to source keyspace default:default"
 }
}
--- PASS: TestRecursiveMutationN1QL (18.47s)
=== RUN   TestFlexReset1
2021/12/10 05:41:17 Waiting for app: TestFlexReset1 to get un-deployed
2021/12/10 05:41:17 App: TestFlexReset1 got un-deployed
2021/12/10 05:41:17 No eventing-con process running
2021/12/10 05:41:17 Function: TestFlexReset1 delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestFlexReset1 not found"
 }
}
2021/12/10 05:41:19 Context: Bucket flush, response code: 200 dump: 
2021/12/10 05:41:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 05:41:25 Context: Bucket flush, response code: 200 dump: 
2021/12/10 05:41:30 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 05:41:30 Starting bucket ops to source bucket
2021/12/10 05:41:31 Deploying app: TestFlexReset1. Item count src bucket: 0 dst bucket: 0
2021/12/10 05:41:32 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestFlexReset1' in metakv",
  "warnings": null
 }
}
2021/12/10 05:41:37 Waiting for app: TestFlexReset1 to get deployed
2021/12/10 05:41:42 Waiting for app: TestFlexReset1 to get deployed
2021/12/10 05:41:47 Waiting for app: TestFlexReset1 to get deployed
2021/12/10 05:41:52 Waiting for app: TestFlexReset1 to get deployed
2021/12/10 05:41:52 App: TestFlexReset1 got deployed
2021/12/10 05:41:52 src & dst bucket item count matched up. src bucket count: 1000 dst bucket count: 1000
2021/12/10 05:41:52 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 05:41:52 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 05:41:57 Waiting for app: TestFlexReset1 to get un-deployed
2021/12/10 05:42:03 Waiting for app: TestFlexReset1 to get un-deployed
2021/12/10 05:42:08 Waiting for app: TestFlexReset1 to get un-deployed
2021/12/10 05:42:13 Waiting for app: TestFlexReset1 to get un-deployed
2021/12/10 05:42:13 App: TestFlexReset1 got un-deployed
2021/12/10 05:42:13 No eventing-con process running
2021/12/10 05:42:13 Function: TestFlexReset1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestFlexReset1 deleting in the background"
}
2021/12/10 05:42:14 Context: Bucket flush, response code: 200 dump: 
2021/12/10 05:42:24 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/10 05:42:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 05:42:26 Context: Bucket flush, response code: 200 dump: 
2021/12/10 05:42:36 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/10 05:42:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestFlexReset1 (84.19s)
=== RUN   TestFlexReset2
2021/12/10 05:42:41 Waiting for app: TestFlexReset2 to get un-deployed
2021/12/10 05:42:41 App: TestFlexReset2 got un-deployed
2021/12/10 05:42:41 No eventing-con process running
2021/12/10 05:42:41 Function: TestFlexReset2 delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestFlexReset2 not found"
 }
}
2021/12/10 05:42:43 Context: Bucket flush, response code: 200 dump: 
2021/12/10 05:42:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 05:42:50 Context: Bucket flush, response code: 200 dump: 
2021/12/10 05:42:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 05:42:55 Starting bucket ops to source bucket
2021/12/10 05:42:55 Deploying app: TestFlexReset2. Item count src bucket: 0 dst bucket: 0
2021/12/10 05:42:56 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestFlexReset2' in metakv",
  "warnings": null
 }
}
2021/12/10 05:43:01 Waiting for app: TestFlexReset2 to get deployed
2021/12/10 05:43:06 Waiting for app: TestFlexReset2 to get deployed
2021/12/10 05:43:11 Waiting for app: TestFlexReset2 to get deployed
2021/12/10 05:43:16 Waiting for app: TestFlexReset2 to get deployed
2021/12/10 05:43:16 App: TestFlexReset2 got deployed
2021/12/10 05:43:21 Waiting for dst bucket item count to get to: 1000 curr count: 211
2021/12/10 05:43:21 src & dst bucket item count matched up. src bucket count: 1000 dst bucket count: 1000
2021/12/10 05:43:21 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 05:43:22 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 05:43:27 Waiting for app: TestFlexReset2 to get un-deployed
2021/12/10 05:43:32 Waiting for app: TestFlexReset2 to get un-deployed
2021/12/10 05:43:37 Waiting for app: TestFlexReset2 to get un-deployed
2021/12/10 05:43:42 Waiting for app: TestFlexReset2 to get un-deployed
2021/12/10 05:43:42 App: TestFlexReset2 got un-deployed
2021/12/10 05:43:42 No eventing-con process running
2021/12/10 05:43:42 Function: TestFlexReset2 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestFlexReset2 deleting in the background"
}
2021/12/10 05:43:43 Context: Bucket flush, response code: 200 dump: 
2021/12/10 05:43:53 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/10 05:43:58 Waiting for dst bucket item count to get to: 0 curr count: 500
2021/12/10 05:43:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 05:44:00 Context: Bucket flush, response code: 200 dump: 
2021/12/10 05:44:10 Waiting for dst bucket item count to get to: 0 curr count: 766
2021/12/10 05:44:15 Waiting for dst bucket item count to get to: 0 curr count: 766
2021/12/10 05:44:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestFlexReset2 (99.18s)
=== RUN   TestN1QLLabelledBreak
2021/12/10 05:44:20 Waiting for app: TestN1QLLabelledBreak to get un-deployed
2021/12/10 05:44:20 App: TestN1QLLabelledBreak got un-deployed
2021/12/10 05:44:20 No eventing-con process running
2021/12/10 05:44:20 Function: TestN1QLLabelledBreak delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLLabelledBreak not found"
 }
}
2021/12/10 05:44:22 Context: Bucket flush, response code: 200 dump: 
2021/12/10 05:44:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 05:44:29 Context: Bucket flush, response code: 200 dump: 
2021/12/10 05:44:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 05:44:34 Deploying app: TestN1QLLabelledBreak. Item count src bucket: 0 dst bucket: 0
2021/12/10 05:44:35 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLLabelledBreak' in metakv",
  "warnings": null
 }
}
2021/12/10 05:44:40 Waiting for app: TestN1QLLabelledBreak to get deployed
2021/12/10 05:44:45 Waiting for app: TestN1QLLabelledBreak to get deployed
2021/12/10 05:44:50 Waiting for app: TestN1QLLabelledBreak to get deployed
2021/12/10 05:44:55 Waiting for app: TestN1QLLabelledBreak to get deployed
2021/12/10 05:44:55 App: TestN1QLLabelledBreak got deployed
2021/12/10 05:44:55 Starting bucket ops to source bucket
2021/12/10 05:45:04 Waiting for dst bucket item count to get to: 10000 curr count: 0
2021/12/10 05:45:09 Waiting for dst bucket item count to get to: 10000 curr count: 139
2021/12/10 05:45:14 Waiting for dst bucket item count to get to: 10000 curr count: 431
2021/12/10 05:45:20 Waiting for dst bucket item count to get to: 10000 curr count: 4103
2021/12/10 05:45:25 Waiting for dst bucket item count to get to: 10000 curr count: 4103
2021/12/10 05:45:30 Waiting for dst bucket item count to get to: 10000 curr count: 8070
2021/12/10 05:45:35 Waiting for dst bucket item count to get to: 10000 curr count: 8070
2021/12/10 05:45:35 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/10 05:45:35 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 05:45:35 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 05:45:40 Waiting for app: TestN1QLLabelledBreak to get un-deployed
2021/12/10 05:45:45 Waiting for app: TestN1QLLabelledBreak to get un-deployed
2021/12/10 05:45:50 Waiting for app: TestN1QLLabelledBreak to get un-deployed
2021/12/10 05:45:55 Waiting for app: TestN1QLLabelledBreak to get un-deployed
2021/12/10 05:45:55 App: TestN1QLLabelledBreak got un-deployed
2021/12/10 05:45:55 No eventing-con process running
2021/12/10 05:45:55 Function: TestN1QLLabelledBreak delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLLabelledBreak deleting in the background"
}
2021/12/10 05:45:57 Context: Bucket flush, response code: 200 dump: 
2021/12/10 05:46:07 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/10 05:46:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 05:46:08 Context: Bucket flush, response code: 200 dump: 
2021/12/10 05:46:18 Waiting for dst bucket item count to get to: 0 curr count: 8249
2021/12/10 05:46:23 Waiting for dst bucket item count to get to: 0 curr count: 3249
2021/12/10 05:46:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLLabelledBreak (128.04s)
=== RUN   TestN1QLUnlabelledBreak
2021/12/10 05:46:28 Waiting for app: TestN1QLUnlabelledBreak to get un-deployed
2021/12/10 05:46:28 App: TestN1QLUnlabelledBreak got un-deployed
2021/12/10 05:46:28 No eventing-con process running
2021/12/10 05:46:28 Function: TestN1QLUnlabelledBreak delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLUnlabelledBreak not found"
 }
}
2021/12/10 05:46:30 Context: Bucket flush, response code: 200 dump: 
2021/12/10 05:46:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 05:46:37 Context: Bucket flush, response code: 200 dump: 
2021/12/10 05:46:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 05:46:42 Deploying app: TestN1QLUnlabelledBreak. Item count src bucket: 0 dst bucket: 0
2021/12/10 05:46:43 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLUnlabelledBreak' in metakv",
  "warnings": null
 }
}
2021/12/10 05:46:48 Waiting for app: TestN1QLUnlabelledBreak to get deployed
2021/12/10 05:46:53 Waiting for app: TestN1QLUnlabelledBreak to get deployed
2021/12/10 05:46:58 Waiting for app: TestN1QLUnlabelledBreak to get deployed
2021/12/10 05:47:03 Waiting for app: TestN1QLUnlabelledBreak to get deployed
2021/12/10 05:47:03 App: TestN1QLUnlabelledBreak got deployed
2021/12/10 05:47:03 Starting bucket ops to source bucket
2021/12/10 05:47:12 Waiting for dst bucket item count to get to: 10000 curr count: 0
2021/12/10 05:47:17 Waiting for dst bucket item count to get to: 10000 curr count: 0
2021/12/10 05:47:22 Waiting for dst bucket item count to get to: 10000 curr count: 1531
2021/12/10 05:47:27 Waiting for dst bucket item count to get to: 10000 curr count: 4321
2021/12/10 05:47:32 Waiting for dst bucket item count to get to: 10000 curr count: 7150
2021/12/10 05:47:37 Waiting for dst bucket item count to get to: 10000 curr count: 8870
2021/12/10 05:47:37 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/10 05:47:37 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 05:47:37 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 05:47:42 Waiting for app: TestN1QLUnlabelledBreak to get un-deployed
2021/12/10 05:47:47 Waiting for app: TestN1QLUnlabelledBreak to get un-deployed
2021/12/10 05:47:52 Waiting for app: TestN1QLUnlabelledBreak to get un-deployed
2021/12/10 05:47:58 Waiting for app: TestN1QLUnlabelledBreak to get un-deployed
2021/12/10 05:47:58 App: TestN1QLUnlabelledBreak got un-deployed
2021/12/10 05:47:58 No eventing-con process running
2021/12/10 05:47:58 Function: TestN1QLUnlabelledBreak delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLUnlabelledBreak deleting in the background"
}
2021/12/10 05:47:59 Context: Bucket flush, response code: 200 dump: 
2021/12/10 05:48:09 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/10 05:48:14 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/10 05:48:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 05:48:16 Context: Bucket flush, response code: 200 dump: 
2021/12/10 05:48:26 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/10 05:48:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLUnlabelledBreak (122.84s)
=== RUN   TestN1QLThrowStatement
2021/12/10 05:48:31 Waiting for app: TestN1QLThrowStatement to get un-deployed
2021/12/10 05:48:31 App: TestN1QLThrowStatement got un-deployed
2021/12/10 05:48:31 No eventing-con process running
2021/12/10 05:48:31 Function: TestN1QLThrowStatement delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLThrowStatement not found"
 }
}
2021/12/10 05:48:33 Context: Bucket flush, response code: 200 dump: 
2021/12/10 05:48:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 05:48:40 Context: Bucket flush, response code: 200 dump: 
2021/12/10 05:48:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 05:48:45 Deploying app: TestN1QLThrowStatement. Item count src bucket: 0 dst bucket: 0
2021/12/10 05:48:46 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLThrowStatement' in metakv",
  "warnings": null
 }
}
2021/12/10 05:48:51 Waiting for app: TestN1QLThrowStatement to get deployed
2021/12/10 05:48:56 Waiting for app: TestN1QLThrowStatement to get deployed
2021/12/10 05:49:01 Waiting for app: TestN1QLThrowStatement to get deployed
2021/12/10 05:49:06 Waiting for app: TestN1QLThrowStatement to get deployed
2021/12/10 05:49:06 App: TestN1QLThrowStatement got deployed
2021/12/10 05:49:06 Starting bucket ops to source bucket
2021/12/10 05:49:15 Waiting for dst bucket item count to get to: 10000 curr count: 72
2021/12/10 05:49:20 Waiting for dst bucket item count to get to: 10000 curr count: 306
2021/12/10 05:49:25 Waiting for dst bucket item count to get to: 10000 curr count: 306
2021/12/10 05:49:30 Waiting for dst bucket item count to get to: 10000 curr count: 3762
2021/12/10 05:49:35 Waiting for dst bucket item count to get to: 10000 curr count: 3762
2021/12/10 05:49:40 Waiting for dst bucket item count to get to: 10000 curr count: 7822
2021/12/10 05:49:45 Waiting for dst bucket item count to get to: 10000 curr count: 7822
2021/12/10 05:49:45 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/10 05:49:45 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 05:49:46 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 05:49:51 Waiting for app: TestN1QLThrowStatement to get un-deployed
2021/12/10 05:49:56 Waiting for app: TestN1QLThrowStatement to get un-deployed
2021/12/10 05:50:01 Waiting for app: TestN1QLThrowStatement to get un-deployed
2021/12/10 05:50:06 Waiting for app: TestN1QLThrowStatement to get un-deployed
2021/12/10 05:50:06 App: TestN1QLThrowStatement got un-deployed
2021/12/10 05:50:06 No eventing-con process running
2021/12/10 05:50:06 Function: TestN1QLThrowStatement delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLThrowStatement deleting in the background"
}
2021/12/10 05:50:07 Context: Bucket flush, response code: 200 dump: 
2021/12/10 05:50:17 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/10 05:50:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 05:50:19 Context: Bucket flush, response code: 200 dump: 
2021/12/10 05:50:29 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/10 05:50:34 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/10 05:50:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLThrowStatement (128.09s)
=== RUN   TestN1QLNestedForLoop
2021/12/10 05:50:39 Waiting for app: TestN1QLNestedForLoop to get un-deployed
2021/12/10 05:50:39 App: TestN1QLNestedForLoop got un-deployed
2021/12/10 05:50:39 No eventing-con process running
2021/12/10 05:50:39 Function: TestN1QLNestedForLoop delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLNestedForLoop not found"
 }
}
2021/12/10 05:50:41 Context: Bucket flush, response code: 200 dump: 
2021/12/10 05:50:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 05:50:48 Context: Bucket flush, response code: 200 dump: 
2021/12/10 05:50:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 05:50:53 Deploying app: TestN1QLNestedForLoop. Item count src bucket: 0 dst bucket: 0
2021/12/10 05:50:54 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLNestedForLoop' in metakv",
  "warnings": null
 }
}
2021/12/10 05:50:59 Waiting for app: TestN1QLNestedForLoop to get deployed
2021/12/10 05:51:04 Waiting for app: TestN1QLNestedForLoop to get deployed
2021/12/10 05:51:09 Waiting for app: TestN1QLNestedForLoop to get deployed
2021/12/10 05:51:14 Waiting for app: TestN1QLNestedForLoop to get deployed
2021/12/10 05:51:14 App: TestN1QLNestedForLoop got deployed
2021/12/10 05:51:14 Starting bucket ops to source bucket
2021/12/10 05:51:23 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 05:51:28 Waiting for dst bucket item count to get to: 5000 curr count: 109
2021/12/10 05:51:33 Waiting for dst bucket item count to get to: 5000 curr count: 387
2021/12/10 05:51:38 Waiting for dst bucket item count to get to: 5000 curr count: 1507
2021/12/10 05:51:43 Waiting for dst bucket item count to get to: 5000 curr count: 2719
2021/12/10 05:51:48 Waiting for dst bucket item count to get to: 5000 curr count: 3938
2021/12/10 05:51:53 Waiting for dst bucket item count to get to: 5000 curr count: 4948
2021/12/10 05:51:53 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/10 05:51:53 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 05:51:53 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 05:51:58 Waiting for app: TestN1QLNestedForLoop to get un-deployed
2021/12/10 05:52:03 Waiting for app: TestN1QLNestedForLoop to get un-deployed
2021/12/10 05:52:08 Waiting for app: TestN1QLNestedForLoop to get un-deployed
2021/12/10 05:52:13 Waiting for app: TestN1QLNestedForLoop to get un-deployed
2021/12/10 05:52:13 App: TestN1QLNestedForLoop got un-deployed
2021/12/10 05:52:13 No eventing-con process running
2021/12/10 05:52:13 Function: TestN1QLNestedForLoop delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLNestedForLoop deleting in the background"
}
2021/12/10 05:52:15 Context: Bucket flush, response code: 200 dump: 
2021/12/10 05:52:25 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/10 05:52:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 05:52:27 Context: Bucket flush, response code: 200 dump: 
2021/12/10 05:52:37 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/10 05:52:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLNestedForLoop (122.62s)
=== RUN   TestN1QLAutoClose
2021/12/10 05:52:42 Waiting for app: TestN1QLAutoClose to get un-deployed
2021/12/10 05:52:42 App: TestN1QLAutoClose got un-deployed
2021/12/10 05:52:42 No eventing-con process running
2021/12/10 05:52:42 Function: TestN1QLAutoClose delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLAutoClose not found"
 }
}
2021/12/10 05:52:44 Context: Bucket flush, response code: 200 dump: 
2021/12/10 05:52:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 05:52:50 Context: Bucket flush, response code: 200 dump: 
2021/12/10 05:52:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 05:52:55 Deploying app: TestN1QLAutoClose. Item count src bucket: 0 dst bucket: 0
2021/12/10 05:52:57 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAutoClose' in metakv",
  "warnings": null
 }
}
2021/12/10 05:53:02 Waiting for app: TestN1QLAutoClose to get deployed
2021/12/10 05:53:07 Waiting for app: TestN1QLAutoClose to get deployed
2021/12/10 05:53:12 Waiting for app: TestN1QLAutoClose to get deployed
2021/12/10 05:53:17 Waiting for app: TestN1QLAutoClose to get deployed
2021/12/10 05:53:17 App: TestN1QLAutoClose got deployed
2021/12/10 05:53:17 Starting bucket ops to source bucket
2021/12/10 05:53:22 Waiting for dst bucket item count to get to: 500 curr count: 0
2021/12/10 05:53:27 Waiting for dst bucket item count to get to: 500 curr count: 1
2021/12/10 05:53:32 Waiting for dst bucket item count to get to: 500 curr count: 1
2021/12/10 05:53:37 Waiting for dst bucket item count to get to: 500 curr count: 24
2021/12/10 05:53:42 Waiting for dst bucket item count to get to: 500 curr count: 24
2021/12/10 05:53:47 Waiting for dst bucket item count to get to: 500 curr count: 46
2021/12/10 05:53:52 Waiting for dst bucket item count to get to: 500 curr count: 46
2021/12/10 05:53:57 Waiting for dst bucket item count to get to: 500 curr count: 69
2021/12/10 05:54:02 Waiting for dst bucket item count to get to: 500 curr count: 69
2021/12/10 05:54:07 Waiting for dst bucket item count to get to: 500 curr count: 90
2021/12/10 05:54:12 Waiting for dst bucket item count to get to: 500 curr count: 90
2021/12/10 05:54:17 Waiting for dst bucket item count to get to: 500 curr count: 111
2021/12/10 05:54:22 Waiting for dst bucket item count to get to: 500 curr count: 111
2021/12/10 05:54:27 Waiting for dst bucket item count to get to: 500 curr count: 111
2021/12/10 05:54:32 Waiting for dst bucket item count to get to: 500 curr count: 133
2021/12/10 05:54:37 Waiting for dst bucket item count to get to: 500 curr count: 133
2021/12/10 05:54:42 Waiting for dst bucket item count to get to: 500 curr count: 153
2021/12/10 05:54:47 Waiting for dst bucket item count to get to: 500 curr count: 164
2021/12/10 05:54:52 Waiting for dst bucket item count to get to: 500 curr count: 176
2021/12/10 05:54:57 Waiting for dst bucket item count to get to: 500 curr count: 188
2021/12/10 05:55:02 Waiting for dst bucket item count to get to: 500 curr count: 199
2021/12/10 05:55:07 Waiting for dst bucket item count to get to: 500 curr count: 209
2021/12/10 05:55:12 Waiting for dst bucket item count to get to: 500 curr count: 220
2021/12/10 05:55:17 Waiting for dst bucket item count to get to: 500 curr count: 229
2021/12/10 05:55:22 Waiting for dst bucket item count to get to: 500 curr count: 239
2021/12/10 05:55:27 Waiting for dst bucket item count to get to: 500 curr count: 248
2021/12/10 05:55:32 Waiting for dst bucket item count to get to: 500 curr count: 260
2021/12/10 05:55:37 Waiting for dst bucket item count to get to: 500 curr count: 270
2021/12/10 05:55:42 Waiting for dst bucket item count to get to: 500 curr count: 281
2021/12/10 05:55:47 Waiting for dst bucket item count to get to: 500 curr count: 293
2021/12/10 05:55:52 Waiting for dst bucket item count to get to: 500 curr count: 301
2021/12/10 05:55:57 Waiting for dst bucket item count to get to: 500 curr count: 313
2021/12/10 05:56:02 Waiting for dst bucket item count to get to: 500 curr count: 324
2021/12/10 05:56:08 Waiting for dst bucket item count to get to: 500 curr count: 336
2021/12/10 05:56:13 Waiting for dst bucket item count to get to: 500 curr count: 344
2021/12/10 05:56:18 Waiting for dst bucket item count to get to: 500 curr count: 363
2021/12/10 05:56:23 Waiting for dst bucket item count to get to: 500 curr count: 363
2021/12/10 05:56:28 Waiting for dst bucket item count to get to: 500 curr count: 385
2021/12/10 05:56:33 Waiting for dst bucket item count to get to: 500 curr count: 385
2021/12/10 05:56:38 Waiting for dst bucket item count to get to: 500 curr count: 407
2021/12/10 05:56:43 Waiting for dst bucket item count to get to: 500 curr count: 407
2021/12/10 05:56:48 Waiting for dst bucket item count to get to: 500 curr count: 428
2021/12/10 05:56:53 Waiting for dst bucket item count to get to: 500 curr count: 428
2021/12/10 05:56:58 Waiting for dst bucket item count to get to: 500 curr count: 450
2021/12/10 05:57:03 Waiting for dst bucket item count to get to: 500 curr count: 450
2021/12/10 05:57:08 Waiting for dst bucket item count to get to: 500 curr count: 471
2021/12/10 05:57:13 Waiting for dst bucket item count to get to: 500 curr count: 471
2021/12/10 05:57:18 Waiting for dst bucket item count to get to: 500 curr count: 492
2021/12/10 05:57:23 Waiting for dst bucket item count to get to: 500 curr count: 492
2021/12/10 05:57:23 src & dst bucket item count matched up. src bucket count: 500 dst bucket count: 500
2021/12/10 05:57:23 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 05:57:23 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 05:57:28 Waiting for app: TestN1QLAutoClose to get un-deployed
2021/12/10 05:57:33 Waiting for app: TestN1QLAutoClose to get un-deployed
2021/12/10 05:57:38 Waiting for app: TestN1QLAutoClose to get un-deployed
2021/12/10 05:57:43 Waiting for app: TestN1QLAutoClose to get un-deployed
2021/12/10 05:57:43 App: TestN1QLAutoClose got un-deployed
2021/12/10 05:57:43 No eventing-con process running
2021/12/10 05:57:43 Function: TestN1QLAutoClose delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLAutoClose deleting in the background"
}
2021/12/10 05:57:45 Context: Bucket flush, response code: 200 dump: 
2021/12/10 05:57:55 Waiting for dst bucket item count to get to: 0 curr count: 500
2021/12/10 05:57:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 05:57:56 Context: Bucket flush, response code: 200 dump: 
2021/12/10 05:58:06 Waiting for dst bucket item count to get to: 0 curr count: 500
2021/12/10 05:58:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLAutoClose (329.40s)
=== RUN   TestN1QLPosParams
2021/12/10 05:58:11 Waiting for app: TestN1QLPosParams to get un-deployed
2021/12/10 05:58:11 App: TestN1QLPosParams got un-deployed
2021/12/10 05:58:11 No eventing-con process running
2021/12/10 05:58:11 Function: TestN1QLPosParams delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLPosParams not found"
 }
}
2021/12/10 05:58:13 Context: Bucket flush, response code: 200 dump: 
2021/12/10 05:58:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 05:58:20 Context: Bucket flush, response code: 200 dump: 
2021/12/10 05:58:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 05:58:25 Deploying app: TestN1QLPosParams. Item count src bucket: 0 dst bucket: 0
2021/12/10 05:58:26 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLPosParams' in metakv",
  "warnings": null
 }
}
2021/12/10 05:58:31 Waiting for app: TestN1QLPosParams to get deployed
2021/12/10 05:58:36 Waiting for app: TestN1QLPosParams to get deployed
2021/12/10 05:58:41 Waiting for app: TestN1QLPosParams to get deployed
2021/12/10 05:58:46 Waiting for app: TestN1QLPosParams to get deployed
2021/12/10 05:58:46 App: TestN1QLPosParams got deployed
2021/12/10 05:58:46 Starting bucket ops to source bucket
2021/12/10 05:58:56 Waiting for dst bucket item count to get to: 5000 curr count: 706
2021/12/10 05:59:01 Waiting for dst bucket item count to get to: 5000 curr count: 706
2021/12/10 05:59:06 Waiting for dst bucket item count to get to: 5000 curr count: 706
2021/12/10 05:59:06 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/10 05:59:06 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 05:59:06 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 05:59:11 Waiting for app: TestN1QLPosParams to get un-deployed
2021/12/10 05:59:16 Waiting for app: TestN1QLPosParams to get un-deployed
2021/12/10 05:59:21 Waiting for app: TestN1QLPosParams to get un-deployed
2021/12/10 05:59:26 Waiting for app: TestN1QLPosParams to get un-deployed
2021/12/10 05:59:26 App: TestN1QLPosParams got un-deployed
2021/12/10 05:59:26 No eventing-con process running
2021/12/10 05:59:26 Function: TestN1QLPosParams delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLPosParams deleting in the background"
}
2021/12/10 05:59:28 Context: Bucket flush, response code: 200 dump: 
2021/12/10 05:59:38 Waiting for dst bucket item count to get to: 0 curr count: 2782
2021/12/10 05:59:43 Waiting for dst bucket item count to get to: 0 curr count: 282
2021/12/10 05:59:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 05:59:44 Context: Bucket flush, response code: 200 dump: 
2021/12/10 05:59:54 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/10 05:59:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLPosParams (108.19s)
=== RUN   TestN1QLExhaustConnPool
2021/12/10 05:59:59 Waiting for app: TestN1QLExhaustConnPool to get un-deployed
2021/12/10 05:59:59 App: TestN1QLExhaustConnPool got un-deployed
2021/12/10 06:00:00 No eventing-con process running
2021/12/10 06:00:00 Function: TestN1QLExhaustConnPool delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLExhaustConnPool not found"
 }
}
2021/12/10 06:00:01 Context: Bucket flush, response code: 200 dump: 
2021/12/10 06:00:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 06:00:08 Context: Bucket flush, response code: 200 dump: 
2021/12/10 06:00:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 06:00:13 Deploying app: TestN1QLExhaustConnPool. Item count src bucket: 0 dst bucket: 0
2021/12/10 06:00:14 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLExhaustConnPool' in metakv",
  "warnings": null
 }
}
2021/12/10 06:00:19 Waiting for app: TestN1QLExhaustConnPool to get deployed
2021/12/10 06:00:24 Waiting for app: TestN1QLExhaustConnPool to get deployed
2021/12/10 06:00:29 Waiting for app: TestN1QLExhaustConnPool to get deployed
2021/12/10 06:00:34 Waiting for app: TestN1QLExhaustConnPool to get deployed
2021/12/10 06:00:34 App: TestN1QLExhaustConnPool got deployed
2021/12/10 06:00:34 Starting bucket ops to source bucket
2021/12/10 06:00:39 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/10 06:00:44 Waiting for dst bucket item count to get to: 100 curr count: 49
2021/12/10 06:00:49 Waiting for dst bucket item count to get to: 100 curr count: 99
2021/12/10 06:00:49 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/10 06:00:49 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 06:00:50 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 06:00:55 Waiting for app: TestN1QLExhaustConnPool to get un-deployed
2021/12/10 06:01:00 Waiting for app: TestN1QLExhaustConnPool to get un-deployed
2021/12/10 06:01:05 Waiting for app: TestN1QLExhaustConnPool to get un-deployed
2021/12/10 06:01:10 Waiting for app: TestN1QLExhaustConnPool to get un-deployed
2021/12/10 06:01:10 App: TestN1QLExhaustConnPool got un-deployed
2021/12/10 06:01:10 No eventing-con process running
2021/12/10 06:01:10 Function: TestN1QLExhaustConnPool delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLExhaustConnPool deleting in the background"
}
2021/12/10 06:01:11 Context: Bucket flush, response code: 200 dump: 
2021/12/10 06:01:21 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/10 06:01:26 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/10 06:01:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 06:01:28 Context: Bucket flush, response code: 200 dump: 
2021/12/10 06:01:38 Waiting for dst bucket item count to get to: 0 curr count: 73
2021/12/10 06:01:43 Waiting for dst bucket item count to get to: 0 curr count: 23
2021/12/10 06:01:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLExhaustConnPool (108.64s)
=== RUN   TestN1QLGraceSufficientTimeOut
2021/12/10 06:01:48 Waiting for app: TestN1QLGraceSufficientTimeOut to get un-deployed
2021/12/10 06:01:48 App: TestN1QLGraceSufficientTimeOut got un-deployed
2021/12/10 06:01:48 No eventing-con process running
2021/12/10 06:01:48 Function: TestN1QLGraceSufficientTimeOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLGraceSufficientTimeOut not found"
 }
}
2021/12/10 06:01:50 Context: Bucket flush, response code: 200 dump: 
2021/12/10 06:01:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 06:01:57 Context: Bucket flush, response code: 200 dump: 
2021/12/10 06:02:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 06:02:02 Starting bucket ops to source bucket
2021/12/10 06:02:19 Starting bucket ops to source bucket
2021/12/10 06:02:36 Deploying app: TestN1QLGraceSufficientTimeOut. Item count src bucket: 30000 dst bucket: 16299
2021/12/10 06:02:37 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLGraceSufficientTimeOut' in metakv",
  "warnings": null
 }
}
2021/12/10 06:02:42 Waiting for app: TestN1QLGraceSufficientTimeOut to get deployed
2021/12/10 06:02:47 Waiting for app: TestN1QLGraceSufficientTimeOut to get deployed
2021/12/10 06:02:52 Waiting for app: TestN1QLGraceSufficientTimeOut to get deployed
2021/12/10 06:02:57 Waiting for app: TestN1QLGraceSufficientTimeOut to get deployed
2021/12/10 06:02:57 App: TestN1QLGraceSufficientTimeOut got deployed
2021/12/10 06:02:57 Starting bucket ops to source bucket
2021/12/10 06:03:02 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/10 06:03:07 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/10 06:03:12 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/10 06:03:17 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/10 06:03:22 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/10 06:03:27 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/10 06:03:32 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/10 06:03:37 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/10 06:03:42 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/10 06:03:42 src & dst bucket item count matched up. src bucket count: 30001 dst bucket count: 30001
2021/12/10 06:03:42 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 06:03:43 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 06:03:48 Waiting for app: TestN1QLGraceSufficientTimeOut to get un-deployed
2021/12/10 06:03:53 Waiting for app: TestN1QLGraceSufficientTimeOut to get un-deployed
2021/12/10 06:03:58 Waiting for app: TestN1QLGraceSufficientTimeOut to get un-deployed
2021/12/10 06:04:03 Waiting for app: TestN1QLGraceSufficientTimeOut to get un-deployed
2021/12/10 06:04:03 App: TestN1QLGraceSufficientTimeOut got un-deployed
2021/12/10 06:04:03 No eventing-con process running
2021/12/10 06:04:03 Function: TestN1QLGraceSufficientTimeOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLGraceSufficientTimeOut deleting in the background"
}
2021/12/10 06:04:04 Context: Bucket flush, response code: 200 dump: 
2021/12/10 06:04:14 Waiting for dst bucket item count to get to: 0 curr count: 30000
2021/12/10 06:04:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 06:04:16 Context: Bucket flush, response code: 200 dump: 
2021/12/10 06:04:26 Waiting for dst bucket item count to get to: 0 curr count: 30001
2021/12/10 06:04:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLGraceSufficientTimeOut (163.12s)
=== RUN   TestN1QLQueryConsistency
2021/12/10 06:04:31 Waiting for app: n1qlquery_consistency to get un-deployed
2021/12/10 06:04:31 App: n1qlquery_consistency got un-deployed
2021/12/10 06:04:31 No eventing-con process running
2021/12/10 06:04:31 Function: n1qlquery_consistency delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: n1qlquery_consistency not found"
 }
}
2021/12/10 06:04:33 Context: Bucket flush, response code: 200 dump: 
2021/12/10 06:04:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 06:04:40 Context: Bucket flush, response code: 200 dump: 
2021/12/10 06:04:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 06:04:45 Starting bucket ops to source bucket
2021/12/10 06:04:48 Deploying app: n1qlquery_consistency. Item count src bucket: 0 dst bucket: 0
2021/12/10 06:04:49 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'n1qlquery_consistency' in metakv",
  "warnings": null
 }
}
2021/12/10 06:04:54 Waiting for app: n1qlquery_consistency to get deployed
2021/12/10 06:04:59 Waiting for app: n1qlquery_consistency to get deployed
2021/12/10 06:05:04 Waiting for app: n1qlquery_consistency to get deployed
2021/12/10 06:05:09 Waiting for app: n1qlquery_consistency to get deployed
2021/12/10 06:05:09 App: n1qlquery_consistency got deployed
2021/12/10 06:05:14 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 06:05:19 Waiting for dst bucket item count to get to: 5000 curr count: 4442
2021/12/10 06:05:24 Waiting for dst bucket item count to get to: 5000 curr count: 4442
2021/12/10 06:05:24 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/10 06:05:24 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 06:05:24 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 06:05:29 Waiting for app: n1qlquery_consistency to get un-deployed
2021/12/10 06:05:34 Waiting for app: n1qlquery_consistency to get un-deployed
2021/12/10 06:05:39 Waiting for app: n1qlquery_consistency to get un-deployed
2021/12/10 06:05:44 Waiting for app: n1qlquery_consistency to get un-deployed
2021/12/10 06:05:44 App: n1qlquery_consistency got un-deployed
2021/12/10 06:05:44 No eventing-con process running
2021/12/10 06:05:44 Function: n1qlquery_consistency delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: n1qlquery_consistency deleting in the background"
}
2021/12/10 06:05:46 Context: Bucket flush, response code: 200 dump: 
2021/12/10 06:05:56 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/10 06:05:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 06:05:58 Context: Bucket flush, response code: 200 dump: 
2021/12/10 06:06:08 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/10 06:06:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLQueryConsistency (101.33s)
PASS
ok  	github.com/couchbase/eventing/tests/functional_tests	1526.087s

N1ql tests took 0 hours, 25 minutes and 30 seconds.

Starting server
Starting server: attempt 1

Rebalance tests

2021/12/10 06:07:43 Started rebalance
Post "http://127.0.0.1:9000/pools/default": read tcp 127.0.0.1:38344->127.0.0.1:9000: read: connection reset by peer
Quota setup Post "http://127.0.0.1:9000/pools/default": read tcp 127.0.0.1:38344->127.0.0.1:9000: read: connection reset by peer
=== RUN   TestDuplicateWithEventingRebIn
2021/12/10 06:08:01 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
2021/12/10 06:08:01 App: TestDuplicateWithEventingRebIn got un-deployed
2021/12/10 06:08:01 No eventing-con process running
2021/12/10 06:08:01 Function: TestDuplicateWithEventingRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithEventingRebIn not found"
 }
}
2021/12/10 06:08:03 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:08:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 06:08:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 06:08:10 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:08:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 06:08:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:08:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 06:08:20 Deploying app: TestDuplicateWithEventingRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/10 06:08:21 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithEventingRebIn' in metakv",
  "warnings": null
 }
}
2021/12/10 06:08:26 Waiting for app: TestDuplicateWithEventingRebIn to get deployed
2021/12/10 06:08:31 Waiting for app: TestDuplicateWithEventingRebIn to get deployed
2021/12/10 06:08:36 Waiting for app: TestDuplicateWithEventingRebIn to get deployed
2021/12/10 06:08:41 Waiting for app: TestDuplicateWithEventingRebIn to get deployed
2021/12/10 06:08:41 App: TestDuplicateWithEventingRebIn got deployed
2021/12/10 06:08:41 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/10 06:08:41 Starting bucket ops to source bucket
2021/12/10 06:08:55 Removing node(s): [] from the cluster
2021/12/10 06:09:00 Rebalance progress: 0
2021/12/10 06:09:05 Rebalance progress: 0
2021/12/10 06:09:10 Rebalance progress: 25
2021/12/10 06:09:15 Rebalance progress: 25
2021/12/10 06:09:20 Rebalance progress: 25
2021/12/10 06:09:25 Rebalance progress: 98.40425531914893
2021/12/10 06:09:30 Rebalance progress: 98.40425531914893
2021/12/10 06:09:35 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:09:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:09:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 06:09:35 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9302/api/v1/stats": dial tcp 127.0.0.1:9302: connect: connection refused
2021/12/10 06:09:35 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/10 06:09:35 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/10 06:09:40 Rebalance progress: 25
2021/12/10 06:09:45 Rebalance progress: 25
2021/12/10 06:09:50 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:09:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:09:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 06:09:50 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9302/api/v1/stats": dial tcp 127.0.0.1:9302: connect: connection refused
2021/12/10 06:09:50 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/10 06:09:50 Waiting for src bucket item count to get to: 200000 curr count: 95725
2021/12/10 06:09:55 Waiting for src bucket item count to get to: 200000 curr count: 112707
2021/12/10 06:10:00 Waiting for src bucket item count to get to: 200000 curr count: 112707
2021/12/10 06:10:05 Waiting for src bucket item count to get to: 200000 curr count: 129091
2021/12/10 06:10:10 Waiting for src bucket item count to get to: 200000 curr count: 129091
2021/12/10 06:10:15 Waiting for src bucket item count to get to: 200000 curr count: 145987
2021/12/10 06:10:20 Waiting for src bucket item count to get to: 200000 curr count: 145987
2021/12/10 06:10:25 Waiting for src bucket item count to get to: 200000 curr count: 162881
2021/12/10 06:10:30 Waiting for src bucket item count to get to: 200000 curr count: 162881
2021/12/10 06:10:35 Waiting for src bucket item count to get to: 200000 curr count: 179090
2021/12/10 06:10:40 Waiting for src bucket item count to get to: 200000 curr count: 179090
2021/12/10 06:10:45 Waiting for src bucket item count to get to: 200000 curr count: 195628
2021/12/10 06:10:50 Waiting for src bucket item count to get to: 200000 curr count: 195628
2021/12/10 06:10:55 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/10 06:10:55 Post SET ops. Expected item count: 200000 got 200000
2021/12/10 06:10:55 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 06:11:00 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
2021/12/10 06:11:05 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
2021/12/10 06:11:10 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
2021/12/10 06:11:15 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
2021/12/10 06:11:15 App: TestDuplicateWithEventingRebIn got un-deployed
2021/12/10 06:11:15 No eventing-con process running
2021/12/10 06:11:15 Function: TestDuplicateWithEventingRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithEventingRebIn deleting in the background"
}
2021/12/10 06:11:17 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:11:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 06:11:27 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/10 06:11:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 06:11:29 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:11:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 06:11:39 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/10 06:11:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithEventingRebIn (227.68s)
=== RUN   TestDuplicateWithEventingRebOut
2021/12/10 06:11:44 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/10 06:11:44 Removing node(s): [] from the cluster
2021/12/10 06:11:49 Rebalance progress: 100
2021/12/10 06:11:54 Waiting for app: TestDuplicateWithEventingRebOut to get un-deployed
2021/12/10 06:11:54 App: TestDuplicateWithEventingRebOut got un-deployed
2021/12/10 06:11:54 No eventing-con process running
2021/12/10 06:11:54 Function: TestDuplicateWithEventingRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithEventingRebOut not found"
 }
}
2021/12/10 06:11:56 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:12:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 06:12:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 06:12:03 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:12:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 06:12:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:12:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 06:12:13 Deploying app: TestDuplicateWithEventingRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/10 06:12:14 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithEventingRebOut' in metakv",
  "warnings": null
 }
}
2021/12/10 06:12:19 Waiting for app: TestDuplicateWithEventingRebOut to get deployed
2021/12/10 06:12:24 Waiting for app: TestDuplicateWithEventingRebOut to get deployed
2021/12/10 06:12:29 Waiting for app: TestDuplicateWithEventingRebOut to get deployed
2021/12/10 06:12:34 Waiting for app: TestDuplicateWithEventingRebOut to get deployed
2021/12/10 06:12:34 App: TestDuplicateWithEventingRebOut got deployed
2021/12/10 06:12:34 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/10 06:12:34 Starting bucket ops to source bucket
2021/12/10 06:12:39 Rebalance progress: 25
2021/12/10 06:12:44 Rebalance progress: 25
2021/12/10 06:12:49 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:12:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:12:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 06:12:49 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9302/api/v1/stats": dial tcp 127.0.0.1:9302: connect: connection refused
2021/12/10 06:12:49 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/10 06:12:49 Waiting for src bucket item count to get to: 200000 curr count: 8840
2021/12/10 06:12:54 Waiting for src bucket item count to get to: 200000 curr count: 25311
2021/12/10 06:12:59 Waiting for src bucket item count to get to: 200000 curr count: 25311
2021/12/10 06:13:04 Waiting for src bucket item count to get to: 200000 curr count: 41993
2021/12/10 06:13:09 Waiting for src bucket item count to get to: 200000 curr count: 41993
2021/12/10 06:13:14 Waiting for src bucket item count to get to: 200000 curr count: 58325
2021/12/10 06:13:19 Waiting for src bucket item count to get to: 200000 curr count: 58325
2021/12/10 06:13:24 Waiting for src bucket item count to get to: 200000 curr count: 58325
2021/12/10 06:13:29 Waiting for src bucket item count to get to: 200000 curr count: 74636
2021/12/10 06:13:34 Waiting for src bucket item count to get to: 200000 curr count: 74636
2021/12/10 06:13:39 Waiting for src bucket item count to get to: 200000 curr count: 90878
2021/12/10 06:13:44 Waiting for src bucket item count to get to: 200000 curr count: 90878
2021/12/10 06:13:49 Waiting for src bucket item count to get to: 200000 curr count: 107661
2021/12/10 06:13:54 Waiting for src bucket item count to get to: 200000 curr count: 107661
2021/12/10 06:13:59 Waiting for src bucket item count to get to: 200000 curr count: 124073
2021/12/10 06:14:04 Waiting for src bucket item count to get to: 200000 curr count: 124073
2021/12/10 06:14:09 Waiting for src bucket item count to get to: 200000 curr count: 140885
2021/12/10 06:14:14 Waiting for src bucket item count to get to: 200000 curr count: 157469
2021/12/10 06:14:19 Waiting for src bucket item count to get to: 200000 curr count: 157469
2021/12/10 06:14:24 Waiting for src bucket item count to get to: 200000 curr count: 174009
2021/12/10 06:14:29 Waiting for src bucket item count to get to: 200000 curr count: 174009
2021/12/10 06:14:34 Waiting for src bucket item count to get to: 200000 curr count: 190198
2021/12/10 06:14:39 Waiting for src bucket item count to get to: 200000 curr count: 190198
2021/12/10 06:14:44 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/10 06:14:44 Post SET ops. Expected item count: 200000 got 200000
2021/12/10 06:14:44 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 06:14:49 Waiting for app: TestDuplicateWithEventingRebOut to get un-deployed
2021/12/10 06:14:54 Waiting for app: TestDuplicateWithEventingRebOut to get un-deployed
2021/12/10 06:14:59 Waiting for app: TestDuplicateWithEventingRebOut to get un-deployed
2021/12/10 06:15:04 Waiting for app: TestDuplicateWithEventingRebOut to get un-deployed
2021/12/10 06:15:04 App: TestDuplicateWithEventingRebOut got un-deployed
2021/12/10 06:15:04 No eventing-con process running
2021/12/10 06:15:04 Function: TestDuplicateWithEventingRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithEventingRebOut deleting in the background"
}
2021/12/10 06:15:06 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:15:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 06:15:16 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/10 06:15:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 06:15:18 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:15:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 06:15:28 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/10 06:15:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithEventingRebOut (228.66s)
=== RUN   TestDuplicateWithEventingSwapReb
2021/12/10 06:15:38 Waiting for app: TestDuplicateWithEventingSwapReb to get un-deployed
2021/12/10 06:15:38 App: TestDuplicateWithEventingSwapReb got un-deployed
2021/12/10 06:15:38 No eventing-con process running
2021/12/10 06:15:38 Function: TestDuplicateWithEventingSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithEventingSwapReb not found"
 }
}
2021/12/10 06:15:39 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:15:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 06:15:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 06:15:46 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:15:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 06:15:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:15:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 06:15:56 Deploying app: TestDuplicateWithEventingSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/10 06:15:57 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithEventingSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/10 06:16:02 Waiting for app: TestDuplicateWithEventingSwapReb to get deployed
2021/12/10 06:16:07 Waiting for app: TestDuplicateWithEventingSwapReb to get deployed
2021/12/10 06:16:12 Waiting for app: TestDuplicateWithEventingSwapReb to get deployed
2021/12/10 06:16:17 Waiting for app: TestDuplicateWithEventingSwapReb to get deployed
2021/12/10 06:16:17 App: TestDuplicateWithEventingSwapReb got deployed
2021/12/10 06:16:17 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/10 06:16:17 Starting bucket ops to source bucket
2021/12/10 06:16:18 Removing node(s): [] from the cluster
2021/12/10 06:16:23 Rebalance progress: 25
2021/12/10 06:16:28 Rebalance progress: 25
2021/12/10 06:16:33 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:16:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:16:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 06:16:33 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9302/api/v1/stats": dial tcp 127.0.0.1:9302: connect: connection refused
2021/12/10 06:16:33 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/10 06:16:33 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/10 06:16:39 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/10 06:16:44 Rebalance progress: 16.66666666666667
2021/12/10 06:16:49 Rebalance progress: 16.66666666666667
2021/12/10 06:16:54 Rebalance progress: 16.66666666666667
2021/12/10 06:16:59 Rebalance progress: 30.4029304029304
2021/12/10 06:17:04 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:17:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:17:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 06:17:04 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/10 06:17:04 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/10 06:17:10 Rebalance progress: 16.66666666666667
2021/12/10 06:17:15 Rebalance progress: 16.66666666666667
2021/12/10 06:17:20 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:17:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:17:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 06:17:20 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/10 06:17:20 Waiting for src bucket item count to get to: 200000 curr count: 78555
2021/12/10 06:17:25 Waiting for src bucket item count to get to: 200000 curr count: 95091
2021/12/10 06:17:30 Waiting for src bucket item count to get to: 200000 curr count: 95091
2021/12/10 06:17:35 Waiting for src bucket item count to get to: 200000 curr count: 111767
2021/12/10 06:17:40 Waiting for src bucket item count to get to: 200000 curr count: 111767
2021/12/10 06:17:45 Waiting for src bucket item count to get to: 200000 curr count: 128607
2021/12/10 06:17:50 Waiting for src bucket item count to get to: 200000 curr count: 128607
2021/12/10 06:17:55 Waiting for src bucket item count to get to: 200000 curr count: 144885
2021/12/10 06:18:00 Waiting for src bucket item count to get to: 200000 curr count: 144885
2021/12/10 06:18:05 Waiting for src bucket item count to get to: 200000 curr count: 161666
2021/12/10 06:18:10 Waiting for src bucket item count to get to: 200000 curr count: 161666
2021/12/10 06:18:15 Waiting for src bucket item count to get to: 200000 curr count: 177667
2021/12/10 06:18:20 Waiting for src bucket item count to get to: 200000 curr count: 177667
2021/12/10 06:18:25 Waiting for src bucket item count to get to: 200000 curr count: 194303
2021/12/10 06:18:30 Waiting for src bucket item count to get to: 200000 curr count: 194303
2021/12/10 06:18:35 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/10 06:18:35 Post SET ops. Expected item count: 200000 got 200000
2021/12/10 06:18:35 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 06:18:40 Waiting for app: TestDuplicateWithEventingSwapReb to get un-deployed
2021/12/10 06:18:45 Waiting for app: TestDuplicateWithEventingSwapReb to get un-deployed
2021/12/10 06:18:50 Waiting for app: TestDuplicateWithEventingSwapReb to get un-deployed
2021/12/10 06:18:55 Waiting for app: TestDuplicateWithEventingSwapReb to get un-deployed
2021/12/10 06:18:55 App: TestDuplicateWithEventingSwapReb got un-deployed
2021/12/10 06:18:55 No eventing-con process running
2021/12/10 06:18:55 Function: TestDuplicateWithEventingSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithEventingSwapReb deleting in the background"
}
2021/12/10 06:18:57 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:19:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 06:19:07 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/10 06:19:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 06:19:08 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:19:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 06:19:18 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/10 06:19:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithEventingSwapReb (230.75s)
=== RUN   TestDuplicateWithKVRebIn
2021/12/10 06:19:28 Waiting for app: TestDuplicateWithKVRebIn to get un-deployed
2021/12/10 06:19:28 App: TestDuplicateWithKVRebIn got un-deployed
2021/12/10 06:19:28 No eventing-con process running
2021/12/10 06:19:28 Function: TestDuplicateWithKVRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithKVRebIn not found"
 }
}
2021/12/10 06:19:30 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:19:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 06:19:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 06:19:37 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:19:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 06:19:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:19:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 06:19:47 Deploying app: TestDuplicateWithKVRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/10 06:19:48 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithKVRebIn' in metakv",
  "warnings": null
 }
}
2021/12/10 06:19:53 Waiting for app: TestDuplicateWithKVRebIn to get deployed
2021/12/10 06:19:58 Waiting for app: TestDuplicateWithKVRebIn to get deployed
2021/12/10 06:20:03 Waiting for app: TestDuplicateWithKVRebIn to get deployed
2021/12/10 06:20:08 Waiting for app: TestDuplicateWithKVRebIn to get deployed
2021/12/10 06:20:08 App: TestDuplicateWithKVRebIn got deployed
2021/12/10 06:20:08 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/10 06:20:08 Starting bucket ops to source bucket
2021/12/10 06:20:08 Removing node(s): [] from the cluster
2021/12/10 06:20:13 Rebalance progress: 16.66666666666667
2021/12/10 06:20:18 Rebalance progress: 16.66666666666667
2021/12/10 06:20:23 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:20:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:20:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 06:20:23 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/10 06:20:23 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/10 06:20:29 Rebalance progress: 16.66666666666667
2021/12/10 06:20:34 Rebalance progress: 16.66666666666667
2021/12/10 06:20:39 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:20:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:20:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 06:20:39 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/10 06:20:39 Waiting for src bucket item count to get to: 200000 curr count: 33373
2021/12/10 06:20:44 Waiting for src bucket item count to get to: 200000 curr count: 49743
2021/12/10 06:20:49 Waiting for src bucket item count to get to: 200000 curr count: 49743
2021/12/10 06:20:54 Waiting for src bucket item count to get to: 200000 curr count: 66293
2021/12/10 06:20:59 Waiting for src bucket item count to get to: 200000 curr count: 66293
2021/12/10 06:21:04 Waiting for src bucket item count to get to: 200000 curr count: 82277
2021/12/10 06:21:09 Waiting for src bucket item count to get to: 200000 curr count: 82277
2021/12/10 06:21:14 Waiting for src bucket item count to get to: 200000 curr count: 98719
2021/12/10 06:21:19 Waiting for src bucket item count to get to: 200000 curr count: 98719
2021/12/10 06:21:24 Waiting for src bucket item count to get to: 200000 curr count: 115539
2021/12/10 06:21:29 Waiting for src bucket item count to get to: 200000 curr count: 115539
2021/12/10 06:21:34 Waiting for src bucket item count to get to: 200000 curr count: 132317
2021/12/10 06:21:39 Waiting for src bucket item count to get to: 200000 curr count: 132317
2021/12/10 06:21:44 Waiting for src bucket item count to get to: 200000 curr count: 132317
2021/12/10 06:21:49 Waiting for src bucket item count to get to: 200000 curr count: 149008
2021/12/10 06:21:54 Waiting for src bucket item count to get to: 200000 curr count: 149008
2021/12/10 06:21:59 Waiting for src bucket item count to get to: 200000 curr count: 165699
2021/12/10 06:22:04 Waiting for src bucket item count to get to: 200000 curr count: 165699
2021/12/10 06:22:09 Waiting for src bucket item count to get to: 200000 curr count: 181611
2021/12/10 06:22:14 Waiting for src bucket item count to get to: 200000 curr count: 181611
2021/12/10 06:22:19 Waiting for src bucket item count to get to: 200000 curr count: 197992
2021/12/10 06:22:24 Waiting for src bucket item count to get to: 200000 curr count: 197992
2021/12/10 06:22:29 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/10 06:22:29 Post SET ops. Expected item count: 200000 got 200000
2021/12/10 06:22:29 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 06:22:34 Waiting for app: TestDuplicateWithKVRebIn to get un-deployed
2021/12/10 06:22:39 Waiting for app: TestDuplicateWithKVRebIn to get un-deployed
2021/12/10 06:22:44 Waiting for app: TestDuplicateWithKVRebIn to get un-deployed
2021/12/10 06:22:49 Waiting for app: TestDuplicateWithKVRebIn to get un-deployed
2021/12/10 06:22:49 App: TestDuplicateWithKVRebIn got un-deployed
2021/12/10 06:22:49 No eventing-con process running
2021/12/10 06:22:49 Function: TestDuplicateWithKVRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithKVRebIn deleting in the background"
}
2021/12/10 06:22:51 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:22:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 06:23:01 Waiting for dst bucket item count to get to: 0 curr count: 131093
2021/12/10 06:23:06 Waiting for dst bucket item count to get to: 0 curr count: 131093
2021/12/10 06:23:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 06:23:07 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:23:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 06:23:17 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/10 06:23:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithKVRebIn (239.09s)
=== RUN   TestDuplicateWithKVRebOut
2021/12/10 06:23:22 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/10 06:23:23 Removing node(s): [] from the cluster
2021/12/10 06:23:28 Rebalance progress: 100
2021/12/10 06:23:33 Waiting for app: TestDuplicateWithKVRebOut to get un-deployed
2021/12/10 06:23:33 App: TestDuplicateWithKVRebOut got un-deployed
2021/12/10 06:23:33 No eventing-con process running
2021/12/10 06:23:33 Function: TestDuplicateWithKVRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithKVRebOut not found"
 }
}
2021/12/10 06:23:34 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:23:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 06:23:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 06:23:41 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:23:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 06:23:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:23:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 06:23:51 Deploying app: TestDuplicateWithKVRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/10 06:23:52 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithKVRebOut' in metakv",
  "warnings": null
 }
}
2021/12/10 06:23:57 Waiting for app: TestDuplicateWithKVRebOut to get deployed
2021/12/10 06:24:02 Waiting for app: TestDuplicateWithKVRebOut to get deployed
2021/12/10 06:24:07 Waiting for app: TestDuplicateWithKVRebOut to get deployed
2021/12/10 06:24:12 Waiting for app: TestDuplicateWithKVRebOut to get deployed
2021/12/10 06:24:12 App: TestDuplicateWithKVRebOut got deployed
2021/12/10 06:24:12 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/10 06:24:12 Starting bucket ops to source bucket
2021/12/10 06:24:17 Rebalance progress: 16.66666666666667
2021/12/10 06:24:22 Rebalance progress: 16.66666666666667
2021/12/10 06:24:27 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:24:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:24:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 06:24:27 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/10 06:24:27 Waiting for src bucket item count to get to: 200000 curr count: 10561
2021/12/10 06:24:32 Waiting for src bucket item count to get to: 200000 curr count: 26906
2021/12/10 06:24:37 Waiting for src bucket item count to get to: 200000 curr count: 26906
2021/12/10 06:24:42 Waiting for src bucket item count to get to: 200000 curr count: 43364
2021/12/10 06:24:47 Waiting for src bucket item count to get to: 200000 curr count: 43364
2021/12/10 06:24:52 Waiting for src bucket item count to get to: 200000 curr count: 59935
2021/12/10 06:24:57 Waiting for src bucket item count to get to: 200000 curr count: 59935
2021/12/10 06:25:02 Waiting for src bucket item count to get to: 200000 curr count: 59935
2021/12/10 06:25:07 Waiting for src bucket item count to get to: 200000 curr count: 75654
2021/12/10 06:25:12 Waiting for src bucket item count to get to: 200000 curr count: 75654
2021/12/10 06:25:17 Waiting for src bucket item count to get to: 200000 curr count: 91835
2021/12/10 06:25:22 Waiting for src bucket item count to get to: 200000 curr count: 91835
2021/12/10 06:25:28 Waiting for src bucket item count to get to: 200000 curr count: 108723
2021/12/10 06:25:33 Waiting for src bucket item count to get to: 200000 curr count: 108723
2021/12/10 06:25:38 Waiting for src bucket item count to get to: 200000 curr count: 125167
2021/12/10 06:25:43 Waiting for src bucket item count to get to: 200000 curr count: 125167
2021/12/10 06:25:48 Waiting for src bucket item count to get to: 200000 curr count: 141813
2021/12/10 06:25:53 Waiting for src bucket item count to get to: 200000 curr count: 141813
2021/12/10 06:25:58 Waiting for src bucket item count to get to: 200000 curr count: 158609
2021/12/10 06:26:03 Waiting for src bucket item count to get to: 200000 curr count: 158609
2021/12/10 06:26:08 Waiting for src bucket item count to get to: 200000 curr count: 174683
2021/12/10 06:26:13 Waiting for src bucket item count to get to: 200000 curr count: 174683
2021/12/10 06:26:18 Waiting for src bucket item count to get to: 200000 curr count: 190847
2021/12/10 06:26:23 Waiting for src bucket item count to get to: 200000 curr count: 190847
2021/12/10 06:26:28 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/10 06:26:28 Post SET ops. Expected item count: 200000 got 200000
2021/12/10 06:26:28 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 06:26:33 Waiting for app: TestDuplicateWithKVRebOut to get un-deployed
2021/12/10 06:26:38 Waiting for app: TestDuplicateWithKVRebOut to get un-deployed
2021/12/10 06:26:43 Waiting for app: TestDuplicateWithKVRebOut to get un-deployed
2021/12/10 06:26:48 Waiting for app: TestDuplicateWithKVRebOut to get un-deployed
2021/12/10 06:26:48 App: TestDuplicateWithKVRebOut got un-deployed
2021/12/10 06:26:48 No eventing-con process running
2021/12/10 06:26:48 Function: TestDuplicateWithKVRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithKVRebOut deleting in the background"
}
2021/12/10 06:26:49 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:26:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 06:26:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 06:26:56 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:27:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 06:27:06 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/10 06:27:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithKVRebOut (228.85s)
=== RUN   TestDuplicateWithKVSwapReb
2021/12/10 06:27:16 Waiting for app: TestDuplicateWithKVSwapReb to get un-deployed
2021/12/10 06:27:16 App: TestDuplicateWithKVSwapReb got un-deployed
2021/12/10 06:27:16 No eventing-con process running
2021/12/10 06:27:16 Function: TestDuplicateWithKVSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithKVSwapReb not found"
 }
}
2021/12/10 06:27:18 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:27:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 06:27:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 06:27:25 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:27:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 06:27:30 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:27:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 06:27:35 Deploying app: TestDuplicateWithKVSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/10 06:27:36 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithKVSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/10 06:27:41 Waiting for app: TestDuplicateWithKVSwapReb to get deployed
2021/12/10 06:27:46 Waiting for app: TestDuplicateWithKVSwapReb to get deployed
2021/12/10 06:27:51 Waiting for app: TestDuplicateWithKVSwapReb to get deployed
2021/12/10 06:27:56 Waiting for app: TestDuplicateWithKVSwapReb to get deployed
2021/12/10 06:27:56 App: TestDuplicateWithKVSwapReb got deployed
2021/12/10 06:27:56 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/10 06:27:56 Starting bucket ops to source bucket
2021/12/10 06:27:56 Removing node(s): [] from the cluster
2021/12/10 06:28:01 Rebalance progress: 16.66666666666667
2021/12/10 06:28:06 Rebalance progress: 16.66666666666667
2021/12/10 06:28:11 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:28:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:28:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 06:28:11 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/10 06:28:11 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/10 06:28:12 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/10 06:28:17 Rebalance progress: 16.66666666666667
2021/12/10 06:28:22 Rebalance progress: 16.66666666666667
2021/12/10 06:28:27 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:28:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:28:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 06:28:27 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/10 06:28:27 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/10 06:28:32 Rebalance progress: 16.66666666666667
2021/12/10 06:28:37 Rebalance progress: 16.66666666666667
2021/12/10 06:28:42 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:28:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:28:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 06:28:42 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/10 06:28:42 Waiting for src bucket item count to get to: 200000 curr count: 69810
2021/12/10 06:28:47 Waiting for src bucket item count to get to: 200000 curr count: 69810
2021/12/10 06:28:52 Waiting for src bucket item count to get to: 200000 curr count: 85163
2021/12/10 06:28:57 Waiting for src bucket item count to get to: 200000 curr count: 85163
2021/12/10 06:29:02 Waiting for src bucket item count to get to: 200000 curr count: 101713
2021/12/10 06:29:07 Waiting for src bucket item count to get to: 200000 curr count: 101713
2021/12/10 06:29:12 Waiting for src bucket item count to get to: 200000 curr count: 118484
2021/12/10 06:29:17 Waiting for src bucket item count to get to: 200000 curr count: 118484
2021/12/10 06:29:22 Waiting for src bucket item count to get to: 200000 curr count: 135199
2021/12/10 06:29:27 Waiting for src bucket item count to get to: 200000 curr count: 135199
2021/12/10 06:29:32 Waiting for src bucket item count to get to: 200000 curr count: 151952
2021/12/10 06:29:37 Waiting for src bucket item count to get to: 200000 curr count: 151952
2021/12/10 06:29:42 Waiting for src bucket item count to get to: 200000 curr count: 151952
2021/12/10 06:29:47 Waiting for src bucket item count to get to: 200000 curr count: 168447
2021/12/10 06:29:52 Waiting for src bucket item count to get to: 200000 curr count: 168447
2021/12/10 06:29:57 Waiting for src bucket item count to get to: 200000 curr count: 183611
2021/12/10 06:30:02 Waiting for src bucket item count to get to: 200000 curr count: 183611
2021/12/10 06:30:07 Waiting for src bucket item count to get to: 200000 curr count: 199737
2021/12/10 06:30:12 Waiting for src bucket item count to get to: 200000 curr count: 199737
2021/12/10 06:30:17 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/10 06:30:17 Post SET ops. Expected item count: 200000 got 200000
2021/12/10 06:30:17 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 06:30:22 Waiting for app: TestDuplicateWithKVSwapReb to get un-deployed
2021/12/10 06:30:27 Waiting for app: TestDuplicateWithKVSwapReb to get un-deployed
2021/12/10 06:30:32 Waiting for app: TestDuplicateWithKVSwapReb to get un-deployed
2021/12/10 06:30:37 Waiting for app: TestDuplicateWithKVSwapReb to get un-deployed
2021/12/10 06:30:37 App: TestDuplicateWithKVSwapReb got un-deployed
2021/12/10 06:30:37 No eventing-con process running
2021/12/10 06:30:37 Function: TestDuplicateWithKVSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithKVSwapReb deleting in the background"
}
2021/12/10 06:30:39 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:30:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 06:30:49 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/10 06:30:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 06:30:51 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:30:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 06:31:01 Waiting for dst bucket item count to get to: 0 curr count: 135406
2021/12/10 06:31:06 Waiting for dst bucket item count to get to: 0 curr count: 135406
2021/12/10 06:31:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithKVSwapReb (239.37s)
=== RUN   TestEventingRebKVOpsOnUpdateBucketOpOneByOne
2021/12/10 06:31:16 Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/10 06:31:16 App: TestEventingRebKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/10 06:31:16 No eventing-con process running
2021/12/10 06:31:16 Function: TestEventingRebKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpOneByOne not found"
 }
}
2021/12/10 06:31:17 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:31:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 06:31:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 06:31:24 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:31:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 06:31:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:31:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 06:31:34 Deploying app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne. Item count src bucket: 0 dst bucket: 0
2021/12/10 06:31:35 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebKVOpsOnUpdateBucketOpOneByOne' in metakv",
  "warnings": null
 }
}
2021/12/10 06:31:40 Starting bucket ops to source bucket
2021/12/10 06:31:45 Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/10 06:31:50 Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/10 06:31:55 Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/10 06:31:55 App: TestEventingRebKVOpsOnUpdateBucketOpOneByOne got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:31:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:31:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 06:31:55 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/10 06:31:55 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/10 06:31:57 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/10 06:32:02 Rebalance progress: 16.66666666666667
2021/12/10 06:32:07 Rebalance progress: 16.66666666666667
2021/12/10 06:32:12 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:32:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:32:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 06:32:12 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/10 06:32:12 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/10 06:32:12 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/10 06:32:17 Rebalance progress: 16.66666666666667
2021/12/10 06:32:22 Rebalance progress: 16.66666666666667
2021/12/10 06:32:27 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:32:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:32:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 06:32:27 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/10 06:32:27 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/10 06:32:33 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/10 06:32:38 Rebalance progress: 12.5
2021/12/10 06:32:43 Rebalance progress: 12.5
2021/12/10 06:32:48 Rebalance progress: 12.5
2021/12/10 06:32:53 Rebalance progress: 36.91860465116279
2021/12/10 06:32:58 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:32:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:32:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 06:32:58 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/10 06:33:03 Rebalance progress: 12.5
2021/12/10 06:33:08 Rebalance progress: 12.5
2021/12/10 06:33:13 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:33:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:33:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 06:33:13 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/10 06:33:18 Rebalance progress: 12.5
2021/12/10 06:33:23 Rebalance progress: 12.5
2021/12/10 06:33:28 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:33:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:33:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 06:33:28 clusterSetup.go:253: Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/10 06:33:33 Rebalance progress: 12.5
2021/12/10 06:33:38 Rebalance progress: 12.5
2021/12/10 06:33:43 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:33:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:33:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 06:33:43 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 06:33:48 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/10 06:33:53 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/10 06:33:58 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/10 06:34:03 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/10 06:34:03 clusterSetup.go:405: App: TestEventingRebKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/10 06:34:03 eventingSetup.go:46: No eventing-con process running
2021/12/10 06:34:03 eventingSetup.go:418: Function: TestEventingRebKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpOneByOne deleting in the background"
}
2021/12/10 06:34:05 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:34:10 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 06:34:15 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12272
2021/12/10 06:34:15 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 06:34:17 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:34:22 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 06:34:27 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12272
2021/12/10 06:34:27 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebKVOpsOnUpdateBucketOpOneByOne (201.12s)
=== RUN   TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce
2021/12/10 06:34:37 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get un-deployed
2021/12/10 06:34:37 clusterSetup.go:405: App: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce got un-deployed
2021/12/10 06:34:37 eventingSetup.go:46: No eventing-con process running
2021/12/10 06:34:37 eventingSetup.go:418: Function: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce not found"
 }
}
2021/12/10 06:34:38 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:34:43 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 06:34:43 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 06:34:45 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:34:50 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 06:34:50 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:34:55 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 06:34:55 eventingSetup.go:125: Deploying app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce. Item count src bucket: 0 dst bucket: 0
2021/12/10 06:34:56 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce' in metakv",
  "warnings": null
 }
}
2021/12/10 06:35:01 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/10 06:35:06 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get deployed
2021/12/10 06:35:11 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get deployed
2021/12/10 06:35:16 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get deployed
2021/12/10 06:35:16 clusterSetup.go:354: App: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:35:16 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:35:16 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 06:35:16 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/10 06:35:17 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/10 06:35:17 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/10 06:35:17 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/10 06:35:22 Rebalance progress: 12.5
2021/12/10 06:35:27 Rebalance progress: 12.5
2021/12/10 06:35:32 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:35:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:35:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 06:35:32 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002 127.0.0.1:19003] from the cluster
2021/12/10 06:35:38 Rebalance progress: 12.5
2021/12/10 06:35:43 Rebalance progress: 12.5
2021/12/10 06:35:48 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:35:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:35:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 06:35:48 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 06:35:53 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get un-deployed
2021/12/10 06:35:58 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get un-deployed
2021/12/10 06:36:03 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get un-deployed
2021/12/10 06:36:08 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get un-deployed
2021/12/10 06:36:08 clusterSetup.go:405: App: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce got un-deployed
2021/12/10 06:36:08 eventingSetup.go:46: No eventing-con process running
2021/12/10 06:36:08 eventingSetup.go:418: Function: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce deleting in the background"
}
2021/12/10 06:36:10 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:36:15 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 06:36:15 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 06:36:16 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:36:21 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 06:36:26 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4603
2021/12/10 06:36:26 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce (119.55s)
=== RUN   TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings
2021/12/10 06:36:36 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/10 06:36:36 clusterSetup.go:405: App: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings got un-deployed
2021/12/10 06:36:36 eventingSetup.go:46: No eventing-con process running
2021/12/10 06:36:36 eventingSetup.go:418: Function: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings not found"
 }
}
2021/12/10 06:36:38 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:36:43 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 06:36:43 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 06:36:45 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:36:50 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 06:36:50 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:36:55 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 06:36:55 eventingSetup.go:125: Deploying app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/10 06:36:56 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/10 06:37:01 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/10 06:37:06 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get deployed
2021/12/10 06:37:11 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get deployed
2021/12/10 06:37:16 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get deployed
2021/12/10 06:37:16 clusterSetup.go:354: App: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:37:16 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:37:16 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 06:37:16 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/10 06:37:16 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/10 06:37:21 Rebalance progress: 12.5
2021/12/10 06:37:26 Rebalance progress: 12.5
2021/12/10 06:37:31 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:37:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:37:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 06:37:31 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/10 06:37:31 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/10 06:37:36 Rebalance progress: 12.5
2021/12/10 06:37:41 Rebalance progress: 12.5
2021/12/10 06:37:46 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:37:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:37:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 06:37:47 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/10 06:37:47 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/10 06:37:52 Rebalance progress: 12.5
2021/12/10 06:37:57 Rebalance progress: 12.5
2021/12/10 06:38:02 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:38:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:38:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 06:38:02 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/10 06:38:07 Rebalance progress: 12.5
2021/12/10 06:38:12 Rebalance progress: 12.5
2021/12/10 06:38:17 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:38:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:38:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 06:38:17 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/10 06:38:22 Rebalance progress: 12.5
2021/12/10 06:38:27 Rebalance progress: 12.5
2021/12/10 06:38:32 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:38:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:38:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 06:38:32 clusterSetup.go:253: Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/10 06:38:37 Rebalance progress: 12.5
2021/12/10 06:38:42 Rebalance progress: 12.5
2021/12/10 06:38:47 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:38:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:38:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 06:38:47 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 06:38:52 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/10 06:38:57 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/10 06:39:02 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/10 06:39:07 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/10 06:39:07 clusterSetup.go:405: App: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings got un-deployed
2021/12/10 06:39:07 eventingSetup.go:46: No eventing-con process running
2021/12/10 06:39:07 eventingSetup.go:418: Function: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings deleting in the background"
}
2021/12/10 06:39:09 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:39:14 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 06:39:19 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 10621
2021/12/10 06:39:19 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 06:39:21 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:39:26 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 06:39:31 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 9202
2021/12/10 06:39:36 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 9202
2021/12/10 06:39:36 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings (189.59s)
=== RUN   TestEventingSwapRebOnUpdateBucketOp
2021/12/10 06:39:46 clusterSetup.go:397: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get un-deployed
2021/12/10 06:39:46 clusterSetup.go:405: App: TestEventingSwapRebOnUpdateBucketOp got un-deployed
2021/12/10 06:39:46 eventingSetup.go:46: No eventing-con process running
2021/12/10 06:39:46 eventingSetup.go:418: Function: TestEventingSwapRebOnUpdateBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingSwapRebOnUpdateBucketOp not found"
 }
}
2021/12/10 06:39:48 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:39:53 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 06:39:53 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 06:39:54 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:39:59 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 06:39:59 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:39:59 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 06:39:59 eventingSetup.go:125: Deploying app: TestEventingSwapRebOnUpdateBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/10 06:40:00 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingSwapRebOnUpdateBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/10 06:40:05 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/10 06:40:10 clusterSetup.go:346: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get deployed
2021/12/10 06:40:15 clusterSetup.go:346: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get deployed
2021/12/10 06:40:20 clusterSetup.go:346: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get deployed
2021/12/10 06:40:20 clusterSetup.go:354: App: TestEventingSwapRebOnUpdateBucketOp got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:40:20 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:40:20 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 06:40:21 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/10 06:40:21 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/10 06:40:26 Rebalance progress: 12.5
2021/12/10 06:40:31 Rebalance progress: 12.5
2021/12/10 06:40:36 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:40:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:40:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 06:40:36 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/10 06:40:36 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/10 06:40:41 Rebalance progress: 12.5
2021/12/10 06:40:46 Rebalance progress: 12.5
2021/12/10 06:40:51 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:40:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:40:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 06:40:51 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/10 06:40:56 Rebalance progress: 12.5
2021/12/10 06:41:01 Rebalance progress: 12.5
2021/12/10 06:41:06 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:41:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:41:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 06:41:07 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 06:41:12 clusterSetup.go:397: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get un-deployed
2021/12/10 06:41:17 clusterSetup.go:397: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get un-deployed
2021/12/10 06:41:22 clusterSetup.go:397: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get un-deployed
2021/12/10 06:41:27 clusterSetup.go:397: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get un-deployed
2021/12/10 06:41:27 clusterSetup.go:405: App: TestEventingSwapRebOnUpdateBucketOp got un-deployed
2021/12/10 06:41:27 eventingSetup.go:46: No eventing-con process running
2021/12/10 06:41:27 eventingSetup.go:418: Function: TestEventingSwapRebOnUpdateBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingSwapRebOnUpdateBucketOp deleting in the background"
}
2021/12/10 06:41:28 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:41:33 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 06:41:38 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6096
2021/12/10 06:41:38 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 06:41:40 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:41:45 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 06:41:50 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1774
2021/12/10 06:41:55 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1774
2021/12/10 06:41:55 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingSwapRebOnUpdateBucketOp (139.36s)
=== RUN   TestMetaRollbackWithEventingReb
2021/12/10 06:42:00 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/10 06:42:01 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/10 06:42:06 Rebalance progress: 100
2021/12/10 06:42:11 Waiting for app: TestMetaRollbackWithEventingReb to get un-deployed
2021/12/10 06:42:11 App: TestMetaRollbackWithEventingReb got un-deployed
2021/12/10 06:42:11 No eventing-con process running
2021/12/10 06:42:11 Function: TestMetaRollbackWithEventingReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMetaRollbackWithEventingReb not found"
 }
}
2021/12/10 06:42:13 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:42:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 06:42:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 06:42:19 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:42:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 06:42:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:42:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 06:42:24 Deploying app: TestMetaRollbackWithEventingReb. Item count src bucket: 0 dst bucket: 0
2021/12/10 06:42:25 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMetaRollbackWithEventingReb' in metakv",
  "warnings": null
 }
}
2021/12/10 06:42:30 Starting bucket ops to source bucket
2021/12/10 06:42:35 Waiting for app: TestMetaRollbackWithEventingReb to get deployed
2021/12/10 06:42:40 Waiting for app: TestMetaRollbackWithEventingReb to get deployed
2021/12/10 06:42:45 Waiting for app: TestMetaRollbackWithEventingReb to get deployed
2021/12/10 06:42:45 App: TestMetaRollbackWithEventingReb got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:42:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:42:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 06:42:45 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/10 06:42:46 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/10 06:42:51 Rebalance progress: 12.5
2021/12/10 06:42:56 Rebalance progress: 12.5
2021/12/10 06:43:01 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:43:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:43:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 06:43:01 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002] from the cluster
2021/12/10 06:43:01 bucketOps.go:134: Metakv lookup failed, err: Get "/_metakv/eventing/tempApps/TestMetaRollbackWithEventingReb/0": Unable to initialize cbauth's revrpc: cbauth environment variable CBAUTH_REVRPC_URL is not set
2021/12/10 06:43:06 Rebalance progress: 12.5
2021/12/10 06:43:11 Rebalance progress: 12.5
2021/12/10 06:43:16 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:43:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:43:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 06:43:16 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 06:43:21 clusterSetup.go:397: Waiting for app: TestMetaRollbackWithEventingReb to get un-deployed
2021/12/10 06:43:26 clusterSetup.go:397: Waiting for app: TestMetaRollbackWithEventingReb to get un-deployed
2021/12/10 06:43:31 clusterSetup.go:397: Waiting for app: TestMetaRollbackWithEventingReb to get un-deployed
2021/12/10 06:43:36 clusterSetup.go:397: Waiting for app: TestMetaRollbackWithEventingReb to get un-deployed
2021/12/10 06:43:36 clusterSetup.go:405: App: TestMetaRollbackWithEventingReb got un-deployed
2021/12/10 06:43:36 eventingSetup.go:46: No eventing-con process running
2021/12/10 06:43:36 eventingSetup.go:418: Function: TestMetaRollbackWithEventingReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMetaRollbackWithEventingReb deleting in the background"
}
2021/12/10 06:43:38 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:43:43 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 06:43:48 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4540
2021/12/10 06:43:48 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 06:43:50 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:43:55 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 06:43:55 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestMetaRollbackWithEventingReb (119.40s)
=== RUN   TestMetaPartialRollbackWithEventingReb
2021/12/10 06:44:00 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/10 06:44:00 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/10 06:44:05 Rebalance progress: 100
2021/12/10 06:44:10 Waiting for app: TestMetaPartialRollbackWithEventingReb to get un-deployed
2021/12/10 06:44:10 App: TestMetaPartialRollbackWithEventingReb got un-deployed
2021/12/10 06:44:10 No eventing-con process running
2021/12/10 06:44:10 Function: TestMetaPartialRollbackWithEventingReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMetaPartialRollbackWithEventingReb not found"
 }
}
2021/12/10 06:44:12 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:44:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 06:44:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 06:44:18 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:44:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 06:44:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:44:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 06:44:23 Deploying app: TestMetaPartialRollbackWithEventingReb. Item count src bucket: 0 dst bucket: 0
2021/12/10 06:44:24 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMetaPartialRollbackWithEventingReb' in metakv",
  "warnings": null
 }
}
2021/12/10 06:44:29 Starting bucket ops to source bucket
2021/12/10 06:44:34 Waiting for app: TestMetaPartialRollbackWithEventingReb to get deployed
2021/12/10 06:44:39 Waiting for app: TestMetaPartialRollbackWithEventingReb to get deployed
2021/12/10 06:44:44 Waiting for app: TestMetaPartialRollbackWithEventingReb to get deployed
2021/12/10 06:44:45 App: TestMetaPartialRollbackWithEventingReb got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:44:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:44:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 06:44:45 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/10 06:44:45 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/10 06:44:50 Rebalance progress: 12.5
2021/12/10 06:44:55 Rebalance progress: 12.5
2021/12/10 06:45:00 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:45:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:45:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 06:45:00 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002] from the cluster
2021/12/10 06:45:00 bucketOps.go:65: Metakv lookup failed, err: Get "/_metakv/eventing/tempApps/TestMetaPartialRollbackWithEventingReb/0": Unable to initialize cbauth's revrpc: cbauth environment variable CBAUTH_REVRPC_URL is not set
2021/12/10 06:45:05 Rebalance progress: 12.5
2021/12/10 06:45:10 Rebalance progress: 12.5
2021/12/10 06:45:15 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:45:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:45:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 06:45:15 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 06:45:15 bucketOps.go:65: Metakv lookup failed, err: Get "/_metakv/eventing/tempApps/TestMetaPartialRollbackWithEventingReb/0": Unable to initialize cbauth's revrpc: cbauth environment variable CBAUTH_REVRPC_URL is not set
2021/12/10 06:45:20 clusterSetup.go:397: Waiting for app: TestMetaPartialRollbackWithEventingReb to get un-deployed
2021/12/10 06:45:25 clusterSetup.go:397: Waiting for app: TestMetaPartialRollbackWithEventingReb to get un-deployed
2021/12/10 06:45:30 clusterSetup.go:397: Waiting for app: TestMetaPartialRollbackWithEventingReb to get un-deployed
2021/12/10 06:45:35 clusterSetup.go:397: Waiting for app: TestMetaPartialRollbackWithEventingReb to get un-deployed
2021/12/10 06:45:35 clusterSetup.go:405: App: TestMetaPartialRollbackWithEventingReb got un-deployed
2021/12/10 06:45:35 eventingSetup.go:46: No eventing-con process running
2021/12/10 06:45:35 eventingSetup.go:418: Function: TestMetaPartialRollbackWithEventingReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMetaPartialRollbackWithEventingReb deleting in the background"
}
2021/12/10 06:45:37 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:45:42 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 06:45:47 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 37800
2021/12/10 06:45:47 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 06:45:49 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:45:54 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 06:45:59 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 37800
2021/12/10 06:45:59 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestMetaPartialRollbackWithEventingReb (124.15s)
=== RUN   TestEventingRebMultipleHandlersOneByOne
2021/12/10 06:46:09 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get un-deployed
2021/12/10 06:46:09 clusterSetup.go:405: App: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op got un-deployed
2021/12/10 06:46:09 eventingSetup.go:46: No eventing-con process running
2021/12/10 06:46:09 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op not found"
 }
}
2021/12/10 06:46:11 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:46:16 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 06:46:16 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 06:46:17 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:46:22 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 06:46:22 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:46:27 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 06:46:27 eventingSetup.go:125: Deploying app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op. Item count src bucket: 0 dst bucket: 0
2021/12/10 06:46:29 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op' in metakv",
  "warnings": null
 }
}
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:46:29 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 06:46:29 eventingSetup.go:125: Deploying app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/10 06:46:30 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebMultipleHandlersOneByOnebucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/10 06:46:35 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/10 06:46:40 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get deployed
2021/12/10 06:46:45 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get deployed
2021/12/10 06:46:50 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get deployed
2021/12/10 06:46:50 clusterSetup.go:354: App: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op got deployed
2021/12/10 06:46:55 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get deployed
2021/12/10 06:47:00 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get deployed
2021/12/10 06:47:05 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get deployed
2021/12/10 06:47:05 clusterSetup.go:354: App: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:47:05 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:47:05 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 06:47:05 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/10 06:47:05 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/10 06:47:10 Rebalance progress: 12.5
2021/12/10 06:47:15 Rebalance progress: 12.5
2021/12/10 06:47:20 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:47:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:47:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 06:47:20 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/10 06:47:21 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/10 06:47:26 Rebalance progress: 12.5
2021/12/10 06:47:31 Rebalance progress: 12.5
2021/12/10 06:47:36 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:47:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:47:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 06:47:36 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/10 06:47:36 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/10 06:47:41 Rebalance progress: 12.5
2021/12/10 06:47:46 Rebalance progress: 12.5
2021/12/10 06:47:51 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:47:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:47:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 06:47:51 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/10 06:47:56 Rebalance progress: 12.5
2021/12/10 06:48:01 Rebalance progress: 12.5
2021/12/10 06:48:06 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:48:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:48:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 06:48:06 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/10 06:48:11 Rebalance progress: 12.5
2021/12/10 06:48:16 Rebalance progress: 12.5
2021/12/10 06:48:21 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:48:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:48:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 06:48:21 clusterSetup.go:253: Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/10 06:48:26 Rebalance progress: 12.5
2021/12/10 06:48:31 Rebalance progress: 12.5
2021/12/10 06:48:36 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:48:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:48:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 06:48:36 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 06:48:41 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get un-deployed
2021/12/10 06:48:47 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get un-deployed
2021/12/10 06:48:52 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get un-deployed
2021/12/10 06:48:57 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get un-deployed
2021/12/10 06:48:57 clusterSetup.go:405: App: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op got un-deployed
2021/12/10 06:48:57 eventingSetup.go:46: No eventing-con process running
2021/12/10 06:48:57 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op deleting in the background"
}
2021/12/10 06:49:00 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:49:05 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 06:49:05 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 06:49:07 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:49:12 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 06:49:17 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12163
2021/12/10 06:49:17 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 06:49:17 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 06:49:22 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get un-deployed
2021/12/10 06:49:27 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get un-deployed
2021/12/10 06:49:32 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get un-deployed
2021/12/10 06:49:37 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get un-deployed
2021/12/10 06:49:37 clusterSetup.go:405: App: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update got un-deployed
2021/12/10 06:49:37 eventingSetup.go:46: No eventing-con process running
2021/12/10 06:49:37 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update deleting in the background"
}
2021/12/10 06:49:39 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:49:44 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 06:49:44 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 06:49:46 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:49:51 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 06:49:51 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebMultipleHandlersOneByOne (232.00s)
=== RUN   TestEventingRebMultipleHandlersAllAtOnce
2021/12/10 06:50:01 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get un-deployed
2021/12/10 06:50:01 clusterSetup.go:405: App: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op got un-deployed
2021/12/10 06:50:01 eventingSetup.go:46: No eventing-con process running
2021/12/10 06:50:01 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op not found"
 }
}
2021/12/10 06:50:03 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:50:08 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 06:50:08 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 06:50:09 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:50:14 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 06:50:14 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:50:19 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 06:50:19 eventingSetup.go:125: Deploying app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op. Item count src bucket: 0 dst bucket: 0
2021/12/10 06:50:20 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op' in metakv",
  "warnings": null
 }
}
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:50:20 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 06:50:20 eventingSetup.go:125: Deploying app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/10 06:50:22 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/10 06:50:27 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/10 06:50:32 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get deployed
2021/12/10 06:50:37 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get deployed
2021/12/10 06:50:42 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get deployed
2021/12/10 06:50:42 clusterSetup.go:354: App: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op got deployed
2021/12/10 06:50:47 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get deployed
2021/12/10 06:50:52 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get deployed
2021/12/10 06:50:57 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get deployed
2021/12/10 06:50:57 clusterSetup.go:354: App: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:50:57 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:50:57 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 06:50:57 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/10 06:50:57 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/10 06:50:57 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/10 06:50:57 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/10 06:51:02 Rebalance progress: 12.5
2021/12/10 06:51:07 Rebalance progress: 12.5
2021/12/10 06:51:12 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:51:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:51:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 06:51:13 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002 127.0.0.1:19003] from the cluster
2021/12/10 06:51:18 Rebalance progress: 12.5
2021/12/10 06:51:23 Rebalance progress: 12.5
2021/12/10 06:51:28 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:51:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:51:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 06:51:28 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 06:51:33 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get un-deployed
2021/12/10 06:51:38 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get un-deployed
2021/12/10 06:51:43 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get un-deployed
2021/12/10 06:51:48 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get un-deployed
2021/12/10 06:51:48 clusterSetup.go:405: App: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op got un-deployed
2021/12/10 06:51:48 eventingSetup.go:46: No eventing-con process running
2021/12/10 06:51:48 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op deleting in the background"
}
2021/12/10 06:51:52 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:51:57 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 06:52:02 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2213
2021/12/10 06:52:07 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2213
2021/12/10 06:52:07 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 06:52:08 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:52:13 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 06:52:18 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6084
2021/12/10 06:52:18 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 06:52:19 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 06:52:24 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get un-deployed
2021/12/10 06:52:29 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get un-deployed
2021/12/10 06:52:34 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get un-deployed
2021/12/10 06:52:39 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get un-deployed
2021/12/10 06:52:39 clusterSetup.go:405: App: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update got un-deployed
2021/12/10 06:52:39 eventingSetup.go:46: No eventing-con process running
2021/12/10 06:52:39 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update deleting in the background"
}
2021/12/10 06:52:40 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:52:45 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 06:52:45 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 06:52:47 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:52:52 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 06:52:52 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebMultipleHandlersAllAtOnce (181.39s)
=== RUN   TestEventingSwapRebMultipleHandlers
2021/12/10 06:53:02 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get un-deployed
2021/12/10 06:53:02 clusterSetup.go:405: App: TestEventingSwapRebMultipleHandlersbucket_op_on_update got un-deployed
2021/12/10 06:53:02 eventingSetup.go:46: No eventing-con process running
2021/12/10 06:53:02 eventingSetup.go:418: Function: TestEventingSwapRebMultipleHandlersbucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingSwapRebMultipleHandlersbucket_op_on_update not found"
 }
}
2021/12/10 06:53:04 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:53:09 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 06:53:09 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 06:53:11 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:53:16 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 06:53:16 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 06:53:21 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get un-deployed
2021/12/10 06:53:21 clusterSetup.go:405: App: TestEventingSwapRebMultipleHandlerssys_test_bucket_op got un-deployed
2021/12/10 06:53:21 eventingSetup.go:46: No eventing-con process running
2021/12/10 06:53:21 eventingSetup.go:418: Function: TestEventingSwapRebMultipleHandlerssys_test_bucket_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingSwapRebMultipleHandlerssys_test_bucket_op not found"
 }
}
2021/12/10 06:53:23 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:53:28 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 06:53:28 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 06:53:29 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:53:34 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 06:53:34 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:53:34 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 06:53:34 eventingSetup.go:125: Deploying app: TestEventingSwapRebMultipleHandlersbucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/10 06:53:35 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingSwapRebMultipleHandlersbucket_op_on_update' in metakv",
  "warnings": null
 }
}
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:53:35 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 06:53:35 eventingSetup.go:125: Deploying app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op. Item count src bucket: 0 dst bucket: 0
2021/12/10 06:53:37 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingSwapRebMultipleHandlerssys_test_bucket_op' in metakv",
  "warnings": null
 }
}
2021/12/10 06:53:42 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/10 06:53:47 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get deployed
2021/12/10 06:53:52 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get deployed
2021/12/10 06:53:52 clusterSetup.go:354: App: TestEventingSwapRebMultipleHandlersbucket_op_on_update got deployed
2021/12/10 06:53:57 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get deployed
2021/12/10 06:54:02 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get deployed
2021/12/10 06:54:07 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get deployed
2021/12/10 06:54:12 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get deployed
2021/12/10 06:54:12 clusterSetup.go:354: App: TestEventingSwapRebMultipleHandlerssys_test_bucket_op got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:54:12 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:54:12 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 06:54:12 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/10 06:54:12 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/10 06:54:17 Rebalance progress: 12.5
2021/12/10 06:54:22 Rebalance progress: 12.5
2021/12/10 06:54:27 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:54:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:54:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 06:54:27 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/10 06:54:27 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/10 06:54:32 Rebalance progress: 12.5
2021/12/10 06:54:37 Rebalance progress: 12.5
2021/12/10 06:54:42 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:54:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:54:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 06:54:43 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/10 06:54:48 Rebalance progress: 12.5
2021/12/10 06:54:53 Rebalance progress: 12.5
2021/12/10 06:54:58 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:54:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:54:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 06:54:58 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 06:55:03 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get un-deployed
2021/12/10 06:55:08 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get un-deployed
2021/12/10 06:55:13 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get un-deployed
2021/12/10 06:55:18 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get un-deployed
2021/12/10 06:55:18 clusterSetup.go:405: App: TestEventingSwapRebMultipleHandlersbucket_op_on_update got un-deployed
2021/12/10 06:55:18 eventingSetup.go:46: No eventing-con process running
2021/12/10 06:55:18 eventingSetup.go:418: Function: TestEventingSwapRebMultipleHandlersbucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingSwapRebMultipleHandlersbucket_op_on_update deleting in the background"
}
2021/12/10 06:55:21 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:55:26 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 06:55:31 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3098
2021/12/10 06:55:36 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3098
2021/12/10 06:55:36 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 06:55:38 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:55:43 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 06:55:48 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7603
2021/12/10 06:55:48 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 06:55:48 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 06:55:53 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get un-deployed
2021/12/10 06:55:58 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get un-deployed
2021/12/10 06:56:03 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get un-deployed
2021/12/10 06:56:08 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get un-deployed
2021/12/10 06:56:08 clusterSetup.go:405: App: TestEventingSwapRebMultipleHandlerssys_test_bucket_op got un-deployed
2021/12/10 06:56:08 eventingSetup.go:46: No eventing-con process running
2021/12/10 06:56:08 eventingSetup.go:418: Function: TestEventingSwapRebMultipleHandlerssys_test_bucket_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingSwapRebMultipleHandlerssys_test_bucket_op deleting in the background"
}
2021/12/10 06:56:10 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:56:15 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 06:56:15 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 06:56:17 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:56:22 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 06:56:22 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingSwapRebMultipleHandlers (209.54s)
=== RUN   TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne
2021/12/10 06:56:32 clusterSetup.go:397: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/10 06:56:32 clusterSetup.go:405: App: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/10 06:56:32 eventingSetup.go:46: No eventing-con process running
2021/12/10 06:56:32 eventingSetup.go:418: Function: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne not found"
 }
}
2021/12/10 06:56:33 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:56:38 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 06:56:38 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 06:56:40 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:56:45 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 06:56:45 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:56:50 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 06:56:50 eventingSetup.go:125: Deploying app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne. Item count src bucket: 0 dst bucket: 0
2021/12/10 06:56:51 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne' in metakv",
  "warnings": null
 }
}
2021/12/10 06:56:56 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/10 06:57:01 clusterSetup.go:346: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/10 06:57:06 clusterSetup.go:346: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/10 06:57:11 clusterSetup.go:346: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/10 06:57:11 clusterSetup.go:354: App: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:57:11 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:57:11 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 06:57:11 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/10 06:57:12 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/10 06:57:32 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:57:33 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:57:33 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:58:03 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:58:03 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 06:58:03 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/10 06:58:08 Rebalance progress: 12.5
2021/12/10 06:58:13 Rebalance progress: 12.5
2021/12/10 06:58:18 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:58:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:58:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 06:58:18 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/10 06:58:18 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/10 06:58:23 Rebalance progress: 12.5
2021/12/10 06:58:28 Rebalance progress: 12.5
2021/12/10 06:58:33 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:58:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:58:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 06:58:33 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/10 06:58:38 Rebalance progress: 12.5
2021/12/10 06:58:43 Rebalance progress: 12.5
2021/12/10 06:58:48 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:58:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:58:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 06:58:48 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 06:58:53 clusterSetup.go:397: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/10 06:58:58 clusterSetup.go:397: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/10 06:59:03 clusterSetup.go:397: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/10 06:59:08 clusterSetup.go:397: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/10 06:59:08 clusterSetup.go:405: App: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/10 06:59:08 eventingSetup.go:46: No eventing-con process running
2021/12/10 06:59:08 eventingSetup.go:418: Function: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne deleting in the background"
}
2021/12/10 06:59:10 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:59:15 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 06:59:20 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2579
2021/12/10 06:59:25 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2579
2021/12/10 06:59:25 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 06:59:27 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:59:32 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 06:59:37 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 11175
2021/12/10 06:59:37 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne (195.21s)
=== RUN   TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne
2021/12/10 06:59:47 clusterSetup.go:397: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/10 06:59:47 clusterSetup.go:405: App: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/10 06:59:47 eventingSetup.go:46: No eventing-con process running
2021/12/10 06:59:47 eventingSetup.go:418: Function: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne not found"
 }
}
2021/12/10 06:59:49 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 06:59:54 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 06:59:54 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 06:59:55 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 07:00:00 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 07:00:00 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 07:00:05 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 07:00:05 eventingSetup.go:125: Deploying app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne. Item count src bucket: 0 dst bucket: 0
2021/12/10 07:00:07 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne' in metakv",
  "warnings": null
 }
}
2021/12/10 07:00:12 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/10 07:00:17 clusterSetup.go:346: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/10 07:00:22 clusterSetup.go:346: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/10 07:00:27 clusterSetup.go:346: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/10 07:00:27 clusterSetup.go:354: App: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 07:00:27 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 07:00:27 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 07:00:27 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/10 07:00:27 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/10 07:00:47 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 07:00:47 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 07:00:47 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 07:00:57 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 07:00:57 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 07:00:57 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/10 07:00:58 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/10 07:01:18 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 07:01:18 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 07:01:18 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 07:01:28 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 07:01:28 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 07:01:28 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/10 07:01:29 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/10 07:01:49 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 07:01:49 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 07:01:49 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 07:01:59 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 07:01:59 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 07:01:59 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/10 07:02:00 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/10 07:02:20 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 07:02:20 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 07:02:20 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 07:02:30 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 07:02:30 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 07:02:30 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/10 07:02:31 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/10 07:02:51 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 07:02:51 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 07:02:51 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 07:03:01 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 07:03:01 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 07:03:01 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/10 07:03:06 Rebalance progress: 12.5
2021/12/10 07:03:11 Rebalance progress: 12.5
2021/12/10 07:03:16 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 07:03:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 07:03:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 07:03:16 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/10 07:03:17 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/10 07:03:22 Rebalance progress: 12.5
2021/12/10 07:03:27 Rebalance progress: 12.5
2021/12/10 07:03:32 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 07:03:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 07:03:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 07:03:32 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/10 07:03:37 Rebalance progress: 12.5
2021/12/10 07:03:42 Rebalance progress: 12.5
2021/12/10 07:03:47 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 07:03:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 07:03:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 07:03:47 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 07:03:52 clusterSetup.go:397: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/10 07:03:57 clusterSetup.go:397: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/10 07:04:02 clusterSetup.go:397: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/10 07:04:07 clusterSetup.go:397: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/10 07:04:07 clusterSetup.go:405: App: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/10 07:04:07 eventingSetup.go:46: No eventing-con process running
2021/12/10 07:04:07 eventingSetup.go:418: Function: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne deleting in the background"
}
2021/12/10 07:04:09 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 07:04:14 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 07:04:19 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 21494
2021/12/10 07:04:19 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 07:04:20 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 07:04:25 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 07:04:30 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 11868
2021/12/10 07:04:35 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 11868
2021/12/10 07:04:35 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne (298.52s)
=== RUN   TestEventingFailoverOnUpdateBucketOp
2021/12/10 07:04:45 clusterSetup.go:397: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get un-deployed
2021/12/10 07:04:45 clusterSetup.go:405: App: TestEventingFailoverOnUpdateBucketOp got un-deployed
2021/12/10 07:04:45 eventingSetup.go:46: No eventing-con process running
2021/12/10 07:04:45 eventingSetup.go:418: Function: TestEventingFailoverOnUpdateBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingFailoverOnUpdateBucketOp not found"
 }
}
2021/12/10 07:04:47 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 07:04:52 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 07:04:52 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 07:04:54 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 07:04:59 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 07:04:59 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 07:05:04 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 07:05:04 eventingSetup.go:125: Deploying app: TestEventingFailoverOnUpdateBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/10 07:05:05 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingFailoverOnUpdateBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/10 07:05:10 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/10 07:05:15 clusterSetup.go:346: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get deployed
2021/12/10 07:05:20 clusterSetup.go:346: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get deployed
2021/12/10 07:05:25 clusterSetup.go:346: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get deployed
2021/12/10 07:05:25 clusterSetup.go:354: App: TestEventingFailoverOnUpdateBucketOp got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 07:05:25 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 07:05:25 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 07:05:25 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/10 07:05:25 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/10 07:05:30 Rebalance progress: 12.5
2021/12/10 07:05:35 Rebalance progress: 12.5
2021/12/10 07:05:40 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 07:05:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 07:05:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 07:05:40 clusterSetup.go:229: Failing over: [127.0.0.1:19001]
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 07:05:50 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 07:05:50 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 07:05:50 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/10 07:05:56 Rebalance progress: 12.5
2021/12/10 07:06:01 Rebalance progress: 12.5
2021/12/10 07:06:06 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 07:06:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 07:06:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 07:06:06 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/10 07:06:06 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/10 07:06:11 Rebalance progress: 12.5
2021/12/10 07:06:16 Rebalance progress: 12.5
2021/12/10 07:06:21 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 07:06:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 07:06:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 07:06:21 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/10 07:06:26 Rebalance progress: 12.5
2021/12/10 07:06:31 Rebalance progress: 12.5
2021/12/10 07:06:36 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 07:06:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 07:06:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 07:06:36 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 07:06:41 clusterSetup.go:397: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get un-deployed
2021/12/10 07:06:46 clusterSetup.go:397: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get un-deployed
2021/12/10 07:06:51 clusterSetup.go:397: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get un-deployed
2021/12/10 07:06:56 clusterSetup.go:397: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get un-deployed
2021/12/10 07:06:56 clusterSetup.go:405: App: TestEventingFailoverOnUpdateBucketOp got un-deployed
2021/12/10 07:06:56 eventingSetup.go:46: No eventing-con process running
2021/12/10 07:06:56 eventingSetup.go:418: Function: TestEventingFailoverOnUpdateBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingFailoverOnUpdateBucketOp deleting in the background"
}
2021/12/10 07:06:58 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 07:07:03 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 07:07:08 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 8585
2021/12/10 07:07:08 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 07:07:10 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 07:07:15 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 07:07:15 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingFailoverOnUpdateBucketOp (159.34s)
=== RUN   TestEventingKVRebalanceOnUpdateBucketOp
2021/12/10 07:07:25 clusterSetup.go:397: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get un-deployed
2021/12/10 07:07:25 clusterSetup.go:405: App: TestEventingKVRebalanceOnUpdateBucketOp got un-deployed
2021/12/10 07:07:25 eventingSetup.go:46: No eventing-con process running
2021/12/10 07:07:25 eventingSetup.go:418: Function: TestEventingKVRebalanceOnUpdateBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingKVRebalanceOnUpdateBucketOp not found"
 }
}
2021/12/10 07:07:27 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 07:07:32 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 07:07:32 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 07:07:33 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 07:07:38 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 07:07:38 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 07:07:43 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 07:07:43 eventingSetup.go:125: Deploying app: TestEventingKVRebalanceOnUpdateBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/10 07:07:44 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingKVRebalanceOnUpdateBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/10 07:07:49 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/10 07:07:54 clusterSetup.go:346: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get deployed
2021/12/10 07:07:59 clusterSetup.go:346: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get deployed
2021/12/10 07:08:04 clusterSetup.go:346: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get deployed
2021/12/10 07:08:04 clusterSetup.go:354: App: TestEventingKVRebalanceOnUpdateBucketOp got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 07:08:04 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 07:08:04 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 07:08:04 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing,kv to the cluster
2021/12/10 07:08:05 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/10 07:08:10 Rebalance progress: 12.5
2021/12/10 07:08:15 Rebalance progress: 12.5
2021/12/10 07:08:20 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 07:08:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 07:08:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 07:08:20 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing,kv to the cluster
2021/12/10 07:08:20 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/10 07:08:25 Rebalance progress: 12.5
2021/12/10 07:08:30 Rebalance progress: 12.5
2021/12/10 07:08:35 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 07:08:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 07:08:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 07:08:35 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/10 07:08:41 Rebalance progress: 12.5
2021/12/10 07:08:46 Rebalance progress: 12.5
2021/12/10 07:08:51 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 07:08:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 07:08:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 07:08:51 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 07:08:56 clusterSetup.go:397: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get un-deployed
2021/12/10 07:09:01 clusterSetup.go:397: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get un-deployed
2021/12/10 07:09:06 clusterSetup.go:397: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get un-deployed
2021/12/10 07:09:11 clusterSetup.go:397: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get un-deployed
2021/12/10 07:09:11 clusterSetup.go:405: App: TestEventingKVRebalanceOnUpdateBucketOp got un-deployed
2021/12/10 07:09:11 eventingSetup.go:46: No eventing-con process running
2021/12/10 07:09:11 eventingSetup.go:418: Function: TestEventingKVRebalanceOnUpdateBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingKVRebalanceOnUpdateBucketOp deleting in the background"
}
2021/12/10 07:09:12 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 07:09:17 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 07:09:22 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6091
2021/12/10 07:09:27 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6091
2021/12/10 07:09:27 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 07:09:29 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 07:09:34 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 07:09:39 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6091
2021/12/10 07:09:39 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingKVRebalanceOnUpdateBucketOp (144.42s)
=== RUN   TestEventingRebInWithOps
2021/12/10 07:09:49 clusterSetup.go:397: Waiting for app: TestEventingRebInWithOps to get un-deployed
2021/12/10 07:09:49 clusterSetup.go:405: App: TestEventingRebInWithOps got un-deployed
2021/12/10 07:09:49 eventingSetup.go:46: No eventing-con process running
2021/12/10 07:09:49 eventingSetup.go:418: Function: TestEventingRebInWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebInWithOps not found"
 }
}
2021/12/10 07:09:51 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 07:09:56 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 07:09:56 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 07:09:58 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 07:10:03 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 07:10:03 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 07:10:08 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 07:10:08 eventingSetup.go:125: Deploying app: TestEventingRebInWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/10 07:10:09 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebInWithOps' in metakv",
  "warnings": null
 }
}
2021/12/10 07:10:14 clusterSetup.go:346: Waiting for app: TestEventingRebInWithOps to get deployed
2021/12/10 07:10:19 clusterSetup.go:346: Waiting for app: TestEventingRebInWithOps to get deployed
2021/12/10 07:10:24 clusterSetup.go:346: Waiting for app: TestEventingRebInWithOps to get deployed
2021/12/10 07:10:29 clusterSetup.go:346: Waiting for app: TestEventingRebInWithOps to get deployed
2021/12/10 07:10:29 clusterSetup.go:354: App: TestEventingRebInWithOps got deployed
2021/12/10 07:10:29 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/10 07:10:29 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/10 07:10:29 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/10 07:10:34 Rebalance progress: 12.5
2021/12/10 07:10:39 Rebalance progress: 12.5
2021/12/10 07:10:44 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 07:10:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 07:10:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 07:10:44 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/10 07:10:50 Rebalance progress: 12.5
2021/12/10 07:10:55 Rebalance progress: 12.5
2021/12/10 07:11:00 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 07:11:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 07:11:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 07:11:00 Waiting for src bucket item count to get to: 200000 curr count: 31091
2021/12/10 07:11:05 Waiting for src bucket item count to get to: 200000 curr count: 47130
2021/12/10 07:11:10 Waiting for src bucket item count to get to: 200000 curr count: 47130
2021/12/10 07:11:15 Waiting for src bucket item count to get to: 200000 curr count: 63438
2021/12/10 07:11:20 Waiting for src bucket item count to get to: 200000 curr count: 63438
2021/12/10 07:11:25 Waiting for src bucket item count to get to: 200000 curr count: 79055
2021/12/10 07:11:30 Waiting for src bucket item count to get to: 200000 curr count: 79055
2021/12/10 07:11:35 Waiting for src bucket item count to get to: 200000 curr count: 95409
2021/12/10 07:11:40 Waiting for src bucket item count to get to: 200000 curr count: 95409
2021/12/10 07:11:45 Waiting for src bucket item count to get to: 200000 curr count: 111793
2021/12/10 07:11:50 Waiting for src bucket item count to get to: 200000 curr count: 128280
2021/12/10 07:11:55 Waiting for src bucket item count to get to: 200000 curr count: 128280
2021/12/10 07:12:00 Waiting for src bucket item count to get to: 200000 curr count: 128280
2021/12/10 07:12:05 Waiting for src bucket item count to get to: 200000 curr count: 144662
2021/12/10 07:12:10 Waiting for src bucket item count to get to: 200000 curr count: 144662
2021/12/10 07:12:15 Waiting for src bucket item count to get to: 200000 curr count: 161052
2021/12/10 07:12:20 Waiting for src bucket item count to get to: 200000 curr count: 161052
2021/12/10 07:12:25 Waiting for src bucket item count to get to: 200000 curr count: 176515
2021/12/10 07:12:30 Waiting for src bucket item count to get to: 200000 curr count: 176515
2021/12/10 07:12:35 Waiting for src bucket item count to get to: 200000 curr count: 192629
2021/12/10 07:12:40 Waiting for src bucket item count to get to: 200000 curr count: 192629
2021/12/10 07:12:45 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/10 07:12:45 Post SET ops. Expected item count: 200000 got 200000
2021/12/10 07:12:45 Starting bucket ops to source bucket
2021/12/10 07:16:30 Waiting for dst bucket item count to get to: 0 curr count: 11241
2021/12/10 07:16:35 Waiting for dst bucket item count to get to: 0 curr count: 11241
2021/12/10 07:16:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 07:16:35 Post DELETE ops. Expected item count: 0 got 0
2021/12/10 07:16:36 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 07:16:41 Waiting for app: TestEventingRebInWithOps to get un-deployed
2021/12/10 07:16:46 Waiting for app: TestEventingRebInWithOps to get un-deployed
2021/12/10 07:16:51 Waiting for app: TestEventingRebInWithOps to get un-deployed
2021/12/10 07:16:56 Waiting for app: TestEventingRebInWithOps to get un-deployed
2021/12/10 07:16:56 App: TestEventingRebInWithOps got un-deployed
2021/12/10 07:16:56 No eventing-con process running
2021/12/10 07:16:56 Function: TestEventingRebInWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebInWithOps deleting in the background"
}
2021/12/10 07:16:57 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 07:17:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 07:17:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 07:17:04 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 07:17:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 07:17:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebInWithOps (449.96s)
=== RUN   TestEventingRebOutWithOps
2021/12/10 07:17:14 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/10 07:17:14 Removing node(s): [] from the cluster
2021/12/10 07:17:19 Rebalance progress: 100
2021/12/10 07:17:24 Waiting for app: TestEventingRebOutWithOps to get un-deployed
2021/12/10 07:17:25 App: TestEventingRebOutWithOps got un-deployed
2021/12/10 07:17:25 No eventing-con process running
2021/12/10 07:17:25 Function: TestEventingRebOutWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebOutWithOps not found"
 }
}
2021/12/10 07:17:26 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 07:17:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 07:17:31 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 07:17:33 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 07:17:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 07:17:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 07:17:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 07:17:43 Deploying app: TestEventingRebOutWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/10 07:17:44 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebOutWithOps' in metakv",
  "warnings": null
 }
}
2021/12/10 07:17:49 Waiting for app: TestEventingRebOutWithOps to get deployed
2021/12/10 07:17:54 Waiting for app: TestEventingRebOutWithOps to get deployed
2021/12/10 07:17:59 Waiting for app: TestEventingRebOutWithOps to get deployed
2021/12/10 07:18:04 Waiting for app: TestEventingRebOutWithOps to get deployed
2021/12/10 07:18:04 App: TestEventingRebOutWithOps got deployed
2021/12/10 07:18:04 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/10 07:18:04 Starting bucket ops to source bucket
2021/12/10 07:18:09 Rebalance progress: 12.5
2021/12/10 07:18:14 Rebalance progress: 12.5
2021/12/10 07:18:19 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 07:18:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 07:18:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 07:18:19 Waiting for src bucket item count to get to: 200000 curr count: 7485
2021/12/10 07:18:24 Waiting for src bucket item count to get to: 200000 curr count: 23615
2021/12/10 07:18:29 Waiting for src bucket item count to get to: 200000 curr count: 23615
2021/12/10 07:18:34 Waiting for src bucket item count to get to: 200000 curr count: 40158
2021/12/10 07:18:39 Waiting for src bucket item count to get to: 200000 curr count: 40158
2021/12/10 07:18:44 Waiting for src bucket item count to get to: 200000 curr count: 56800
2021/12/10 07:18:49 Waiting for src bucket item count to get to: 200000 curr count: 56800
2021/12/10 07:18:54 Waiting for src bucket item count to get to: 200000 curr count: 56800
2021/12/10 07:18:59 Waiting for src bucket item count to get to: 200000 curr count: 73363
2021/12/10 07:19:04 Waiting for src bucket item count to get to: 200000 curr count: 73363
2021/12/10 07:19:09 Waiting for src bucket item count to get to: 200000 curr count: 88473
2021/12/10 07:19:14 Waiting for src bucket item count to get to: 200000 curr count: 104855
2021/12/10 07:19:19 Waiting for src bucket item count to get to: 200000 curr count: 104855
2021/12/10 07:19:24 Waiting for src bucket item count to get to: 200000 curr count: 121572
2021/12/10 07:19:29 Waiting for src bucket item count to get to: 200000 curr count: 121572
2021/12/10 07:19:34 Waiting for src bucket item count to get to: 200000 curr count: 138280
2021/12/10 07:19:39 Waiting for src bucket item count to get to: 200000 curr count: 138280
2021/12/10 07:19:44 Waiting for src bucket item count to get to: 200000 curr count: 154645
2021/12/10 07:19:49 Waiting for src bucket item count to get to: 200000 curr count: 154645
2021/12/10 07:19:54 Waiting for src bucket item count to get to: 200000 curr count: 170705
2021/12/10 07:19:59 Waiting for src bucket item count to get to: 200000 curr count: 170705
2021/12/10 07:20:04 Waiting for src bucket item count to get to: 200000 curr count: 185654
2021/12/10 07:20:09 Waiting for src bucket item count to get to: 200000 curr count: 185654
2021/12/10 07:20:14 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/10 07:20:14 Post SET ops. Expected item count: 200000 got 200000
2021/12/10 07:20:14 Starting bucket ops to source bucket
2021/12/10 07:23:59 Waiting for dst bucket item count to get to: 0 curr count: 8816
2021/12/10 07:24:04 Waiting for dst bucket item count to get to: 0 curr count: 8816
2021/12/10 07:24:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 07:24:04 Post DELETE ops. Expected item count: 0 got 0
2021/12/10 07:24:04 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 07:24:09 Waiting for app: TestEventingRebOutWithOps to get un-deployed
2021/12/10 07:24:15 Waiting for app: TestEventingRebOutWithOps to get un-deployed
2021/12/10 07:24:20 Waiting for app: TestEventingRebOutWithOps to get un-deployed
2021/12/10 07:24:25 Waiting for app: TestEventingRebOutWithOps to get un-deployed
2021/12/10 07:24:25 App: TestEventingRebOutWithOps got un-deployed
2021/12/10 07:24:25 No eventing-con process running
2021/12/10 07:24:25 Function: TestEventingRebOutWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebOutWithOps deleting in the background"
}
2021/12/10 07:24:26 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 07:24:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 07:24:31 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 07:24:33 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 07:24:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 07:24:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebOutWithOps (448.96s)
=== RUN   TestEventingSwapRebWithOps
2021/12/10 07:24:48 Waiting for app: TestEventingSwapRebWithOps to get un-deployed
2021/12/10 07:24:48 App: TestEventingSwapRebWithOps got un-deployed
2021/12/10 07:24:48 No eventing-con process running
2021/12/10 07:24:48 Function: TestEventingSwapRebWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingSwapRebWithOps not found"
 }
}
2021/12/10 07:24:50 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 07:24:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 07:24:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 07:24:57 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 07:25:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 07:25:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 07:25:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 07:25:07 Deploying app: TestEventingSwapRebWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/10 07:25:08 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingSwapRebWithOps' in metakv",
  "warnings": null
 }
}
2021/12/10 07:25:13 Waiting for app: TestEventingSwapRebWithOps to get deployed
2021/12/10 07:25:18 Waiting for app: TestEventingSwapRebWithOps to get deployed
2021/12/10 07:25:23 Waiting for app: TestEventingSwapRebWithOps to get deployed
2021/12/10 07:25:28 Waiting for app: TestEventingSwapRebWithOps to get deployed
2021/12/10 07:25:28 App: TestEventingSwapRebWithOps got deployed
2021/12/10 07:25:28 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/10 07:25:28 Starting bucket ops to source bucket
2021/12/10 07:25:28 Removing node(s): [] from the cluster
2021/12/10 07:25:33 Rebalance progress: 12.5
2021/12/10 07:25:38 Rebalance progress: 12.5
2021/12/10 07:25:43 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 07:25:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 07:25:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 07:25:43 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/10 07:25:43 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/10 07:25:48 Rebalance progress: 12.5
2021/12/10 07:25:53 Rebalance progress: 12.5
2021/12/10 07:25:58 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 07:25:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 07:25:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 07:25:58 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/10 07:26:03 Rebalance progress: 12.5
2021/12/10 07:26:08 Rebalance progress: 12.5
2021/12/10 07:26:13 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 07:26:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 07:26:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 07:26:13 Waiting for src bucket item count to get to: 200000 curr count: 64878
2021/12/10 07:26:18 Waiting for src bucket item count to get to: 200000 curr count: 64878
2021/12/10 07:26:23 Waiting for src bucket item count to get to: 200000 curr count: 79998
2021/12/10 07:26:29 Waiting for src bucket item count to get to: 200000 curr count: 79998
2021/12/10 07:26:34 Waiting for src bucket item count to get to: 200000 curr count: 96384
2021/12/10 07:26:39 Waiting for src bucket item count to get to: 200000 curr count: 96384
2021/12/10 07:26:44 Waiting for src bucket item count to get to: 200000 curr count: 113042
2021/12/10 07:26:49 Waiting for src bucket item count to get to: 200000 curr count: 113042
2021/12/10 07:26:54 Waiting for src bucket item count to get to: 200000 curr count: 113042
2021/12/10 07:26:59 Waiting for src bucket item count to get to: 200000 curr count: 129711
2021/12/10 07:27:04 Waiting for src bucket item count to get to: 200000 curr count: 129711
2021/12/10 07:27:09 Waiting for src bucket item count to get to: 200000 curr count: 146271
2021/12/10 07:27:14 Waiting for src bucket item count to get to: 200000 curr count: 146271
2021/12/10 07:27:19 Waiting for src bucket item count to get to: 200000 curr count: 162793
2021/12/10 07:27:24 Waiting for src bucket item count to get to: 200000 curr count: 162793
2021/12/10 07:27:29 Waiting for src bucket item count to get to: 200000 curr count: 178122
2021/12/10 07:27:34 Waiting for src bucket item count to get to: 200000 curr count: 178122
2021/12/10 07:27:39 Waiting for src bucket item count to get to: 200000 curr count: 194399
2021/12/10 07:27:44 Waiting for src bucket item count to get to: 200000 curr count: 194399
2021/12/10 07:27:49 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/10 07:27:49 Post SET ops. Expected item count: 200000 got 200000
2021/12/10 07:27:49 Starting bucket ops to source bucket
2021/12/10 07:31:34 Waiting for dst bucket item count to get to: 0 curr count: 18022
2021/12/10 07:31:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 07:31:34 Post DELETE ops. Expected item count: 0 got 0
2021/12/10 07:31:34 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 07:31:39 Waiting for app: TestEventingSwapRebWithOps to get un-deployed
2021/12/10 07:31:44 Waiting for app: TestEventingSwapRebWithOps to get un-deployed
2021/12/10 07:31:49 Waiting for app: TestEventingSwapRebWithOps to get un-deployed
2021/12/10 07:31:54 Waiting for app: TestEventingSwapRebWithOps to get un-deployed
2021/12/10 07:31:54 App: TestEventingSwapRebWithOps got un-deployed
2021/12/10 07:31:54 No eventing-con process running
2021/12/10 07:31:54 Function: TestEventingSwapRebWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingSwapRebWithOps deleting in the background"
}
2021/12/10 07:31:56 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 07:32:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 07:32:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 07:32:03 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 07:32:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 07:32:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingSwapRebWithOps (449.55s)
=== RUN   TestKVRebInWithOps
2021/12/10 07:32:18 Waiting for app: TestKVRebInWithOps to get un-deployed
2021/12/10 07:32:18 App: TestKVRebInWithOps got un-deployed
2021/12/10 07:32:18 No eventing-con process running
2021/12/10 07:32:18 Function: TestKVRebInWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVRebInWithOps not found"
 }
}
2021/12/10 07:32:19 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 07:32:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 07:32:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 07:32:26 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 07:32:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 07:32:31 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 07:32:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 07:32:36 Deploying app: TestKVRebInWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/10 07:32:37 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVRebInWithOps' in metakv",
  "warnings": null
 }
}
2021/12/10 07:32:42 Waiting for app: TestKVRebInWithOps to get deployed
2021/12/10 07:32:47 Waiting for app: TestKVRebInWithOps to get deployed
2021/12/10 07:32:52 Waiting for app: TestKVRebInWithOps to get deployed
2021/12/10 07:32:57 Waiting for app: TestKVRebInWithOps to get deployed
2021/12/10 07:32:57 App: TestKVRebInWithOps got deployed
2021/12/10 07:32:57 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/10 07:32:57 Starting bucket ops to source bucket
2021/12/10 07:32:58 Removing node(s): [] from the cluster
2021/12/10 07:33:03 Rebalance progress: 12.5
2021/12/10 07:33:08 Rebalance progress: 12.5
2021/12/10 07:33:13 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 07:33:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 07:33:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 07:33:13 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/10 07:33:18 Rebalance progress: 12.5
2021/12/10 07:33:23 Rebalance progress: 12.5
2021/12/10 07:33:28 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 07:33:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 07:33:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 07:33:28 Waiting for src bucket item count to get to: 200000 curr count: 33824
2021/12/10 07:33:33 Waiting for src bucket item count to get to: 200000 curr count: 50043
2021/12/10 07:33:38 Waiting for src bucket item count to get to: 200000 curr count: 50043
2021/12/10 07:33:43 Waiting for src bucket item count to get to: 200000 curr count: 66428
2021/12/10 07:33:48 Waiting for src bucket item count to get to: 200000 curr count: 66428
2021/12/10 07:33:53 Waiting for src bucket item count to get to: 200000 curr count: 81684
2021/12/10 07:33:58 Waiting for src bucket item count to get to: 200000 curr count: 81684
2021/12/10 07:34:03 Waiting for src bucket item count to get to: 200000 curr count: 97977
2021/12/10 07:34:08 Waiting for src bucket item count to get to: 200000 curr count: 97977
2021/12/10 07:34:13 Waiting for src bucket item count to get to: 200000 curr count: 114593
2021/12/10 07:34:18 Waiting for src bucket item count to get to: 200000 curr count: 114593
2021/12/10 07:34:23 Waiting for src bucket item count to get to: 200000 curr count: 131189
2021/12/10 07:34:28 Waiting for src bucket item count to get to: 200000 curr count: 131189
2021/12/10 07:34:33 Waiting for src bucket item count to get to: 200000 curr count: 147507
2021/12/10 07:34:38 Waiting for src bucket item count to get to: 200000 curr count: 147507
2021/12/10 07:34:43 Waiting for src bucket item count to get to: 200000 curr count: 164204
2021/12/10 07:34:48 Waiting for src bucket item count to get to: 200000 curr count: 164204
2021/12/10 07:34:53 Waiting for src bucket item count to get to: 200000 curr count: 164204
2021/12/10 07:34:58 Waiting for src bucket item count to get to: 200000 curr count: 179620
2021/12/10 07:35:03 Waiting for src bucket item count to get to: 200000 curr count: 179620
2021/12/10 07:35:08 Waiting for src bucket item count to get to: 200000 curr count: 196013
2021/12/10 07:35:13 Waiting for src bucket item count to get to: 200000 curr count: 196013
2021/12/10 07:35:18 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/10 07:35:18 Post SET ops. Expected item count: 200000 got 200000
2021/12/10 07:35:18 Starting bucket ops to source bucket
2021/12/10 07:39:04 Waiting for dst bucket item count to get to: 0 curr count: 17750
2021/12/10 07:39:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 07:39:04 Post DELETE ops. Expected item count: 0 got 0
2021/12/10 07:39:04 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 07:39:09 Waiting for app: TestKVRebInWithOps to get un-deployed
2021/12/10 07:39:14 Waiting for app: TestKVRebInWithOps to get un-deployed
2021/12/10 07:39:19 Waiting for app: TestKVRebInWithOps to get un-deployed
2021/12/10 07:39:24 Waiting for app: TestKVRebInWithOps to get un-deployed
2021/12/10 07:39:24 App: TestKVRebInWithOps got un-deployed
2021/12/10 07:39:24 No eventing-con process running
2021/12/10 07:39:24 Function: TestKVRebInWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVRebInWithOps deleting in the background"
}
2021/12/10 07:39:26 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 07:39:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 07:39:31 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 07:39:33 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 07:39:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 07:39:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebInWithOps (449.92s)
=== RUN   TestKVRebOutWithOps
2021/12/10 07:39:43 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/10 07:39:43 Removing node(s): [] from the cluster
2021/12/10 07:39:48 Rebalance progress: 100
2021/12/10 07:39:53 Waiting for app: TestKVRebOutWithOps to get un-deployed
2021/12/10 07:39:53 App: TestKVRebOutWithOps got un-deployed
2021/12/10 07:39:53 No eventing-con process running
2021/12/10 07:39:53 Function: TestKVRebOutWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVRebOutWithOps not found"
 }
}
2021/12/10 07:39:55 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 07:40:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 07:40:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 07:40:02 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 07:40:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 07:40:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 07:40:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 07:40:12 Deploying app: TestKVRebOutWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/10 07:40:13 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVRebOutWithOps' in metakv",
  "warnings": null
 }
}
2021/12/10 07:40:18 Waiting for app: TestKVRebOutWithOps to get deployed
2021/12/10 07:40:23 Waiting for app: TestKVRebOutWithOps to get deployed
2021/12/10 07:40:28 Waiting for app: TestKVRebOutWithOps to get deployed
2021/12/10 07:40:33 Waiting for app: TestKVRebOutWithOps to get deployed
2021/12/10 07:40:33 App: TestKVRebOutWithOps got deployed
2021/12/10 07:40:33 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/10 07:40:33 Starting bucket ops to source bucket
2021/12/10 07:40:38 Rebalance progress: 12.5
2021/12/10 07:40:43 Rebalance progress: 12.5
2021/12/10 07:40:48 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 07:40:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 07:40:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 07:40:48 Waiting for src bucket item count to get to: 200000 curr count: 9600
2021/12/10 07:40:53 Waiting for src bucket item count to get to: 200000 curr count: 25727
2021/12/10 07:40:58 Waiting for src bucket item count to get to: 200000 curr count: 25727
2021/12/10 07:41:03 Waiting for src bucket item count to get to: 200000 curr count: 42265
2021/12/10 07:41:08 Waiting for src bucket item count to get to: 200000 curr count: 42265
2021/12/10 07:41:13 Waiting for src bucket item count to get to: 200000 curr count: 42265
2021/12/10 07:41:18 Waiting for src bucket item count to get to: 200000 curr count: 58333
2021/12/10 07:41:23 Waiting for src bucket item count to get to: 200000 curr count: 58333
2021/12/10 07:41:28 Waiting for src bucket item count to get to: 200000 curr count: 74155
2021/12/10 07:41:33 Waiting for src bucket item count to get to: 200000 curr count: 74155
2021/12/10 07:41:38 Waiting for src bucket item count to get to: 200000 curr count: 90109
2021/12/10 07:41:43 Waiting for src bucket item count to get to: 200000 curr count: 90109
2021/12/10 07:41:48 Waiting for src bucket item count to get to: 200000 curr count: 106586
2021/12/10 07:41:53 Waiting for src bucket item count to get to: 200000 curr count: 106586
2021/12/10 07:41:58 Waiting for src bucket item count to get to: 200000 curr count: 122957
2021/12/10 07:42:03 Waiting for src bucket item count to get to: 200000 curr count: 122957
2021/12/10 07:42:08 Waiting for src bucket item count to get to: 200000 curr count: 139350
2021/12/10 07:42:13 Waiting for src bucket item count to get to: 200000 curr count: 139350
2021/12/10 07:42:18 Waiting for src bucket item count to get to: 200000 curr count: 155668
2021/12/10 07:42:23 Waiting for src bucket item count to get to: 200000 curr count: 155668
2021/12/10 07:42:28 Waiting for src bucket item count to get to: 200000 curr count: 171876
2021/12/10 07:42:33 Waiting for src bucket item count to get to: 200000 curr count: 171876
2021/12/10 07:42:38 Waiting for src bucket item count to get to: 200000 curr count: 187477
2021/12/10 07:42:43 Waiting for src bucket item count to get to: 200000 curr count: 187477
2021/12/10 07:42:48 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/10 07:42:48 Post SET ops. Expected item count: 200000 got 200000
2021/12/10 07:42:48 Starting bucket ops to source bucket
2021/12/10 07:46:33 Waiting for dst bucket item count to get to: 0 curr count: 15873
2021/12/10 07:46:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 07:46:33 Post DELETE ops. Expected item count: 0 got 0
2021/12/10 07:46:33 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 07:46:38 Waiting for app: TestKVRebOutWithOps to get un-deployed
2021/12/10 07:46:43 Waiting for app: TestKVRebOutWithOps to get un-deployed
2021/12/10 07:46:48 Waiting for app: TestKVRebOutWithOps to get un-deployed
2021/12/10 07:46:53 Waiting for app: TestKVRebOutWithOps to get un-deployed
2021/12/10 07:46:53 App: TestKVRebOutWithOps got un-deployed
2021/12/10 07:46:53 No eventing-con process running
2021/12/10 07:46:53 Function: TestKVRebOutWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVRebOutWithOps deleting in the background"
}
2021/12/10 07:46:55 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 07:47:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 07:47:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 07:47:02 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 07:47:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 07:47:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebOutWithOps (449.35s)
=== RUN   TestKVSwapRebWithOps
2021/12/10 07:47:17 Waiting for app: TestKVSwapRebWithOps to get un-deployed
2021/12/10 07:47:17 App: TestKVSwapRebWithOps got un-deployed
2021/12/10 07:47:17 No eventing-con process running
2021/12/10 07:47:17 Function: TestKVSwapRebWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVSwapRebWithOps not found"
 }
}
2021/12/10 07:47:19 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 07:47:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 07:47:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 07:47:25 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 07:47:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 07:47:30 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 07:47:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 07:47:35 Deploying app: TestKVSwapRebWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/10 07:47:37 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVSwapRebWithOps' in metakv",
  "warnings": null
 }
}
2021/12/10 07:47:42 Waiting for app: TestKVSwapRebWithOps to get deployed
2021/12/10 07:47:47 Waiting for app: TestKVSwapRebWithOps to get deployed
2021/12/10 07:47:52 Waiting for app: TestKVSwapRebWithOps to get deployed
2021/12/10 07:47:57 Waiting for app: TestKVSwapRebWithOps to get deployed
2021/12/10 07:47:57 App: TestKVSwapRebWithOps got deployed
2021/12/10 07:47:57 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/10 07:47:57 Starting bucket ops to source bucket
2021/12/10 07:47:57 Removing node(s): [] from the cluster
2021/12/10 07:48:02 Rebalance progress: 12.5
2021/12/10 07:48:07 Rebalance progress: 12.5
2021/12/10 07:48:12 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 07:48:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 07:48:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 07:48:12 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/10 07:48:12 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/10 07:48:17 Rebalance progress: 12.5
2021/12/10 07:48:22 Rebalance progress: 12.5
2021/12/10 07:48:27 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 07:48:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 07:48:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 07:48:27 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/10 07:48:32 Rebalance progress: 12.5
2021/12/10 07:48:37 Rebalance progress: 12.5
2021/12/10 07:48:42 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 07:48:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 07:48:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 07:48:43 Waiting for src bucket item count to get to: 200000 curr count: 66874
2021/12/10 07:48:48 Waiting for src bucket item count to get to: 200000 curr count: 66874
2021/12/10 07:48:53 Waiting for src bucket item count to get to: 200000 curr count: 81793
2021/12/10 07:48:58 Waiting for src bucket item count to get to: 200000 curr count: 81793
2021/12/10 07:49:03 Waiting for src bucket item count to get to: 200000 curr count: 98386
2021/12/10 07:49:08 Waiting for src bucket item count to get to: 200000 curr count: 98386
2021/12/10 07:49:13 Waiting for src bucket item count to get to: 200000 curr count: 114717
2021/12/10 07:49:18 Waiting for src bucket item count to get to: 200000 curr count: 114717
2021/12/10 07:49:23 Waiting for src bucket item count to get to: 200000 curr count: 130979
2021/12/10 07:49:28 Waiting for src bucket item count to get to: 200000 curr count: 130979
2021/12/10 07:49:33 Waiting for src bucket item count to get to: 200000 curr count: 147530
2021/12/10 07:49:38 Waiting for src bucket item count to get to: 200000 curr count: 147530
2021/12/10 07:49:43 Waiting for src bucket item count to get to: 200000 curr count: 147530
2021/12/10 07:49:48 Waiting for src bucket item count to get to: 200000 curr count: 164133
2021/12/10 07:49:53 Waiting for src bucket item count to get to: 200000 curr count: 164133
2021/12/10 07:49:58 Waiting for src bucket item count to get to: 200000 curr count: 179216
2021/12/10 07:50:03 Waiting for src bucket item count to get to: 200000 curr count: 179216
2021/12/10 07:50:08 Waiting for src bucket item count to get to: 200000 curr count: 195799
2021/12/10 07:50:13 Waiting for src bucket item count to get to: 200000 curr count: 195799
2021/12/10 07:50:18 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/10 07:50:18 Post SET ops. Expected item count: 200000 got 200000
2021/12/10 07:50:18 Starting bucket ops to source bucket
2021/12/10 07:54:03 Waiting for dst bucket item count to get to: 0 curr count: 16317
2021/12/10 07:54:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 07:54:03 Post DELETE ops. Expected item count: 0 got 0
2021/12/10 07:54:03 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 07:54:08 Waiting for app: TestKVSwapRebWithOps to get un-deployed
2021/12/10 07:54:13 Waiting for app: TestKVSwapRebWithOps to get un-deployed
2021/12/10 07:54:18 Waiting for app: TestKVSwapRebWithOps to get un-deployed
2021/12/10 07:54:23 Waiting for app: TestKVSwapRebWithOps to get un-deployed
2021/12/10 07:54:23 App: TestKVSwapRebWithOps got un-deployed
2021/12/10 07:54:23 No eventing-con process running
2021/12/10 07:54:23 Function: TestKVSwapRebWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVSwapRebWithOps deleting in the background"
}
2021/12/10 07:54:25 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 07:54:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 07:54:30 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 07:54:32 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 07:54:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 07:54:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVSwapRebWithOps (449.91s)
=== RUN   TestEventingRebInWithTimers
2021/12/10 07:54:47 Waiting for app: TestEventingRebInWithTimers to get un-deployed
2021/12/10 07:54:47 App: TestEventingRebInWithTimers got un-deployed
2021/12/10 07:54:47 No eventing-con process running
2021/12/10 07:54:47 Function: TestEventingRebInWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebInWithTimers not found"
 }
}
2021/12/10 07:54:49 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 07:54:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 07:54:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 07:54:55 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 07:55:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 07:55:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 07:55:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 07:55:05 Deploying app: TestEventingRebInWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/10 07:55:06 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebInWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/10 07:55:11 Waiting for app: TestEventingRebInWithTimers to get deployed
2021/12/10 07:55:16 Waiting for app: TestEventingRebInWithTimers to get deployed
2021/12/10 07:55:21 Waiting for app: TestEventingRebInWithTimers to get deployed
2021/12/10 07:55:27 Waiting for app: TestEventingRebInWithTimers to get deployed
2021/12/10 07:55:27 App: TestEventingRebInWithTimers got deployed
2021/12/10 07:55:27 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/10 07:55:27 Starting bucket ops to source bucket
2021/12/10 07:55:27 Removing node(s): [] from the cluster
2021/12/10 07:55:32 Rebalance progress: 12.5
2021/12/10 07:55:37 Rebalance progress: 12.5
2021/12/10 07:55:42 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 07:55:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 07:55:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 07:55:42 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/10 07:55:47 Rebalance progress: 12.5
2021/12/10 07:55:52 Rebalance progress: 12.5
2021/12/10 07:55:57 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 07:55:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 07:55:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 07:55:57 Waiting for src bucket item count to get to: 50000 curr count: 28424
2021/12/10 07:56:02 Waiting for src bucket item count to get to: 50000 curr count: 39564
2021/12/10 07:56:07 Waiting for src bucket item count to get to: 50000 curr count: 39564
2021/12/10 07:56:17 Waiting for dst bucket item count to get to: 50000 curr count: 36638
2021/12/10 07:56:22 Waiting for dst bucket item count to get to: 50000 curr count: 36638
2021/12/10 07:56:27 Waiting for dst bucket item count to get to: 50000 curr count: 46642
2021/12/10 07:56:32 Waiting for dst bucket item count to get to: 50000 curr count: 46642
2021/12/10 07:56:32 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/10 07:56:32 Post SET ops. Expected item count: 50000 got 50000
2021/12/10 07:56:32 Starting bucket ops to source bucket
2021/12/10 07:57:42 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/10 07:57:47 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/10 07:57:52 Waiting for dst bucket item count to get to: 0 curr count: 48220
2021/12/10 07:57:57 Waiting for dst bucket item count to get to: 0 curr count: 48220
2021/12/10 07:58:02 Waiting for dst bucket item count to get to: 0 curr count: 39876
2021/12/10 07:58:07 Waiting for dst bucket item count to get to: 0 curr count: 39876
2021/12/10 07:58:12 Waiting for dst bucket item count to get to: 0 curr count: 30471
2021/12/10 07:58:17 Waiting for dst bucket item count to get to: 0 curr count: 30471
2021/12/10 07:58:22 Waiting for dst bucket item count to get to: 0 curr count: 5015
2021/12/10 07:58:27 Waiting for dst bucket item count to get to: 0 curr count: 5015
2021/12/10 07:58:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 07:58:27 Post DELETE ops. Expected item count: 0 got 0
2021/12/10 07:58:27 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 07:58:32 Waiting for app: TestEventingRebInWithTimers to get un-deployed
2021/12/10 07:58:37 Waiting for app: TestEventingRebInWithTimers to get un-deployed
2021/12/10 07:58:42 Waiting for app: TestEventingRebInWithTimers to get un-deployed
2021/12/10 07:58:47 Waiting for app: TestEventingRebInWithTimers to get un-deployed
2021/12/10 07:58:47 App: TestEventingRebInWithTimers got un-deployed
2021/12/10 07:58:47 No eventing-con process running
2021/12/10 07:58:47 Function: TestEventingRebInWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebInWithTimers deleting in the background"
}
2021/12/10 07:58:49 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 07:58:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 07:58:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 07:58:55 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 07:59:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 07:59:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebInWithTimers (263.58s)
=== RUN   TestEventingRebOutWithTimers
2021/12/10 07:59:05 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/10 07:59:06 Removing node(s): [] from the cluster
2021/12/10 07:59:11 Rebalance progress: 100
2021/12/10 07:59:16 Waiting for app: TestEventingRebOutWithTimers to get un-deployed
2021/12/10 07:59:16 App: TestEventingRebOutWithTimers got un-deployed
2021/12/10 07:59:16 No eventing-con process running
2021/12/10 07:59:16 Function: TestEventingRebOutWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebOutWithTimers not found"
 }
}
2021/12/10 07:59:17 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 07:59:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 07:59:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 07:59:24 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 07:59:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 07:59:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 07:59:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 07:59:34 Deploying app: TestEventingRebOutWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/10 07:59:35 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebOutWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/10 07:59:40 Waiting for app: TestEventingRebOutWithTimers to get deployed
2021/12/10 07:59:45 Waiting for app: TestEventingRebOutWithTimers to get deployed
2021/12/10 07:59:50 Waiting for app: TestEventingRebOutWithTimers to get deployed
2021/12/10 07:59:55 Waiting for app: TestEventingRebOutWithTimers to get deployed
2021/12/10 07:59:55 App: TestEventingRebOutWithTimers got deployed
2021/12/10 07:59:55 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/10 07:59:55 Starting bucket ops to source bucket
2021/12/10 08:00:00 Rebalance progress: 12.5
2021/12/10 08:00:05 Rebalance progress: 12.5
2021/12/10 08:00:10 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:00:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:00:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 08:00:11 Waiting for src bucket item count to get to: 50000 curr count: 19393
2021/12/10 08:00:16 Waiting for src bucket item count to get to: 50000 curr count: 19393
2021/12/10 08:00:21 Waiting for src bucket item count to get to: 50000 curr count: 31395
2021/12/10 08:00:26 Waiting for src bucket item count to get to: 50000 curr count: 31395
2021/12/10 08:00:31 Waiting for src bucket item count to get to: 50000 curr count: 43144
2021/12/10 08:00:36 Waiting for src bucket item count to get to: 50000 curr count: 43144
2021/12/10 08:00:41 Waiting for src bucket item count to get to: 50000 curr count: 43144
2021/12/10 08:00:51 Waiting for dst bucket item count to get to: 50000 curr count: 36088
2021/12/10 08:00:56 Waiting for dst bucket item count to get to: 50000 curr count: 36088
2021/12/10 08:01:01 Waiting for dst bucket item count to get to: 50000 curr count: 43579
2021/12/10 08:01:06 Waiting for dst bucket item count to get to: 50000 curr count: 43579
2021/12/10 08:01:06 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/10 08:01:06 Post SET ops. Expected item count: 50000 got 50000
2021/12/10 08:01:06 Starting bucket ops to source bucket
2021/12/10 08:02:15 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/10 08:02:20 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/10 08:02:25 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/10 08:02:30 Waiting for dst bucket item count to get to: 0 curr count: 47038
2021/12/10 08:02:35 Waiting for dst bucket item count to get to: 0 curr count: 47038
2021/12/10 08:02:40 Waiting for dst bucket item count to get to: 0 curr count: 34366
2021/12/10 08:02:45 Waiting for dst bucket item count to get to: 0 curr count: 34366
2021/12/10 08:02:50 Waiting for dst bucket item count to get to: 0 curr count: 15855
2021/12/10 08:02:55 Waiting for dst bucket item count to get to: 0 curr count: 15855
2021/12/10 08:03:00 Waiting for dst bucket item count to get to: 0 curr count: 3562
2021/12/10 08:03:05 Waiting for dst bucket item count to get to: 0 curr count: 3562
2021/12/10 08:03:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 08:03:05 Post DELETE ops. Expected item count: 0 got 0
2021/12/10 08:03:05 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 08:03:10 Waiting for app: TestEventingRebOutWithTimers to get un-deployed
2021/12/10 08:03:15 Waiting for app: TestEventingRebOutWithTimers to get un-deployed
2021/12/10 08:03:20 Waiting for app: TestEventingRebOutWithTimers to get un-deployed
2021/12/10 08:03:25 Waiting for app: TestEventingRebOutWithTimers to get un-deployed
2021/12/10 08:03:25 App: TestEventingRebOutWithTimers got un-deployed
2021/12/10 08:03:25 No eventing-con process running
2021/12/10 08:03:25 Function: TestEventingRebOutWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebOutWithTimers deleting in the background"
}
2021/12/10 08:03:27 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:03:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 08:03:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 08:03:33 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:03:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 08:03:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebOutWithTimers (277.90s)
=== RUN   TestEventingSwapRebWithTimers
2021/12/10 08:03:48 Waiting for app: TestEventingSwapRebWithTimers to get un-deployed
2021/12/10 08:03:48 App: TestEventingSwapRebWithTimers got un-deployed
2021/12/10 08:03:48 No eventing-con process running
2021/12/10 08:03:48 Function: TestEventingSwapRebWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingSwapRebWithTimers not found"
 }
}
2021/12/10 08:03:50 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:03:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 08:03:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 08:03:57 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:04:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 08:04:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:04:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 08:04:07 Deploying app: TestEventingSwapRebWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/10 08:04:08 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingSwapRebWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/10 08:04:13 Waiting for app: TestEventingSwapRebWithTimers to get deployed
2021/12/10 08:04:18 Waiting for app: TestEventingSwapRebWithTimers to get deployed
2021/12/10 08:04:23 Waiting for app: TestEventingSwapRebWithTimers to get deployed
2021/12/10 08:04:28 Waiting for app: TestEventingSwapRebWithTimers to get deployed
2021/12/10 08:04:28 App: TestEventingSwapRebWithTimers got deployed
2021/12/10 08:04:28 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/10 08:04:28 Starting bucket ops to source bucket
2021/12/10 08:04:28 Removing node(s): [] from the cluster
2021/12/10 08:04:33 Rebalance progress: 12.5
2021/12/10 08:04:38 Rebalance progress: 12.5
2021/12/10 08:04:43 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:04:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:04:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 08:04:43 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/10 08:04:44 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/10 08:04:49 Rebalance progress: 12.5
2021/12/10 08:04:54 Rebalance progress: 12.5
2021/12/10 08:04:59 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:04:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:04:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 08:04:59 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/10 08:05:04 Rebalance progress: 12.5
2021/12/10 08:05:09 Rebalance progress: 12.5
2021/12/10 08:05:14 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:05:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:05:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 08:05:14 Waiting for src bucket item count to get to: 50000 curr count: 49634
2021/12/10 08:05:19 Waiting for src bucket item count to get to: 50000 curr count: 49634
2021/12/10 08:05:29 Waiting for dst bucket item count to get to: 50000 curr count: 42134
2021/12/10 08:05:34 Waiting for dst bucket item count to get to: 50000 curr count: 42134
2021/12/10 08:05:34 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/10 08:05:34 Post SET ops. Expected item count: 50000 got 50000
2021/12/10 08:05:34 Starting bucket ops to source bucket
2021/12/10 08:06:43 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/10 08:06:48 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/10 08:06:53 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/10 08:06:58 Waiting for dst bucket item count to get to: 0 curr count: 41552
2021/12/10 08:07:03 Waiting for dst bucket item count to get to: 0 curr count: 41552
2021/12/10 08:07:08 Waiting for dst bucket item count to get to: 0 curr count: 30488
2021/12/10 08:07:13 Waiting for dst bucket item count to get to: 0 curr count: 30488
2021/12/10 08:07:18 Waiting for dst bucket item count to get to: 0 curr count: 15801
2021/12/10 08:07:23 Waiting for dst bucket item count to get to: 0 curr count: 15801
2021/12/10 08:07:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 08:07:23 Post DELETE ops. Expected item count: 0 got 0
2021/12/10 08:07:23 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 08:07:28 Waiting for app: TestEventingSwapRebWithTimers to get un-deployed
2021/12/10 08:07:33 Waiting for app: TestEventingSwapRebWithTimers to get un-deployed
2021/12/10 08:07:38 Waiting for app: TestEventingSwapRebWithTimers to get un-deployed
2021/12/10 08:07:43 Waiting for app: TestEventingSwapRebWithTimers to get un-deployed
2021/12/10 08:07:43 App: TestEventingSwapRebWithTimers got un-deployed
2021/12/10 08:07:43 No eventing-con process running
2021/12/10 08:07:43 Function: TestEventingSwapRebWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingSwapRebWithTimers deleting in the background"
}
2021/12/10 08:07:45 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:07:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 08:07:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 08:07:52 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:07:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 08:07:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingSwapRebWithTimers (258.42s)
=== RUN   TestKVRebInWithTimers
2021/12/10 08:08:07 Waiting for app: TestKVRebInWithTimers to get un-deployed
2021/12/10 08:08:07 App: TestKVRebInWithTimers got un-deployed
2021/12/10 08:08:07 No eventing-con process running
2021/12/10 08:08:07 Function: TestKVRebInWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVRebInWithTimers not found"
 }
}
2021/12/10 08:08:08 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:08:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 08:08:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 08:08:15 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:08:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 08:08:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:08:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 08:08:25 Deploying app: TestKVRebInWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/10 08:08:26 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVRebInWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/10 08:08:31 Waiting for app: TestKVRebInWithTimers to get deployed
2021/12/10 08:08:36 Waiting for app: TestKVRebInWithTimers to get deployed
2021/12/10 08:08:41 Waiting for app: TestKVRebInWithTimers to get deployed
2021/12/10 08:08:46 Waiting for app: TestKVRebInWithTimers to get deployed
2021/12/10 08:08:46 App: TestKVRebInWithTimers got deployed
2021/12/10 08:08:46 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/10 08:08:46 Starting bucket ops to source bucket
2021/12/10 08:08:47 Removing node(s): [] from the cluster
2021/12/10 08:08:52 Rebalance progress: 12.5
2021/12/10 08:08:57 Rebalance progress: 12.5
2021/12/10 08:09:02 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:09:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:09:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 08:09:02 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/10 08:09:07 Rebalance progress: 12.5
2021/12/10 08:09:12 Rebalance progress: 12.5
2021/12/10 08:09:17 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:09:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:09:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 08:09:17 Waiting for src bucket item count to get to: 50000 curr count: 29318
2021/12/10 08:09:22 Waiting for src bucket item count to get to: 50000 curr count: 40253
2021/12/10 08:09:27 Waiting for src bucket item count to get to: 50000 curr count: 40253
2021/12/10 08:09:37 Waiting for dst bucket item count to get to: 50000 curr count: 33984
2021/12/10 08:09:42 Waiting for dst bucket item count to get to: 50000 curr count: 33984
2021/12/10 08:09:47 Waiting for dst bucket item count to get to: 50000 curr count: 44820
2021/12/10 08:09:52 Waiting for dst bucket item count to get to: 50000 curr count: 44820
2021/12/10 08:09:57 Waiting for dst bucket item count to get to: 50000 curr count: 44820
2021/12/10 08:09:57 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/10 08:09:57 Post SET ops. Expected item count: 50000 got 50000
2021/12/10 08:09:57 Starting bucket ops to source bucket
2021/12/10 08:11:06 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/10 08:11:11 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/10 08:11:16 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/10 08:11:21 Waiting for dst bucket item count to get to: 0 curr count: 47559
2021/12/10 08:11:26 Waiting for dst bucket item count to get to: 0 curr count: 47559
2021/12/10 08:11:31 Waiting for dst bucket item count to get to: 0 curr count: 37526
2021/12/10 08:11:36 Waiting for dst bucket item count to get to: 0 curr count: 37526
2021/12/10 08:11:41 Waiting for dst bucket item count to get to: 0 curr count: 16584
2021/12/10 08:11:46 Waiting for dst bucket item count to get to: 0 curr count: 16584
2021/12/10 08:11:51 Waiting for dst bucket item count to get to: 0 curr count: 2968
2021/12/10 08:11:56 Waiting for dst bucket item count to get to: 0 curr count: 2968
2021/12/10 08:11:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 08:11:56 Post DELETE ops. Expected item count: 0 got 0
2021/12/10 08:11:56 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 08:12:01 Waiting for app: TestKVRebInWithTimers to get un-deployed
2021/12/10 08:12:06 Waiting for app: TestKVRebInWithTimers to get un-deployed
2021/12/10 08:12:11 Waiting for app: TestKVRebInWithTimers to get un-deployed
2021/12/10 08:12:16 Waiting for app: TestKVRebInWithTimers to get un-deployed
2021/12/10 08:12:16 App: TestKVRebInWithTimers got un-deployed
2021/12/10 08:12:16 No eventing-con process running
2021/12/10 08:12:16 Function: TestKVRebInWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVRebInWithTimers deleting in the background"
}
2021/12/10 08:12:18 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:12:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 08:12:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 08:12:25 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:12:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 08:12:30 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebInWithTimers (273.22s)
=== RUN   TestKVRebOutWithTimers
2021/12/10 08:12:35 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/10 08:12:35 Removing node(s): [] from the cluster
2021/12/10 08:12:40 Rebalance progress: 100
2021/12/10 08:12:45 Waiting for app: TestKVRebOutWithTimers to get un-deployed
2021/12/10 08:12:45 App: TestKVRebOutWithTimers got un-deployed
2021/12/10 08:12:45 No eventing-con process running
2021/12/10 08:12:45 Function: TestKVRebOutWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVRebOutWithTimers not found"
 }
}
2021/12/10 08:12:47 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:12:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 08:12:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 08:12:54 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:12:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 08:12:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:13:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 08:13:04 Deploying app: TestKVRebOutWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/10 08:13:05 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVRebOutWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/10 08:13:10 Waiting for app: TestKVRebOutWithTimers to get deployed
2021/12/10 08:13:15 Waiting for app: TestKVRebOutWithTimers to get deployed
2021/12/10 08:13:20 Waiting for app: TestKVRebOutWithTimers to get deployed
2021/12/10 08:13:25 Waiting for app: TestKVRebOutWithTimers to get deployed
2021/12/10 08:13:25 App: TestKVRebOutWithTimers got deployed
2021/12/10 08:13:25 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/10 08:13:25 Starting bucket ops to source bucket
2021/12/10 08:13:30 Rebalance progress: 12.5
2021/12/10 08:13:35 Rebalance progress: 12.5
2021/12/10 08:13:40 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:13:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:13:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 08:13:40 Waiting for src bucket item count to get to: 50000 curr count: 19434
2021/12/10 08:13:45 Waiting for src bucket item count to get to: 50000 curr count: 19434
2021/12/10 08:13:50 Waiting for src bucket item count to get to: 50000 curr count: 30449
2021/12/10 08:13:55 Waiting for src bucket item count to get to: 50000 curr count: 30449
2021/12/10 08:14:00 Waiting for src bucket item count to get to: 50000 curr count: 42607
2021/12/10 08:14:05 Waiting for src bucket item count to get to: 50000 curr count: 42607
2021/12/10 08:14:10 Waiting for src bucket item count to get to: 50000 curr count: 42607
2021/12/10 08:14:20 Waiting for dst bucket item count to get to: 50000 curr count: 38488
2021/12/10 08:14:25 Waiting for dst bucket item count to get to: 50000 curr count: 38488
2021/12/10 08:14:30 Waiting for dst bucket item count to get to: 50000 curr count: 46621
2021/12/10 08:14:35 Waiting for dst bucket item count to get to: 50000 curr count: 46621
2021/12/10 08:14:35 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/10 08:14:35 Post SET ops. Expected item count: 50000 got 50000
2021/12/10 08:14:35 Starting bucket ops to source bucket
2021/12/10 08:15:45 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/10 08:15:50 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/10 08:15:55 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/10 08:16:00 Waiting for dst bucket item count to get to: 0 curr count: 47138
2021/12/10 08:16:05 Waiting for dst bucket item count to get to: 0 curr count: 47138
2021/12/10 08:16:10 Waiting for dst bucket item count to get to: 0 curr count: 33114
2021/12/10 08:16:15 Waiting for dst bucket item count to get to: 0 curr count: 33114
2021/12/10 08:16:20 Waiting for dst bucket item count to get to: 0 curr count: 12609
2021/12/10 08:16:25 Waiting for dst bucket item count to get to: 0 curr count: 12609
2021/12/10 08:16:30 Waiting for dst bucket item count to get to: 0 curr count: 435
2021/12/10 08:16:35 Waiting for dst bucket item count to get to: 0 curr count: 435
2021/12/10 08:16:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 08:16:35 Post DELETE ops. Expected item count: 0 got 0
2021/12/10 08:16:35 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 08:16:40 Waiting for app: TestKVRebOutWithTimers to get un-deployed
2021/12/10 08:16:45 Waiting for app: TestKVRebOutWithTimers to get un-deployed
2021/12/10 08:16:50 Waiting for app: TestKVRebOutWithTimers to get un-deployed
2021/12/10 08:16:55 Waiting for app: TestKVRebOutWithTimers to get un-deployed
2021/12/10 08:16:55 App: TestKVRebOutWithTimers got un-deployed
2021/12/10 08:16:55 No eventing-con process running
2021/12/10 08:16:55 Function: TestKVRebOutWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVRebOutWithTimers deleting in the background"
}
2021/12/10 08:16:57 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:17:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 08:17:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 08:17:04 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:17:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 08:17:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebOutWithTimers (278.76s)
=== RUN   TestKVSwapRebWithTimers
2021/12/10 08:17:19 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/10 08:17:19 App: TestKVSwapRebWithTimers got un-deployed
2021/12/10 08:17:19 No eventing-con process running
2021/12/10 08:17:19 Function: TestKVSwapRebWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVSwapRebWithTimers not found"
 }
}
2021/12/10 08:17:20 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:17:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 08:17:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 08:17:27 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:17:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 08:17:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:17:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 08:17:37 Deploying app: TestKVSwapRebWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/10 08:17:38 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVSwapRebWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/10 08:17:43 Waiting for app: TestKVSwapRebWithTimers to get deployed
2021/12/10 08:17:48 Waiting for app: TestKVSwapRebWithTimers to get deployed
2021/12/10 08:17:53 Waiting for app: TestKVSwapRebWithTimers to get deployed
2021/12/10 08:17:58 Waiting for app: TestKVSwapRebWithTimers to get deployed
2021/12/10 08:17:58 App: TestKVSwapRebWithTimers got deployed
2021/12/10 08:17:58 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/10 08:17:58 Starting bucket ops to source bucket
2021/12/10 08:17:59 Removing node(s): [] from the cluster
2021/12/10 08:18:04 Rebalance progress: 12.5
2021/12/10 08:18:09 Rebalance progress: 12.5
2021/12/10 08:18:14 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:18:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:18:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 08:18:14 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/10 08:18:14 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/10 08:18:20 Rebalance progress: 12.5
2021/12/10 08:18:25 Rebalance progress: 12.5
2021/12/10 08:18:30 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:18:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:18:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 08:18:30 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/10 08:18:35 Rebalance progress: 12.5
2021/12/10 08:18:40 Rebalance progress: 12.5
2021/12/10 08:18:45 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:18:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:18:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 08:18:45 Waiting for src bucket item count to get to: 50000 curr count: 49004
2021/12/10 08:18:50 Waiting for src bucket item count to get to: 50000 curr count: 49004
2021/12/10 08:19:00 Waiting for dst bucket item count to get to: 50000 curr count: 44674
2021/12/10 08:19:05 Waiting for dst bucket item count to get to: 50000 curr count: 44674
2021/12/10 08:19:05 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/10 08:19:05 Post SET ops. Expected item count: 50000 got 50000
2021/12/10 08:19:05 Starting bucket ops to source bucket
2021/12/10 08:20:14 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/10 08:20:19 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/10 08:20:24 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/10 08:20:29 Waiting for dst bucket item count to get to: 0 curr count: 42994
2021/12/10 08:20:34 Waiting for dst bucket item count to get to: 0 curr count: 42994
2021/12/10 08:20:39 Waiting for dst bucket item count to get to: 0 curr count: 31252
2021/12/10 08:20:44 Waiting for dst bucket item count to get to: 0 curr count: 31252
2021/12/10 08:20:49 Waiting for dst bucket item count to get to: 0 curr count: 15237
2021/12/10 08:20:54 Waiting for dst bucket item count to get to: 0 curr count: 15237
2021/12/10 08:20:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 08:20:54 Post DELETE ops. Expected item count: 0 got 0
2021/12/10 08:20:54 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 08:20:59 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/10 08:21:04 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/10 08:21:09 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/10 08:21:14 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/10 08:21:19 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/10 08:21:24 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/10 08:21:24 App: TestKVSwapRebWithTimers got un-deployed
2021/12/10 08:21:24 No eventing-con process running
2021/12/10 08:21:24 Function: TestKVSwapRebWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVSwapRebWithTimers deleting in the background"
}
2021/12/10 08:21:26 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:21:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 08:21:31 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 08:21:33 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:21:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 08:21:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVSwapRebWithTimers (269.15s)
=== RUN   TestDuplicateTimerCreationEventingRebIn
2021/12/10 08:21:48 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/10 08:21:48 App: TestDuplicateTimerCreationEventingRebIn got un-deployed
2021/12/10 08:21:48 No eventing-con process running
2021/12/10 08:21:48 Function: TestDuplicateTimerCreationEventingRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationEventingRebIn not found"
 }
}
2021/12/10 08:21:50 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:21:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 08:21:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 08:21:56 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:22:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 08:22:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:22:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 08:22:06 Deploying app: TestDuplicateTimerCreationEventingRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/10 08:22:08 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationEventingRebIn' in metakv",
  "warnings": null
 }
}
2021/12/10 08:22:13 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get deployed
2021/12/10 08:22:18 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get deployed
2021/12/10 08:22:23 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get deployed
2021/12/10 08:22:28 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get deployed
2021/12/10 08:22:28 App: TestDuplicateTimerCreationEventingRebIn got deployed
2021/12/10 08:22:28 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/10 08:22:28 Starting bucket ops to source bucket
2021/12/10 08:22:28 Removing node(s): [] from the cluster
2021/12/10 08:22:33 Rebalance progress: 12.5
2021/12/10 08:22:38 Rebalance progress: 12.5
2021/12/10 08:22:43 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:22:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:22:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 08:22:43 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/10 08:22:48 Rebalance progress: 12.5
2021/12/10 08:22:53 Rebalance progress: 12.5
2021/12/10 08:22:58 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:22:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:22:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 08:22:58 Waiting for src bucket item count to get to: 50000 curr count: 27504
2021/12/10 08:23:03 Waiting for src bucket item count to get to: 50000 curr count: 37274
2021/12/10 08:23:08 Waiting for src bucket item count to get to: 50000 curr count: 37274
2021/12/10 08:23:13 Waiting for src bucket item count to get to: 50000 curr count: 48238
2021/12/10 08:23:18 Waiting for src bucket item count to get to: 50000 curr count: 48238
2021/12/10 08:23:28 Waiting for dst bucket item count to get to: 50000 curr count: 40286
2021/12/10 08:23:33 Waiting for dst bucket item count to get to: 50000 curr count: 40286
2021/12/10 08:23:38 Waiting for dst bucket item count to get to: 50000 curr count: 47505
2021/12/10 08:23:43 Waiting for dst bucket item count to get to: 50000 curr count: 47505
2021/12/10 08:23:43 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/10 08:23:43 Post SET ops. Expected item count: 50000 got 50000
2021/12/10 08:23:43 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 08:23:48 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/10 08:23:53 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/10 08:23:58 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/10 08:24:03 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/10 08:24:08 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/10 08:24:13 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/10 08:24:13 App: TestDuplicateTimerCreationEventingRebIn got un-deployed
2021/12/10 08:24:13 No eventing-con process running
2021/12/10 08:24:14 Function: TestDuplicateTimerCreationEventingRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationEventingRebIn deleting in the background"
}
2021/12/10 08:24:15 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:24:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 08:24:25 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/10 08:24:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 08:24:27 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:24:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 08:24:37 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/10 08:24:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationEventingRebIn (179.18s)
=== RUN   TestDuplicateTimerCreationEventingRebOut
2021/12/10 08:24:42 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/10 08:24:42 Removing node(s): [] from the cluster
2021/12/10 08:24:48 Rebalance progress: 100
2021/12/10 08:24:53 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/10 08:24:53 App: TestDuplicateTimerCreationEventingRebOut got un-deployed
2021/12/10 08:24:53 No eventing-con process running
2021/12/10 08:24:53 Function: TestDuplicateTimerCreationEventingRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationEventingRebOut not found"
 }
}
2021/12/10 08:24:54 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:24:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 08:24:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 08:25:01 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:25:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 08:25:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:25:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 08:25:11 Deploying app: TestDuplicateTimerCreationEventingRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/10 08:25:12 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationEventingRebOut' in metakv",
  "warnings": null
 }
}
2021/12/10 08:25:17 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get deployed
2021/12/10 08:25:22 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get deployed
2021/12/10 08:25:27 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get deployed
2021/12/10 08:25:32 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get deployed
2021/12/10 08:25:32 App: TestDuplicateTimerCreationEventingRebOut got deployed
2021/12/10 08:25:32 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/10 08:25:32 Starting bucket ops to source bucket
2021/12/10 08:25:37 Rebalance progress: 12.5
2021/12/10 08:25:42 Rebalance progress: 12.5
2021/12/10 08:25:47 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:25:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:25:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 08:25:47 Waiting for src bucket item count to get to: 50000 curr count: 8597
2021/12/10 08:25:52 Waiting for src bucket item count to get to: 50000 curr count: 22725
2021/12/10 08:25:57 Waiting for src bucket item count to get to: 50000 curr count: 22725
2021/12/10 08:26:02 Waiting for src bucket item count to get to: 50000 curr count: 36093
2021/12/10 08:26:07 Waiting for src bucket item count to get to: 50000 curr count: 36093
2021/12/10 08:26:12 Waiting for src bucket item count to get to: 50000 curr count: 45201
2021/12/10 08:26:17 Waiting for src bucket item count to get to: 50000 curr count: 45201
2021/12/10 08:26:22 Waiting for src bucket item count to get to: 50000 curr count: 45201
2021/12/10 08:26:32 Waiting for dst bucket item count to get to: 50000 curr count: 32046
2021/12/10 08:26:37 Waiting for dst bucket item count to get to: 50000 curr count: 32046
2021/12/10 08:26:42 Waiting for dst bucket item count to get to: 50000 curr count: 45975
2021/12/10 08:26:47 Waiting for dst bucket item count to get to: 50000 curr count: 45975
2021/12/10 08:26:47 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/10 08:26:47 Post SET ops. Expected item count: 50000 got 50000
2021/12/10 08:26:48 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 08:26:53 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/10 08:26:58 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/10 08:27:03 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/10 08:27:08 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/10 08:27:13 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/10 08:27:18 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/10 08:27:18 App: TestDuplicateTimerCreationEventingRebOut got un-deployed
2021/12/10 08:27:18 No eventing-con process running
2021/12/10 08:27:18 Function: TestDuplicateTimerCreationEventingRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationEventingRebOut deleting in the background"
}
2021/12/10 08:27:19 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:27:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 08:27:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 08:27:26 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:27:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 08:27:36 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/10 08:27:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationEventingRebOut (179.20s)
=== RUN   TestDuplicateTimerCreationEventingSwapReb
2021/12/10 08:27:46 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/10 08:27:46 App: TestDuplicateTimerCreationEventingSwapReb got un-deployed
2021/12/10 08:27:46 No eventing-con process running
2021/12/10 08:27:46 Function: TestDuplicateTimerCreationEventingSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationEventingSwapReb not found"
 }
}
2021/12/10 08:27:48 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:27:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 08:27:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 08:27:55 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:28:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 08:28:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:28:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 08:28:05 Deploying app: TestDuplicateTimerCreationEventingSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/10 08:28:06 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationEventingSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/10 08:28:11 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get deployed
2021/12/10 08:28:16 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get deployed
2021/12/10 08:28:21 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get deployed
2021/12/10 08:28:26 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get deployed
2021/12/10 08:28:26 App: TestDuplicateTimerCreationEventingSwapReb got deployed
2021/12/10 08:28:26 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/10 08:28:26 Starting bucket ops to source bucket
2021/12/10 08:28:27 Removing node(s): [] from the cluster
2021/12/10 08:28:32 Rebalance progress: 12.5
2021/12/10 08:28:37 Rebalance progress: 12.5
2021/12/10 08:28:42 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:28:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:28:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 08:28:42 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/10 08:28:42 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/10 08:28:47 Rebalance progress: 12.5
2021/12/10 08:28:52 Rebalance progress: 12.5
2021/12/10 08:28:57 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:28:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:28:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 08:28:57 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/10 08:29:02 Rebalance progress: 12.5
2021/12/10 08:29:07 Rebalance progress: 12.5
2021/12/10 08:29:12 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:29:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:29:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 08:29:12 Waiting for src bucket item count to get to: 50000 curr count: 49580
2021/12/10 08:29:17 Waiting for src bucket item count to get to: 50000 curr count: 49580
2021/12/10 08:29:27 Waiting for dst bucket item count to get to: 50000 curr count: 38673
2021/12/10 08:29:32 Waiting for dst bucket item count to get to: 50000 curr count: 38673
2021/12/10 08:29:37 Waiting for dst bucket item count to get to: 50000 curr count: 49901
2021/12/10 08:29:42 Waiting for dst bucket item count to get to: 50000 curr count: 49901
2021/12/10 08:29:42 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/10 08:29:42 Post SET ops. Expected item count: 50000 got 50000
2021/12/10 08:29:42 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 08:29:47 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/10 08:29:52 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/10 08:29:57 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/10 08:30:02 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/10 08:30:07 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/10 08:30:07 App: TestDuplicateTimerCreationEventingSwapReb got un-deployed
2021/12/10 08:30:07 No eventing-con process running
2021/12/10 08:30:08 Function: TestDuplicateTimerCreationEventingSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationEventingSwapReb deleting in the background"
}
2021/12/10 08:30:09 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:30:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 08:30:19 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/10 08:30:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 08:30:21 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:30:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 08:30:31 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/10 08:30:36 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/10 08:30:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationEventingSwapReb (179.80s)
=== RUN   TestDuplicateTimerCreationKVRebIn
2021/12/10 08:30:46 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/10 08:30:46 App: TestDuplicateTimerCreationKVRebIn got un-deployed
2021/12/10 08:30:46 No eventing-con process running
2021/12/10 08:30:46 Function: TestDuplicateTimerCreationKVRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationKVRebIn not found"
 }
}
2021/12/10 08:30:48 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:30:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 08:30:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 08:30:55 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:31:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 08:31:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:31:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 08:31:05 Deploying app: TestDuplicateTimerCreationKVRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/10 08:31:06 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationKVRebIn' in metakv",
  "warnings": null
 }
}
2021/12/10 08:31:11 Waiting for app: TestDuplicateTimerCreationKVRebIn to get deployed
2021/12/10 08:31:16 Waiting for app: TestDuplicateTimerCreationKVRebIn to get deployed
2021/12/10 08:31:21 Waiting for app: TestDuplicateTimerCreationKVRebIn to get deployed
2021/12/10 08:31:26 Waiting for app: TestDuplicateTimerCreationKVRebIn to get deployed
2021/12/10 08:31:26 App: TestDuplicateTimerCreationKVRebIn got deployed
2021/12/10 08:31:26 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/10 08:31:26 Starting bucket ops to source bucket
2021/12/10 08:31:26 Removing node(s): [] from the cluster
2021/12/10 08:31:31 Rebalance progress: 12.5
2021/12/10 08:31:36 Rebalance progress: 12.5
2021/12/10 08:31:41 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:31:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:31:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 08:31:42 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/10 08:31:47 Rebalance progress: 12.5
2021/12/10 08:31:52 Rebalance progress: 12.5
2021/12/10 08:31:57 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:31:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:31:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 08:31:57 Waiting for src bucket item count to get to: 50000 curr count: 30855
2021/12/10 08:32:02 Waiting for src bucket item count to get to: 50000 curr count: 42533
2021/12/10 08:32:07 Waiting for src bucket item count to get to: 50000 curr count: 42533
2021/12/10 08:32:17 Waiting for dst bucket item count to get to: 50000 curr count: 26712
2021/12/10 08:32:22 Waiting for dst bucket item count to get to: 50000 curr count: 26712
2021/12/10 08:32:27 Waiting for dst bucket item count to get to: 50000 curr count: 36320
2021/12/10 08:32:32 Waiting for dst bucket item count to get to: 50000 curr count: 36320
2021/12/10 08:32:37 Waiting for dst bucket item count to get to: 50000 curr count: 48468
2021/12/10 08:32:42 Waiting for dst bucket item count to get to: 50000 curr count: 48468
2021/12/10 08:32:42 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/10 08:32:42 Post SET ops. Expected item count: 50000 got 50000
2021/12/10 08:32:42 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 08:32:47 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/10 08:32:52 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/10 08:32:57 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/10 08:33:02 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/10 08:33:07 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/10 08:33:12 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/10 08:33:12 App: TestDuplicateTimerCreationKVRebIn got un-deployed
2021/12/10 08:33:12 No eventing-con process running
2021/12/10 08:33:12 Function: TestDuplicateTimerCreationKVRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationKVRebIn deleting in the background"
}
2021/12/10 08:33:14 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:33:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 08:33:24 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/10 08:33:29 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/10 08:33:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 08:33:30 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:33:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 08:33:40 Waiting for dst bucket item count to get to: 0 curr count: 29319
2021/12/10 08:33:46 Waiting for dst bucket item count to get to: 0 curr count: 29319
2021/12/10 08:33:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationKVRebIn (189.49s)
=== RUN   TestDuplicateTimerCreationKVRebOut
2021/12/10 08:33:51 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/10 08:33:51 Removing node(s): [] from the cluster
2021/12/10 08:33:56 Rebalance progress: 100
2021/12/10 08:34:01 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/10 08:34:01 App: TestDuplicateTimerCreationKVRebOut got un-deployed
2021/12/10 08:34:01 No eventing-con process running
2021/12/10 08:34:01 Function: TestDuplicateTimerCreationKVRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationKVRebOut not found"
 }
}
2021/12/10 08:34:03 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:34:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 08:34:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 08:34:09 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:34:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 08:34:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:34:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 08:34:19 Deploying app: TestDuplicateTimerCreationKVRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/10 08:34:20 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationKVRebOut' in metakv",
  "warnings": null
 }
}
2021/12/10 08:34:25 Waiting for app: TestDuplicateTimerCreationKVRebOut to get deployed
2021/12/10 08:34:30 Waiting for app: TestDuplicateTimerCreationKVRebOut to get deployed
2021/12/10 08:34:35 Waiting for app: TestDuplicateTimerCreationKVRebOut to get deployed
2021/12/10 08:34:40 Waiting for app: TestDuplicateTimerCreationKVRebOut to get deployed
2021/12/10 08:34:41 App: TestDuplicateTimerCreationKVRebOut got deployed
2021/12/10 08:34:41 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/10 08:34:41 Starting bucket ops to source bucket
2021/12/10 08:34:46 Rebalance progress: 12.5
2021/12/10 08:34:51 Rebalance progress: 12.5
2021/12/10 08:34:56 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:34:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:34:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 08:34:56 Waiting for src bucket item count to get to: 50000 curr count: 11290
2021/12/10 08:35:01 Waiting for src bucket item count to get to: 50000 curr count: 25311
2021/12/10 08:35:06 Waiting for src bucket item count to get to: 50000 curr count: 25311
2021/12/10 08:35:11 Waiting for src bucket item count to get to: 50000 curr count: 38040
2021/12/10 08:35:16 Waiting for src bucket item count to get to: 50000 curr count: 38040
2021/12/10 08:35:21 Waiting for src bucket item count to get to: 50000 curr count: 38040
2021/12/10 08:35:26 Waiting for src bucket item count to get to: 50000 curr count: 47741
2021/12/10 08:35:31 Waiting for src bucket item count to get to: 50000 curr count: 47741
2021/12/10 08:35:41 Waiting for dst bucket item count to get to: 50000 curr count: 31715
2021/12/10 08:35:46 Waiting for dst bucket item count to get to: 50000 curr count: 31715
2021/12/10 08:35:51 Waiting for dst bucket item count to get to: 50000 curr count: 44105
2021/12/10 08:35:56 Waiting for dst bucket item count to get to: 50000 curr count: 44105
2021/12/10 08:35:56 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/10 08:35:56 Post SET ops. Expected item count: 50000 got 50000
2021/12/10 08:35:56 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 08:36:01 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/10 08:36:06 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/10 08:36:11 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/10 08:36:16 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/10 08:36:21 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/10 08:36:26 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/10 08:36:26 App: TestDuplicateTimerCreationKVRebOut got un-deployed
2021/12/10 08:36:26 No eventing-con process running
2021/12/10 08:36:26 Function: TestDuplicateTimerCreationKVRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationKVRebOut deleting in the background"
}
2021/12/10 08:36:28 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:36:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 08:36:38 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/10 08:36:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 08:36:40 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:36:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 08:36:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationKVRebOut (179.06s)
=== RUN   TestDuplicateTimerCreationKVSwapReb
2021/12/10 08:36:55 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/10 08:36:55 App: TestDuplicateTimerCreationKVSwapReb got un-deployed
2021/12/10 08:36:55 No eventing-con process running
2021/12/10 08:36:55 Function: TestDuplicateTimerCreationKVSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationKVSwapReb not found"
 }
}
2021/12/10 08:36:56 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:37:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 08:37:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 08:37:03 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:37:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 08:37:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:37:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 08:37:13 Deploying app: TestDuplicateTimerCreationKVSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/10 08:37:14 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationKVSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/10 08:37:19 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get deployed
2021/12/10 08:37:24 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get deployed
2021/12/10 08:37:29 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get deployed
2021/12/10 08:37:34 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get deployed
2021/12/10 08:37:34 App: TestDuplicateTimerCreationKVSwapReb got deployed
2021/12/10 08:37:34 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/10 08:37:34 Starting bucket ops to source bucket
2021/12/10 08:37:35 Removing node(s): [] from the cluster
2021/12/10 08:37:40 Rebalance progress: 12.5
2021/12/10 08:37:45 Rebalance progress: 12.5
2021/12/10 08:37:50 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:37:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:37:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 08:37:50 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/10 08:37:50 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/10 08:37:55 Rebalance progress: 12.5
2021/12/10 08:38:00 Rebalance progress: 12.5
2021/12/10 08:38:05 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:38:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:38:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 08:38:05 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/10 08:38:10 Rebalance progress: 12.5
2021/12/10 08:38:15 Rebalance progress: 12.5
2021/12/10 08:38:20 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:38:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:38:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 08:38:20 Waiting for src bucket item count to get to: 50000 curr count: 41981
2021/12/10 08:38:30 Waiting for dst bucket item count to get to: 50000 curr count: 29749
2021/12/10 08:38:35 Waiting for dst bucket item count to get to: 50000 curr count: 35551
2021/12/10 08:38:40 Waiting for dst bucket item count to get to: 50000 curr count: 35551
2021/12/10 08:38:45 Waiting for dst bucket item count to get to: 50000 curr count: 49955
2021/12/10 08:38:50 Waiting for dst bucket item count to get to: 50000 curr count: 49955
2021/12/10 08:38:50 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/10 08:38:50 Post SET ops. Expected item count: 50000 got 50000
2021/12/10 08:38:50 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 08:38:55 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/10 08:39:00 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/10 08:39:06 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/10 08:39:11 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/10 08:39:16 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/10 08:39:21 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/10 08:39:21 App: TestDuplicateTimerCreationKVSwapReb got un-deployed
2021/12/10 08:39:21 No eventing-con process running
2021/12/10 08:39:21 Function: TestDuplicateTimerCreationKVSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationKVSwapReb deleting in the background"
}
2021/12/10 08:39:22 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:39:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 08:39:32 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/10 08:39:37 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/10 08:39:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 08:39:39 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:39:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 08:39:49 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/10 08:39:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationKVSwapReb (184.59s)
=== RUN   TestDuplicateTimerFiringEventingRebIn
2021/12/10 08:39:59 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/10 08:39:59 App: TestDuplicateTimerFiringEventingRebIn got un-deployed
2021/12/10 08:39:59 No eventing-con process running
2021/12/10 08:39:59 Function: TestDuplicateTimerFiringEventingRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringEventingRebIn not found"
 }
}
2021/12/10 08:40:01 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:40:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 08:40:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 08:40:08 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:40:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 08:40:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:40:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 08:40:18 Deploying app: TestDuplicateTimerFiringEventingRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/10 08:40:19 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringEventingRebIn' in metakv",
  "warnings": null
 }
}
2021/12/10 08:40:24 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get deployed
2021/12/10 08:40:29 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get deployed
2021/12/10 08:40:34 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get deployed
2021/12/10 08:40:39 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get deployed
2021/12/10 08:40:39 App: TestDuplicateTimerFiringEventingRebIn got deployed
2021/12/10 08:40:39 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/10 08:40:39 Starting bucket ops to source bucket
2021/12/10 08:40:39 Removing node(s): [] from the cluster
2021/12/10 08:40:44 Rebalance progress: 12.5
2021/12/10 08:40:49 Rebalance progress: 12.5
2021/12/10 08:40:54 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:40:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:40:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 08:40:54 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/10 08:41:00 Rebalance progress: 12.5
2021/12/10 08:41:05 Rebalance progress: 12.5
2021/12/10 08:41:10 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:41:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:41:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 08:41:10 Waiting for src bucket item count to get to: 50000 curr count: 26710
2021/12/10 08:41:15 Waiting for src bucket item count to get to: 50000 curr count: 36163
2021/12/10 08:41:20 Waiting for src bucket item count to get to: 50000 curr count: 36163
2021/12/10 08:41:25 Waiting for src bucket item count to get to: 50000 curr count: 47525
2021/12/10 08:41:30 Waiting for src bucket item count to get to: 50000 curr count: 47525
2021/12/10 08:41:40 Waiting for dst bucket item count to get to: 50000 curr count: 40723
2021/12/10 08:41:45 Waiting for dst bucket item count to get to: 50000 curr count: 40723
2021/12/10 08:41:50 Waiting for dst bucket item count to get to: 50000 curr count: 48804
2021/12/10 08:41:55 Waiting for dst bucket item count to get to: 50000 curr count: 48804
2021/12/10 08:41:55 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/10 08:41:55 Post SET ops. Expected item count: 50000 got 50000
2021/12/10 08:41:55 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 08:42:00 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/10 08:42:05 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/10 08:42:10 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/10 08:42:15 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/10 08:42:20 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/10 08:42:25 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/10 08:42:25 App: TestDuplicateTimerFiringEventingRebIn got un-deployed
2021/12/10 08:42:25 No eventing-con process running
2021/12/10 08:42:25 Function: TestDuplicateTimerFiringEventingRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringEventingRebIn deleting in the background"
}
2021/12/10 08:42:27 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:42:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 08:42:37 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/10 08:42:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 08:42:38 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:42:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 08:42:48 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/10 08:42:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringEventingRebIn (179.31s)
=== RUN   TestDuplicateTimerFiringEventingRebOut
2021/12/10 08:42:53 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/10 08:42:54 Removing node(s): [] from the cluster
2021/12/10 08:42:59 Rebalance progress: 100
2021/12/10 08:43:04 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/10 08:43:04 App: TestDuplicateTimerFiringEventingRebOut got un-deployed
2021/12/10 08:43:04 No eventing-con process running
2021/12/10 08:43:04 Function: TestDuplicateTimerFiringEventingRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringEventingRebOut not found"
 }
}
2021/12/10 08:43:06 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:43:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 08:43:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 08:43:12 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:43:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 08:43:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:43:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 08:43:22 Deploying app: TestDuplicateTimerFiringEventingRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/10 08:43:23 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringEventingRebOut' in metakv",
  "warnings": null
 }
}
2021/12/10 08:43:28 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get deployed
2021/12/10 08:43:33 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get deployed
2021/12/10 08:43:38 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get deployed
2021/12/10 08:43:43 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get deployed
2021/12/10 08:43:43 App: TestDuplicateTimerFiringEventingRebOut got deployed
2021/12/10 08:43:43 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/10 08:43:43 Starting bucket ops to source bucket
2021/12/10 08:43:48 Rebalance progress: 12.5
2021/12/10 08:43:53 Rebalance progress: 12.5
2021/12/10 08:43:58 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:43:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:43:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 08:43:59 Waiting for src bucket item count to get to: 50000 curr count: 7838
2021/12/10 08:44:04 Waiting for src bucket item count to get to: 50000 curr count: 20548
2021/12/10 08:44:09 Waiting for src bucket item count to get to: 50000 curr count: 20548
2021/12/10 08:44:14 Waiting for src bucket item count to get to: 50000 curr count: 31820
2021/12/10 08:44:19 Waiting for src bucket item count to get to: 50000 curr count: 31820
2021/12/10 08:44:24 Waiting for src bucket item count to get to: 50000 curr count: 31820
2021/12/10 08:44:29 Waiting for src bucket item count to get to: 50000 curr count: 42933
2021/12/10 08:44:34 Waiting for src bucket item count to get to: 50000 curr count: 42933
2021/12/10 08:44:44 Waiting for dst bucket item count to get to: 50000 curr count: 40030
2021/12/10 08:44:49 Waiting for dst bucket item count to get to: 50000 curr count: 40030
2021/12/10 08:44:49 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/10 08:44:49 Post SET ops. Expected item count: 50000 got 50000
2021/12/10 08:44:49 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 08:44:54 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/10 08:44:59 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/10 08:45:04 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/10 08:45:09 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/10 08:45:14 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/10 08:45:19 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/10 08:45:19 App: TestDuplicateTimerFiringEventingRebOut got un-deployed
2021/12/10 08:45:19 No eventing-con process running
2021/12/10 08:45:19 Function: TestDuplicateTimerFiringEventingRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringEventingRebOut deleting in the background"
}
2021/12/10 08:45:21 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:45:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 08:45:31 Waiting for dst bucket item count to get to: 0 curr count: 46150
2021/12/10 08:45:36 Waiting for dst bucket item count to get to: 0 curr count: 46150
2021/12/10 08:45:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 08:45:38 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:45:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 08:45:48 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/10 08:45:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringEventingRebOut (179.17s)
=== RUN   TestDuplicateTimerFiringEventingSwapReb
2021/12/10 08:45:58 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/10 08:45:58 App: TestDuplicateTimerFiringEventingSwapReb got un-deployed
2021/12/10 08:45:58 No eventing-con process running
2021/12/10 08:45:58 Function: TestDuplicateTimerFiringEventingSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringEventingSwapReb not found"
 }
}
2021/12/10 08:45:59 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:46:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 08:46:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 08:46:06 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:46:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 08:46:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:46:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 08:46:16 Deploying app: TestDuplicateTimerFiringEventingSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/10 08:46:17 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringEventingSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/10 08:46:22 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get deployed
2021/12/10 08:46:27 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get deployed
2021/12/10 08:46:32 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get deployed
2021/12/10 08:46:37 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get deployed
2021/12/10 08:46:37 App: TestDuplicateTimerFiringEventingSwapReb got deployed
2021/12/10 08:46:37 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/10 08:46:37 Starting bucket ops to source bucket
2021/12/10 08:46:38 Removing node(s): [] from the cluster
2021/12/10 08:46:43 Rebalance progress: 12.5
2021/12/10 08:46:48 Rebalance progress: 12.5
2021/12/10 08:46:53 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:46:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:46:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 08:46:53 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/10 08:46:53 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/10 08:46:58 Rebalance progress: 12.5
2021/12/10 08:47:03 Rebalance progress: 12.5
2021/12/10 08:47:08 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:47:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:47:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 08:47:08 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/10 08:47:13 Rebalance progress: 12.5
2021/12/10 08:47:18 Rebalance progress: 12.5
2021/12/10 08:47:23 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:47:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:47:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 08:47:23 Waiting for src bucket item count to get to: 50000 curr count: 48000
2021/12/10 08:47:28 Waiting for src bucket item count to get to: 50000 curr count: 48000
2021/12/10 08:47:38 Waiting for dst bucket item count to get to: 50000 curr count: 47325
2021/12/10 08:47:43 Waiting for dst bucket item count to get to: 50000 curr count: 47325
2021/12/10 08:47:43 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/10 08:47:43 Post SET ops. Expected item count: 50000 got 50000
2021/12/10 08:47:43 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 08:47:48 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/10 08:47:53 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/10 08:47:58 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/10 08:48:04 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/10 08:48:09 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/10 08:48:14 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/10 08:48:14 App: TestDuplicateTimerFiringEventingSwapReb got un-deployed
2021/12/10 08:48:14 No eventing-con process running
2021/12/10 08:48:14 Function: TestDuplicateTimerFiringEventingSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringEventingSwapReb deleting in the background"
}
2021/12/10 08:48:15 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:48:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 08:48:25 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/10 08:48:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 08:48:27 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:48:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 08:48:37 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/10 08:48:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringEventingSwapReb (169.55s)
=== RUN   TestDuplicateTimerFiringKVRebIn
2021/12/10 08:48:47 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/10 08:48:47 App: TestDuplicateTimerFiringKVRebIn got un-deployed
2021/12/10 08:48:47 No eventing-con process running
2021/12/10 08:48:47 Function: TestDuplicateTimerFiringKVRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringKVRebIn not found"
 }
}
2021/12/10 08:48:49 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:48:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 08:48:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 08:48:56 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:49:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 08:49:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:49:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 08:49:06 Deploying app: TestDuplicateTimerFiringKVRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/10 08:49:07 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringKVRebIn' in metakv",
  "warnings": null
 }
}
2021/12/10 08:49:12 Waiting for app: TestDuplicateTimerFiringKVRebIn to get deployed
2021/12/10 08:49:17 Waiting for app: TestDuplicateTimerFiringKVRebIn to get deployed
2021/12/10 08:49:22 Waiting for app: TestDuplicateTimerFiringKVRebIn to get deployed
2021/12/10 08:49:27 Waiting for app: TestDuplicateTimerFiringKVRebIn to get deployed
2021/12/10 08:49:27 App: TestDuplicateTimerFiringKVRebIn got deployed
2021/12/10 08:49:27 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/10 08:49:27 Starting bucket ops to source bucket
2021/12/10 08:49:28 Removing node(s): [] from the cluster
2021/12/10 08:49:33 Rebalance progress: 12.5
2021/12/10 08:49:38 Rebalance progress: 12.5
2021/12/10 08:49:43 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:49:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:49:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 08:49:43 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/10 08:49:48 Rebalance progress: 12.5
2021/12/10 08:49:53 Rebalance progress: 12.5
2021/12/10 08:49:58 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:49:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:49:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 08:49:58 Waiting for src bucket item count to get to: 50000 curr count: 27390
2021/12/10 08:50:03 Waiting for src bucket item count to get to: 50000 curr count: 38244
2021/12/10 08:50:08 Waiting for src bucket item count to get to: 50000 curr count: 38244
2021/12/10 08:50:13 Waiting for src bucket item count to get to: 50000 curr count: 49854
2021/12/10 08:50:18 Waiting for src bucket item count to get to: 50000 curr count: 49854
2021/12/10 08:50:28 Waiting for dst bucket item count to get to: 50000 curr count: 46609
2021/12/10 08:50:33 Waiting for dst bucket item count to get to: 50000 curr count: 46609
2021/12/10 08:50:33 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/10 08:50:33 Post SET ops. Expected item count: 50000 got 50000
2021/12/10 08:50:33 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 08:50:38 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/10 08:50:43 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/10 08:50:48 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/10 08:50:53 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/10 08:50:58 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/10 08:51:03 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/10 08:51:03 App: TestDuplicateTimerFiringKVRebIn got un-deployed
2021/12/10 08:51:03 No eventing-con process running
2021/12/10 08:51:03 Function: TestDuplicateTimerFiringKVRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringKVRebIn deleting in the background"
}
2021/12/10 08:51:05 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:51:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 08:51:15 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/10 08:51:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 08:51:17 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:51:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 08:51:27 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/10 08:51:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringKVRebIn (169.74s)
=== RUN   TestDuplicateTimerFiringKVRebOut
2021/12/10 08:51:32 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/10 08:51:32 Removing node(s): [] from the cluster
2021/12/10 08:51:37 Rebalance progress: 100
2021/12/10 08:51:42 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/10 08:51:42 App: TestDuplicateTimerFiringKVRebOut got un-deployed
2021/12/10 08:51:42 No eventing-con process running
2021/12/10 08:51:42 Function: TestDuplicateTimerFiringKVRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringKVRebOut not found"
 }
}
2021/12/10 08:51:44 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:51:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 08:51:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 08:51:51 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:51:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 08:51:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:52:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 08:52:01 Deploying app: TestDuplicateTimerFiringKVRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/10 08:52:02 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringKVRebOut' in metakv",
  "warnings": null
 }
}
2021/12/10 08:52:07 Waiting for app: TestDuplicateTimerFiringKVRebOut to get deployed
2021/12/10 08:52:12 Waiting for app: TestDuplicateTimerFiringKVRebOut to get deployed
2021/12/10 08:52:17 Waiting for app: TestDuplicateTimerFiringKVRebOut to get deployed
2021/12/10 08:52:22 Waiting for app: TestDuplicateTimerFiringKVRebOut to get deployed
2021/12/10 08:52:22 App: TestDuplicateTimerFiringKVRebOut got deployed
2021/12/10 08:52:22 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/10 08:52:22 Starting bucket ops to source bucket
2021/12/10 08:52:27 Rebalance progress: 12.5
2021/12/10 08:52:32 Rebalance progress: 12.5
2021/12/10 08:52:37 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:52:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:52:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 08:52:37 Waiting for src bucket item count to get to: 50000 curr count: 9715
2021/12/10 08:52:42 Waiting for src bucket item count to get to: 50000 curr count: 22411
2021/12/10 08:52:47 Waiting for src bucket item count to get to: 50000 curr count: 22411
2021/12/10 08:52:52 Waiting for src bucket item count to get to: 50000 curr count: 34191
2021/12/10 08:52:57 Waiting for src bucket item count to get to: 50000 curr count: 34191
2021/12/10 08:53:02 Waiting for src bucket item count to get to: 50000 curr count: 34191
2021/12/10 08:53:07 Waiting for src bucket item count to get to: 50000 curr count: 44367
2021/12/10 08:53:12 Waiting for src bucket item count to get to: 50000 curr count: 44367
2021/12/10 08:53:22 Waiting for dst bucket item count to get to: 50000 curr count: 39404
2021/12/10 08:53:27 Waiting for dst bucket item count to get to: 50000 curr count: 39404
2021/12/10 08:53:27 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/10 08:53:27 Post SET ops. Expected item count: 50000 got 50000
2021/12/10 08:53:27 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 08:53:32 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/10 08:53:37 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/10 08:53:42 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/10 08:53:47 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/10 08:53:52 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/10 08:53:58 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/10 08:53:58 App: TestDuplicateTimerFiringKVRebOut got un-deployed
2021/12/10 08:53:58 No eventing-con process running
2021/12/10 08:53:58 Function: TestDuplicateTimerFiringKVRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringKVRebOut deleting in the background"
}
2021/12/10 08:53:59 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:54:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 08:54:09 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/10 08:54:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 08:54:11 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:54:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 08:54:21 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/10 08:54:26 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/10 08:54:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringKVRebOut (179.15s)
=== RUN   TestDuplicateTimerFiringKVSwapReb
2021/12/10 08:54:36 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/10 08:54:36 App: TestDuplicateTimerFiringKVSwapReb got un-deployed
2021/12/10 08:54:36 No eventing-con process running
2021/12/10 08:54:36 Function: TestDuplicateTimerFiringKVSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringKVSwapReb not found"
 }
}
2021/12/10 08:54:38 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:54:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 08:54:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 08:54:45 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:54:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 08:54:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:54:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 08:54:55 Deploying app: TestDuplicateTimerFiringKVSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/10 08:54:56 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringKVSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/10 08:55:01 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get deployed
2021/12/10 08:55:06 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get deployed
2021/12/10 08:55:11 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get deployed
2021/12/10 08:55:16 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get deployed
2021/12/10 08:55:16 App: TestDuplicateTimerFiringKVSwapReb got deployed
2021/12/10 08:55:16 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/10 08:55:16 Starting bucket ops to source bucket
2021/12/10 08:55:16 Removing node(s): [] from the cluster
2021/12/10 08:55:21 Rebalance progress: 12.5
2021/12/10 08:55:26 Rebalance progress: 12.5
2021/12/10 08:55:31 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:55:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:55:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 08:55:32 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/10 08:55:32 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/10 08:55:37 Rebalance progress: 12.5
2021/12/10 08:55:42 Rebalance progress: 12.5
2021/12/10 08:55:47 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:55:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:55:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 08:55:47 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/10 08:55:52 Rebalance progress: 12.5
2021/12/10 08:55:57 Rebalance progress: 12.5
2021/12/10 08:56:02 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:56:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:56:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 08:56:02 Waiting for src bucket item count to get to: 50000 curr count: 48559
2021/12/10 08:56:07 Waiting for src bucket item count to get to: 50000 curr count: 48559
2021/12/10 08:56:17 Waiting for dst bucket item count to get to: 50000 curr count: 46280
2021/12/10 08:56:22 Waiting for dst bucket item count to get to: 50000 curr count: 46280
2021/12/10 08:56:22 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/10 08:56:22 Post SET ops. Expected item count: 50000 got 50000
2021/12/10 08:56:22 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 08:56:27 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/10 08:56:32 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/10 08:56:37 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/10 08:56:42 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/10 08:56:47 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/10 08:56:52 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/10 08:56:52 App: TestDuplicateTimerFiringKVSwapReb got un-deployed
2021/12/10 08:56:52 No eventing-con process running
2021/12/10 08:56:52 Function: TestDuplicateTimerFiringKVSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringKVSwapReb deleting in the background"
}
2021/12/10 08:56:54 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:56:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 08:57:04 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/10 08:57:09 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/10 08:57:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 08:57:11 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:57:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 08:57:21 Waiting for dst bucket item count to get to: 0 curr count: 48502
2021/12/10 08:57:26 Waiting for dst bucket item count to get to: 0 curr count: 48502
2021/12/10 08:57:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringKVSwapReb (179.89s)
=== RUN   TestKVRebOnUpdateBucketOpOneByOne
2021/12/10 08:57:36 Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 08:57:36 App: bucket_op_on_update got un-deployed
2021/12/10 08:57:36 No eventing-con process running
2021/12/10 08:57:36 Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/10 08:57:38 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:57:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 08:57:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 08:57:44 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:57:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 08:57:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:57:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 08:57:54 Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/10 08:57:56 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/10 08:58:01 Starting bucket ops to source bucket
2021/12/10 08:58:06 Waiting for app: bucket_op_on_update to get deployed
2021/12/10 08:58:11 Waiting for app: bucket_op_on_update to get deployed
2021/12/10 08:58:16 Waiting for app: bucket_op_on_update to get deployed
2021/12/10 08:58:16 App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:58:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:58:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 08:58:16 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/10 08:58:16 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/10 08:58:21 Rebalance progress: 12.5
2021/12/10 08:58:26 Rebalance progress: 12.5
2021/12/10 08:58:31 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:58:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:58:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 08:58:31 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/10 08:58:32 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/10 08:58:37 Rebalance progress: 12.5
2021/12/10 08:58:42 Rebalance progress: 12.5
2021/12/10 08:58:47 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:58:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:58:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 08:58:47 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: kv to the cluster
2021/12/10 08:58:47 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/10 08:58:52 Rebalance progress: 12.5
2021/12/10 08:58:57 Rebalance progress: 12.5
2021/12/10 08:59:02 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:59:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:59:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 08:59:02 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/10 08:59:07 Rebalance progress: 12.5
2021/12/10 08:59:12 Rebalance progress: 12.5
2021/12/10 08:59:17 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:59:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:59:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 08:59:17 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/10 08:59:22 Rebalance progress: 12.5
2021/12/10 08:59:27 Rebalance progress: 12.5
2021/12/10 08:59:32 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:59:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:59:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 08:59:32 clusterSetup.go:253: Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/10 08:59:37 Rebalance progress: 12.5
2021/12/10 08:59:42 Rebalance progress: 12.5
2021/12/10 08:59:47 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:59:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 08:59:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 08:59:47 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 08:59:52 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 08:59:57 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 09:00:02 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 09:00:07 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 09:00:12 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 09:00:12 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/10 09:00:12 eventingSetup.go:46: No eventing-con process running
2021/12/10 09:00:13 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/10 09:00:14 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 09:00:19 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 09:00:24 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 10647
2021/12/10 09:00:29 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 10647
2021/12/10 09:00:29 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 09:00:31 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 09:00:36 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 09:00:41 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 10647
2021/12/10 09:00:46 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 10647
2021/12/10 09:00:46 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebOnUpdateBucketOpOneByOne (200.02s)
=== RUN   TestKVForceFailover
2021/12/10 09:01:01 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 09:01:01 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/10 09:01:01 eventingSetup.go:46: No eventing-con process running
2021/12/10 09:01:01 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/10 09:01:03 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 09:01:08 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 09:01:08 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 09:01:10 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 09:01:15 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 09:01:15 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 09:01:15 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/10 09:01:15 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/10 09:01:20 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 09:01:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 09:01:20 Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/10 09:01:21 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/10 09:01:21 Starting bucket ops to source bucket
2021/12/10 09:01:26 Waiting for app: bucket_op_on_update to get deployed
2021/12/10 09:01:31 Waiting for app: bucket_op_on_update to get deployed
2021/12/10 09:01:36 Waiting for app: bucket_op_on_update to get deployed
2021/12/10 09:01:41 Waiting for app: bucket_op_on_update to get deployed
2021/12/10 09:01:41 App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 09:01:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 09:01:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 09:01:41 clusterSetup.go:229: Failing over: [127.0.0.1:19001]
2021/12/10 09:02:01 clusterSetup.go:237: Kicking off failover recovery, type: full
2021/12/10 09:02:01 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/10 09:02:06 Rebalance progress: 12.5
2021/12/10 09:02:11 Rebalance progress: 12.5
2021/12/10 09:02:16 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 09:02:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 09:02:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 09:02:17 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002] from the cluster
2021/12/10 09:02:22 Rebalance progress: 12.5
2021/12/10 09:02:27 Rebalance progress: 12.5
2021/12/10 09:02:32 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 09:02:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 09:02:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 09:02:32 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 09:02:37 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 09:02:42 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 09:02:47 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 09:02:52 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 09:02:57 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 09:03:02 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 09:03:02 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/10 09:03:02 eventingSetup.go:46: No eventing-con process running
2021/12/10 09:03:02 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/10 09:03:04 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 09:03:09 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 09:03:14 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7032
2021/12/10 09:03:19 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7032
2021/12/10 09:03:19 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 09:03:20 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 09:03:25 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 09:03:30 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3814
2021/12/10 09:03:35 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3814
2021/12/10 09:03:35 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVForceFailover (169.50s)
=== RUN   TestKVHardFailoverRecovery
2021/12/10 09:03:46 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 09:03:46 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/10 09:03:46 eventingSetup.go:46: No eventing-con process running
2021/12/10 09:03:46 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/10 09:03:47 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 09:03:52 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 09:03:52 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 09:03:54 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 09:03:59 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 09:03:59 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 09:04:04 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 09:04:04 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/10 09:04:05 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/10 09:04:10 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/10 09:04:15 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/10 09:04:20 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/10 09:04:25 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/10 09:04:25 clusterSetup.go:354: App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 09:04:25 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 09:04:25 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 09:04:25 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/10 09:04:25 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/10 09:04:30 Rebalance progress: 12.5
2021/12/10 09:04:35 Rebalance progress: 12.5
2021/12/10 09:04:40 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 09:04:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 09:04:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 09:04:40 clusterSetup.go:229: Failing over: [127.0.0.1:19001]
2021/12/10 09:05:40 clusterSetup.go:237: Kicking off failover recovery, type: full
2021/12/10 09:05:46 Rebalance progress: 12.5
2021/12/10 09:05:51 Rebalance progress: 12.5
2021/12/10 09:05:56 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 09:05:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 09:05:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 09:05:56 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/10 09:06:01 Rebalance progress: 12.5
2021/12/10 09:06:06 Rebalance progress: 12.5
2021/12/10 09:06:11 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 09:06:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 09:06:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 09:06:11 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 09:06:16 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 09:06:21 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 09:06:26 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 09:06:31 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 09:06:36 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 09:06:41 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 09:06:41 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/10 09:06:41 eventingSetup.go:46: No eventing-con process running
2021/12/10 09:06:41 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/10 09:06:43 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 09:06:48 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 09:06:53 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12042
2021/12/10 09:06:58 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12042
2021/12/10 09:06:58 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 09:07:00 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 09:07:05 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 09:07:05 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVHardFailoverRecovery (209.06s)
=== RUN   TestKVRebOnUpdateBucketOpAllAtOnce
2021/12/10 09:07:15 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 09:07:15 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/10 09:07:15 eventingSetup.go:46: No eventing-con process running
2021/12/10 09:07:15 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/10 09:07:16 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 09:07:21 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 09:07:21 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 09:07:23 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 09:07:28 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 09:07:28 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 09:07:33 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 09:07:33 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/10 09:07:34 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/10 09:07:39 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/10 09:07:44 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/10 09:07:49 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/10 09:07:54 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/10 09:07:54 clusterSetup.go:354: App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 09:07:54 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 09:07:54 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 09:07:54 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/10 09:07:55 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/10 09:07:56 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: kv to the cluster
2021/12/10 09:07:56 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/10 09:08:01 Rebalance progress: 12.5
2021/12/10 09:08:06 Rebalance progress: 12.5
2021/12/10 09:08:11 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 09:08:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 09:08:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 09:08:11 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002 127.0.0.1:19003] from the cluster
2021/12/10 09:08:16 Rebalance progress: 12.5
2021/12/10 09:08:21 Rebalance progress: 12.5
2021/12/10 09:08:26 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 09:08:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 09:08:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 09:08:26 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 09:08:31 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 09:08:36 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 09:08:41 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 09:08:46 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 09:08:51 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 09:08:56 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 09:08:56 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/10 09:08:56 eventingSetup.go:46: No eventing-con process running
2021/12/10 09:08:57 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/10 09:08:58 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 09:09:03 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 09:09:08 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4663
2021/12/10 09:09:08 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 09:09:10 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 09:09:15 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 09:09:20 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 404
2021/12/10 09:09:25 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 404
2021/12/10 09:09:25 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebOnUpdateBucketOpAllAtOnce (140.42s)
=== RUN   TestKVSwapRebOnUpdateBucketOp
2021/12/10 09:09:35 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 09:09:35 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/10 09:09:35 eventingSetup.go:46: No eventing-con process running
2021/12/10 09:09:35 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/10 09:09:37 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 09:09:42 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 09:09:42 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 09:09:43 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 09:09:48 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 09:09:48 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 09:09:48 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 09:09:48 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/10 09:09:50 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/10 09:09:55 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/10 09:10:00 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/10 09:10:05 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/10 09:10:10 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/10 09:10:10 clusterSetup.go:354: App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 09:10:10 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 09:10:10 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 09:10:10 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/10 09:10:10 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/10 09:10:15 Rebalance progress: 12.5
2021/12/10 09:10:20 Rebalance progress: 12.5
2021/12/10 09:10:25 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 09:10:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 09:10:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 09:10:25 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/10 09:10:25 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/10 09:10:30 Rebalance progress: 12.5
2021/12/10 09:10:35 Rebalance progress: 12.5
2021/12/10 09:10:40 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 09:10:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 09:10:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 09:10:40 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/10 09:10:45 Rebalance progress: 12.5
2021/12/10 09:10:50 Rebalance progress: 12.5
2021/12/10 09:10:55 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 09:10:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 09:10:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 09:10:55 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 09:11:00 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 09:11:05 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 09:11:11 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 09:11:16 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 09:11:21 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 09:11:26 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 09:11:26 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/10 09:11:26 eventingSetup.go:46: No eventing-con process running
2021/12/10 09:11:26 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/10 09:11:27 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 09:11:32 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 09:11:37 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6062
2021/12/10 09:11:37 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 09:11:39 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 09:11:44 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 09:11:49 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6062
2021/12/10 09:11:49 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 09:11:54 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 09:11:54 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/10 09:11:54 eventingSetup.go:46: No eventing-con process running
2021/12/10 09:11:54 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/10 09:11:56 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 09:12:01 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 09:12:01 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 09:12:03 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 09:12:08 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 09:12:08 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVSwapRebOnUpdateBucketOp (162.68s)
=== RUN   TestKVRebalanceWithMultipleHandlers
2021/12/10 09:12:18 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 09:12:18 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/10 09:12:18 eventingSetup.go:46: No eventing-con process running
2021/12/10 09:12:18 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/10 09:12:19 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 09:12:24 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 09:12:24 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 09:12:26 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 09:12:31 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 09:12:31 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 09:12:36 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/10 09:12:36 clusterSetup.go:405: App: sys_test_bucket_op got un-deployed
2021/12/10 09:12:36 eventingSetup.go:46: No eventing-con process running
2021/12/10 09:12:36 eventingSetup.go:418: Function: sys_test_bucket_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: sys_test_bucket_op not found"
 }
}
2021/12/10 09:12:38 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 09:12:43 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 09:12:43 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 09:12:45 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 09:12:50 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 09:12:50 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 09:12:50 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 09:12:50 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/10 09:12:51 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 09:12:51 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 09:12:51 eventingSetup.go:125: Deploying app: sys_test_bucket_op. Item count src bucket: 0 dst bucket: 0
2021/12/10 09:12:52 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'sys_test_bucket_op' in metakv",
  "warnings": null
 }
}
2021/12/10 09:12:57 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/10 09:13:02 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/10 09:13:07 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/10 09:13:12 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/10 09:13:12 clusterSetup.go:354: App: bucket_op_on_update got deployed
2021/12/10 09:13:17 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/10 09:13:22 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/10 09:13:27 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/10 09:13:27 clusterSetup.go:354: App: sys_test_bucket_op got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 09:13:27 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 09:13:27 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 09:13:27 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/10 09:13:28 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/10 09:13:33 Rebalance progress: 12.5
2021/12/10 09:13:38 Rebalance progress: 12.5
2021/12/10 09:13:43 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 09:13:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 09:13:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 09:13:43 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/10 09:13:44 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/10 09:13:49 Rebalance progress: 12.5
2021/12/10 09:13:54 Rebalance progress: 12.5
2021/12/10 09:13:59 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 09:13:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 09:13:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 09:13:59 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: kv to the cluster
2021/12/10 09:13:59 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/10 09:14:04 Rebalance progress: 12.5
2021/12/10 09:14:09 Rebalance progress: 12.5
2021/12/10 09:14:14 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 09:14:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 09:14:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 09:14:14 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/10 09:14:20 Rebalance progress: 12.5
2021/12/10 09:14:25 Rebalance progress: 12.5
2021/12/10 09:14:30 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 09:14:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 09:14:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 09:14:30 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/10 09:14:35 Rebalance progress: 12.5
2021/12/10 09:14:40 Rebalance progress: 12.5
2021/12/10 09:14:45 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 09:14:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 09:14:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 09:14:45 clusterSetup.go:253: Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/10 09:14:50 Rebalance progress: 12.5
2021/12/10 09:14:55 Rebalance progress: 12.5
2021/12/10 09:15:00 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 09:15:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 09:15:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 09:15:00 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 09:15:05 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 09:15:10 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 09:15:15 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 09:15:20 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 09:15:25 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 09:15:30 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 09:15:30 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/10 09:15:30 eventingSetup.go:46: No eventing-con process running
2021/12/10 09:15:30 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/10 09:15:34 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 09:15:39 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 09:15:44 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12255
2021/12/10 09:15:44 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 09:15:46 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 09:15:51 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 09:15:56 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12255
2021/12/10 09:15:56 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 09:15:56 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 09:16:01 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/10 09:16:06 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/10 09:16:11 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/10 09:16:16 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/10 09:16:21 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/10 09:16:26 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/10 09:16:26 clusterSetup.go:405: App: sys_test_bucket_op got un-deployed
2021/12/10 09:16:26 eventingSetup.go:46: No eventing-con process running
2021/12/10 09:16:27 eventingSetup.go:418: Function: sys_test_bucket_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: sys_test_bucket_op deleting in the background"
}
2021/12/10 09:16:28 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 09:16:33 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 09:16:33 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 09:16:35 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 09:16:40 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 09:16:40 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebalanceWithMultipleHandlers (272.47s)
=== RUN   TestKVSwapRebalanceWithMultipleHandlers
2021/12/10 09:16:50 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 09:16:50 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/10 09:16:50 eventingSetup.go:46: No eventing-con process running
2021/12/10 09:16:50 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/10 09:16:52 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 09:16:57 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 09:16:57 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 09:16:59 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 09:17:04 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 09:17:04 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 09:17:09 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/10 09:17:09 clusterSetup.go:405: App: sys_test_bucket_op got un-deployed
2021/12/10 09:17:09 eventingSetup.go:46: No eventing-con process running
2021/12/10 09:17:09 eventingSetup.go:418: Function: sys_test_bucket_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: sys_test_bucket_op not found"
 }
}
2021/12/10 09:17:10 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 09:17:15 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 09:17:15 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 09:17:17 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 09:17:22 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 09:17:22 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 09:17:22 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 09:17:22 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/10 09:17:23 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 09:17:23 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 09:17:23 eventingSetup.go:125: Deploying app: sys_test_bucket_op. Item count src bucket: 0 dst bucket: 0
2021/12/10 09:17:24 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'sys_test_bucket_op' in metakv",
  "warnings": null
 }
}
2021/12/10 09:17:29 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/10 09:17:34 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/10 09:17:39 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/10 09:17:44 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/10 09:17:44 clusterSetup.go:354: App: bucket_op_on_update got deployed
2021/12/10 09:17:49 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/10 09:17:54 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/10 09:17:59 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/10 09:17:59 clusterSetup.go:354: App: sys_test_bucket_op got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 09:17:59 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 09:17:59 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 09:18:00 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/10 09:18:00 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/10 09:18:05 Rebalance progress: 12.5
2021/12/10 09:18:10 Rebalance progress: 12.5
2021/12/10 09:18:15 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 09:18:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 09:18:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 09:18:15 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/10 09:18:16 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/10 09:18:21 Rebalance progress: 12.5
2021/12/10 09:18:26 Rebalance progress: 12.5
2021/12/10 09:18:31 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 09:18:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 09:18:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 09:18:31 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/10 09:18:36 Rebalance progress: 12.5
2021/12/10 09:18:41 Rebalance progress: 12.5
2021/12/10 09:18:46 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 09:18:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 09:18:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 09:18:46 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 09:18:51 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 09:18:56 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 09:19:01 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 09:19:06 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 09:19:11 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 09:19:16 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 09:19:16 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/10 09:19:16 eventingSetup.go:46: No eventing-con process running
2021/12/10 09:19:16 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/10 09:19:19 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 09:19:24 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 09:19:25 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 09:19:26 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 09:19:31 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 09:19:36 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7632
2021/12/10 09:19:36 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 09:19:36 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 09:19:41 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/10 09:19:46 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/10 09:19:51 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/10 09:19:57 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/10 09:20:02 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/10 09:20:07 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/10 09:20:07 clusterSetup.go:405: App: sys_test_bucket_op got un-deployed
2021/12/10 09:20:07 eventingSetup.go:46: No eventing-con process running
2021/12/10 09:20:07 eventingSetup.go:418: Function: sys_test_bucket_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: sys_test_bucket_op deleting in the background"
}
2021/12/10 09:20:08 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 09:20:13 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 09:20:13 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 09:20:15 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 09:20:20 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 09:20:20 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVSwapRebalanceWithMultipleHandlers (220.09s)
=== RUN   TestKVRebStopStartKVOpsOnUpdateBucketOpOneByOne
2021/12/10 09:20:30 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 09:20:30 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/10 09:20:30 eventingSetup.go:46: No eventing-con process running
2021/12/10 09:20:30 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/10 09:20:32 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 09:20:37 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 09:20:37 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 09:20:39 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 09:20:44 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 09:20:44 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 09:20:49 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 09:20:49 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/10 09:20:50 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/10 09:20:55 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/10 09:21:00 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/10 09:21:05 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/10 09:21:10 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/10 09:21:10 clusterSetup.go:354: App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 09:21:10 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 09:21:10 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 09:21:10 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/10 09:21:10 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/10 09:21:30 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 09:21:31 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 09:21:31 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 09:21:41 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 09:21:41 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 09:21:41 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/10 09:21:46 Rebalance progress: 12.5
2021/12/10 09:21:51 Rebalance progress: 12.5
2021/12/10 09:21:56 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 09:21:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 09:21:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 09:21:56 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/10 09:21:56 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/10 09:22:01 Rebalance progress: 12.5
2021/12/10 09:22:06 Rebalance progress: 12.5
2021/12/10 09:22:11 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 09:22:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 09:22:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 09:22:11 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/10 09:22:16 Rebalance progress: 12.5
2021/12/10 09:22:21 Rebalance progress: 12.5
2021/12/10 09:22:26 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 09:22:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 09:22:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 09:22:26 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 09:22:31 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 09:22:37 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 09:22:42 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 09:22:47 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 09:22:52 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 09:22:57 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 09:22:57 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/10 09:22:57 eventingSetup.go:46: No eventing-con process running
2021/12/10 09:22:57 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/10 09:22:58 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 09:23:03 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 09:23:08 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 9138
2021/12/10 09:23:08 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 09:23:10 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 09:23:15 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 09:23:20 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3105
2021/12/10 09:23:25 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3105
2021/12/10 09:23:25 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebStopStartKVOpsOnUpdateBucketOpOneByOne (185.06s)
=== RUN   TestKVFailoverOnUpdateBucketOp
2021/12/10 09:23:35 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 09:23:35 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/10 09:23:35 eventingSetup.go:46: No eventing-con process running
2021/12/10 09:23:35 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/10 09:23:37 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 09:23:42 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 09:23:42 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 09:23:44 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 09:23:49 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 09:23:49 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 09:23:54 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 09:23:54 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/10 09:23:55 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/10 09:24:00 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/10 09:24:05 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/10 09:24:10 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/10 09:24:15 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/10 09:24:15 clusterSetup.go:354: App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 09:24:15 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 09:24:15 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 09:24:15 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/10 09:24:15 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/10 09:24:20 Rebalance progress: 12.5
2021/12/10 09:24:25 Rebalance progress: 12.5
2021/12/10 09:24:30 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 09:24:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 09:24:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 09:24:30 clusterSetup.go:229: Failing over: [127.0.0.1:19001]
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 09:24:40 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 09:24:40 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 09:24:40 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/10 09:24:45 Rebalance progress: 12.5
2021/12/10 09:24:50 Rebalance progress: 12.5
2021/12/10 09:24:55 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 09:24:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 09:24:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 09:24:55 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/10 09:24:56 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/10 09:25:01 Rebalance progress: 12.5
2021/12/10 09:25:06 Rebalance progress: 12.5
2021/12/10 09:25:11 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 09:25:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 09:25:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 09:25:11 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/10 09:25:16 Rebalance progress: 12.5
2021/12/10 09:25:21 Rebalance progress: 12.5
2021/12/10 09:25:26 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 09:25:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 09:25:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 09:25:26 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 09:25:31 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 09:25:36 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 09:25:41 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 09:25:46 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 09:25:51 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 09:25:56 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 09:25:56 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/10 09:25:56 eventingSetup.go:46: No eventing-con process running
2021/12/10 09:25:56 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/10 09:25:58 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 09:26:03 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 09:26:08 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 8604
2021/12/10 09:26:08 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 09:26:10 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 09:26:15 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 09:26:20 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 607
2021/12/10 09:26:25 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 607
2021/12/10 09:26:25 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVFailoverOnUpdateBucketOp (179.71s)
=== RUN   TestBootstrapAfterKVHardFailover
2021/12/10 09:26:30 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/10 09:26:30 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/10 09:26:35 Rebalance progress: 100
2021/12/10 09:26:35 Failing over: [127.0.0.1:19001]
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 09:27:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 09:27:05 Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/10 09:27:06 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/10 09:27:11 Removing node(s): [] from the cluster
2021/12/10 09:27:11 Starting bucket ops to source bucket
2021/12/10 09:27:16 Rebalance failed. See logs for detailed reason. You can try again.
2021/12/10 09:27:21 Waiting for app: bucket_op_on_update to get deployed
2021/12/10 09:27:26 Waiting for app: bucket_op_on_update to get deployed
2021/12/10 09:27:26 App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 09:27:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 09:27:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 09:27:27 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 09:27:32 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 09:27:37 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 09:27:42 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 09:27:47 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 09:27:52 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 09:27:57 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 09:27:57 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/10 09:27:57 eventingSetup.go:46: No eventing-con process running
2021/12/10 09:27:57 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/10 09:27:59 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 09:28:04 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 09:28:09 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1508
2021/12/10 09:28:09 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 09:28:10 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 09:28:15 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 09:28:20 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 721
2021/12/10 09:28:25 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 721
2021/12/10 09:28:25 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestBootstrapAfterKVHardFailover (120.42s)
PASS
ok  	github.com/couchbase/eventing/tests/functional_tests	12040.871s

Rebalance tests took 3 hours, 20 minutes and 43 seconds.

Testing Succeeded


Version: versions-10.12.2021-01.52.cfg
Build Log: make-10.12.2021-01.52.log
Server Log: logs-10.12.2021-01.52.tar.gz

Finished