Building

Started building at 2021/12/15 18:29:59
Calculating base
Updating mirror
Basing run on 7.1.0-1915 2c170886c
Updating tree for run 15.12.2021-18.29
gocb is at 635fa6c, changes since last good build: none
libcouchbase is at f6822fc, changes since last good build: none
indexing is at f9f16c4, changes since last good build: 
 f9f16c4b Merge remote-tracking branch 'couchbase/unstable' into HEAD
 a44a8bca MB-48897: Make plasma in-memory compression algorithm configurable
 e778b249 MB-50058 Skip vbuuid check if target seqno is '0' during snapshot merge
 1d2fda80 MB-50006 Code Refactor - handleIndexMergeSnapshot for easy lock/unlock
 08563949 MB-50008 disable OSO mode for magma bucket
kv_engine is at f27b040, changes since last good build: 
 f27b0402f Merge "Merge remote-tracking branch 'couchbase/cheshire-cat' into 'couchbase/master'"
 1717560c1 Merge remote-tracking branch 'couchbase/cheshire-cat' into 'couchbase/master'
 8eb4a46ab Merge commit '515dce7f4' into couchbase/cheshire-cat
 4448a795b Merge commit '7131d278d' into couchbase/cheshire-cat
 c3e8c4965 Merge commit '08f5eb977' into couchbase/cheshire-cat
 144502e52 Merge commit 'e6fe66924' into couchbase/cheshire-cat
 37ebad9ce Merge commit '3010db473' into couchbase/cheshire-cat
 515dce7f4 MB-47318: [BP] Save the 'pristine' message size when consumer buffers
 cc297d27a Merge commit 'caac4630a' into couchbase/cheshire-cat
 549e3ccf2 Merge commit '75d6f9548' into 'couchbase/cheshire-cat'
 1bc94f46b Merge commit '5786728a' into couchbase/cheshire-cat
 7131d278d MB-47318: [BP] Push UpdateFlowControl through to PassiveStream
 645f02e17 MB-47318: [BP] Add disabled tests for the issue
 08f5eb977 MB-47318: [BP] Add a make_wire_encoded_string factory method in xattr utils
 e6fe66924 MB-47318: [BP] Add DcpConsumer related client connection methods
 3010db473 MB-47318: [BP] Add PassiveStream always buffered mode
 caac4630a MB-47318: [BP] Add DcpControl to consumer and allow flow control override
 75d6f9548 MB-47318: [BP] Add DocKey::makeWireEncodedString
 5786728a3 MB-47318: Refactor: Move dcp_snapshot_marker_codec.h/cc
 9db024b87 Merge remote-tracking branch 'couchbase/mad-hatter'
 4a502aeb2 MB-49323: Allow reconfig of interfaces having same host and port
 5d7fd7fb0 MB-49321: Allow reconfig of hostname for interfaces
 b5fa131c0 MB-34280: Set max DCP name to 200 characters
ns_server is at 2e4e840, changes since last good build: 
 2e4e84074 MB-49969: use Inline Array Annotation to inject AngularJS services
 2326815fc MB-49969: check isGroupsAvailable flag along with server_groups!read
 624fcc384 MB-49969: setFormGroup should be called first
 607d91e4a MB-48738: use SET instead of LIST to set array item
 32f336c7a MB-49665: manually set xdcr play/pause spinner
 2a8866fdc MB-49665: erl: change tasks hash if xdcr max_vbreps is changed
 43239b54e MB-49813: remove cluster_run_ui_plugins folder
couchstore is at 6eea212, changes since last good build: none
couchdb is at fca2e6b, changes since last good build: none
testrunner is at c1e9d9b, changes since last good build: 
 c1e9d9b2d support for multiple comp for sub comp regex
Switching eventing to unstable
eventing is at 589c223, changes since last good build: 
 589c223d MB-49969: use Inline Array Annotation to inject AngularJS services
 e5c45d0b MB-49547: Check bucket instance to get kv node for dcp request
Switching eventing-ee to unstable
eventing-ee is at 3ae49c5, changes since last good build: none
Pulling in uncommitted change 160397 at refs/changes/97/160397/3
Total 23 (delta 15), reused 17 (delta 15)
[unstable f5259966] MB-100 : Do not fail n1ql timeout test if we are catching timeout exceptions successfully
 Author: abhijpes 
 Date: Mon Aug 30 08:24:38 2021 +0530
 1 file changed, 2 insertions(+), 2 deletions(-)
Pulling in uncommitted change 163449 at refs/changes/49/163449/1
Total 48 (delta 27), reused 39 (delta 27)
[unstable 88b9751e] MB-100: collect data when test fails
 Author: AnkitPrabhu 
 Date: Tue Oct 12 14:02:31 2021 +0530
 1 file changed, 2 insertions(+)
Pulling in uncommitted change 165976 at refs/changes/76/165976/17
Total 4 (delta 3), reused 4 (delta 3)
[unstable 58e33e5a] MB-48538: Add new audit ids for different events
 Author: AnkitPrabhu 
 Date: Tue Nov 23 14:29:10 2021 +0530
 1 file changed, 337 insertions(+), 61 deletions(-)
Pulling in uncommitted change 166698 at refs/changes/98/166698/3
Total 4 (delta 3), reused 4 (delta 3)
[unstable 11f0c936] MB-48334 : Fail rebalance unless all functions deployed on current encryption level
 Author: abhijpes 
 Date: Wed Dec 1 14:51:29 2021 +0530
 1 file changed, 9 insertions(+)
Pulling in uncommitted change 166812 at refs/changes/12/166812/6
Total 16 (delta 12), reused 16 (delta 12)
[unstable 8c653844] MB-49547: Refresh bucket info from streaming endpoint instance
 Author: AnkitPrabhu 
 Date: Thu Dec 2 13:55:55 2021 +0530
 7 files changed, 147 insertions(+), 235 deletions(-)
Pulling in uncommitted change 166999 at refs/changes/99/166999/11
Total 49 (delta 40), reused 49 (delta 40)
[unstable 741ff40c] MB-46304: Add functionality to disable curl feature
 Author: AnkitPrabhu 
 Date: Mon Dec 6 13:26:12 2021 +0530
 21 files changed, 193 insertions(+), 31 deletions(-)
Pulling in uncommitted change 167196 at refs/changes/96/167196/7
Total 59 (delta 49), reused 59 (delta 49)
[unstable 5ab5addf] MB-49720: Store failover log in checkpoint blob
 Author: AnkitPrabhu 
 Date: Wed Dec 8 15:16:22 2021 +0530
 5 files changed, 147 insertions(+), 72 deletions(-)
Pulling in uncommitted change 167542 at refs/changes/42/167542/5
Total 74 (delta 62), reused 74 (delta 62)
[unstable 74c9adf3] MB-49570: Start dcp stream for a single collection id
 Author: AnkitPrabhu 
 Date: Tue Dec 14 14:27:25 2021 +0530
 7 files changed, 52 insertions(+), 29 deletions(-)
Pulling in uncommitted change 167544 at refs/changes/44/167544/4
Total 82 (delta 68), reused 82 (delta 68)
[unstable e10c3c7f] MB-49570: Add backlog stats on ui
 Author: AnkitPrabhu 
 Date: Tue Dec 14 14:28:01 2021 +0530
 2 files changed, 18 insertions(+), 1 deletion(-)
Pulling in uncommitted change 167183 at refs/changes/83/167183/4
Total 5 (delta 2), reused 5 (delta 2)
[unstable 92d360f] MB-46304: Throw runtime exception when curl feature is disabled
 Author: AnkitPrabhu 
 Date: Wed Dec 8 10:39:19 2021 +0530
 1 file changed, 9 insertions(+), 3 deletions(-)
Building cmakefiles and deps
Building main product

Testing

Started testing at 2021/12/15 18:45:50
Testing mode: rebalance,n1ql,functional
Starting server
Starting server: attempt 1

Functional tests

2021/12/15 18:47:22 Started functional
2021/12/15 18:47:30 Adding node: https://127.0.0.1:19001 with role: kv,index,n1ql to the cluster
2021/12/15 18:47:36 Error in adding nodes : , response: {"otpNode":"n_1@127.0.0.1"}
2021/12/15 18:47:36 Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/15 18:47:42 Removing node(s): [] from the cluster
2021/12/15 18:47:47 Rebalance progress: 0
2021/12/15 18:47:52 Rebalance progress: 1.157407407407407
2021/12/15 18:47:57 Rebalance progress: 5.787037037037037
2021/12/15 18:48:02 Rebalance progress: 9.25925925925926
2021/12/15 18:48:07 Rebalance progress: 13.7622974537037
2021/12/15 18:48:12 Rebalance progress: 18.51851851851852
2021/12/15 18:48:17 Rebalance progress: 21.71947337962963
2021/12/15 18:48:22 Rebalance progress: 26.09592013888889
2021/12/15 18:48:27 Rebalance progress: 100
=== RUN   TestAdvancedGetOps
2021/12/15 18:48:32 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/15 18:48:47 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/15 18:49:02 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/15 18:49:17 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/15 18:49:32 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/15 18:49:47 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/15 18:50:02 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/15 18:50:17 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/15 18:50:32 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/15 18:50:47 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/15 18:51:02 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/15 18:51:17 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/15 18:51:33 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/15 18:51:48 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/15 18:52:03 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/15 18:52:18 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/15 18:52:33 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/15 18:52:48 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/15 18:53:03 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/15 18:53:18 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/15 18:53:33 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/15 18:53:48 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/15 18:54:03 Deploying app: TestAdvancedGetOps. Item count src bucket: 0 dst bucket: 0
2021/12/15 18:54:04 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedGetOps' in metakv",
  "warnings": null
 }
}
2021/12/15 18:54:09 Waiting for app: TestAdvancedGetOps to get deployed
2021/12/15 18:54:14 Waiting for app: TestAdvancedGetOps to get deployed
2021/12/15 18:54:19 Waiting for app: TestAdvancedGetOps to get deployed
2021/12/15 18:54:24 Waiting for app: TestAdvancedGetOps to get deployed
2021/12/15 18:54:24 App: TestAdvancedGetOps got deployed
2021/12/15 18:54:24 Starting bucket ops to source bucket
2021/12/15 18:54:29 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/15 18:54:34 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/15 18:54:34 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/15 18:54:34 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 18:54:34 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/15 18:54:34 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 18:54:39 Waiting for app: TestAdvancedGetOps to get un-deployed
2021/12/15 18:54:44 Waiting for app: TestAdvancedGetOps to get un-deployed
2021/12/15 18:54:49 Waiting for app: TestAdvancedGetOps to get un-deployed
2021/12/15 18:54:54 Waiting for app: TestAdvancedGetOps to get un-deployed
2021/12/15 18:54:54 App: TestAdvancedGetOps got un-deployed
2021/12/15 18:54:54 No eventing-con process running
2021/12/15 18:54:54 Function: TestAdvancedGetOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedGetOps deleting in the background"
}
2021/12/15 18:54:56 Context: Bucket flush, response code: 200 dump: 
2021/12/15 18:55:06 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/15 18:55:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 18:55:08 Context: Bucket flush, response code: 200 dump: 
2021/12/15 18:55:18 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/15 18:55:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedGetOps (405.12s)
=== RUN   TestAdvancedInsertOps
2021/12/15 18:55:18 Deploying app: TestAdvancedInsertOps. Item count src bucket: 0 dst bucket: 0
2021/12/15 18:55:19 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedInsertOps' in metakv",
  "warnings": null
 }
}
2021/12/15 18:55:24 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/15 18:55:29 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/15 18:55:34 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/15 18:55:39 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/15 18:55:39 App: TestAdvancedInsertOps got deployed
2021/12/15 18:55:39 Starting bucket ops to source bucket
2021/12/15 18:55:45 Waiting for dst bucket item count to get to: 1024 curr count: 0
2021/12/15 18:55:50 Waiting for dst bucket item count to get to: 1024 curr count: 299
2021/12/15 18:55:55 Waiting for dst bucket item count to get to: 1024 curr count: 811
2021/12/15 18:55:55 src & dst bucket item count matched up. src bucket count: 1024 dst bucket count: 1024
2021/12/15 18:55:55 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 18:55:55 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/15 18:55:55 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 18:56:00 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/15 18:56:05 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/15 18:56:10 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/15 18:56:15 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/15 18:56:15 App: TestAdvancedInsertOps got un-deployed
2021/12/15 18:56:15 No eventing-con process running
2021/12/15 18:56:15 Function: TestAdvancedInsertOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedInsertOps deleting in the background"
}
2021/12/15 18:56:17 Context: Bucket flush, response code: 200 dump: 
2021/12/15 18:56:27 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/15 18:56:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 18:56:28 Context: Bucket flush, response code: 200 dump: 
2021/12/15 18:56:38 Waiting for dst bucket item count to get to: 0 curr count: 851
2021/12/15 18:56:43 Waiting for dst bucket item count to get to: 0 curr count: 339
2021/12/15 18:56:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 18:56:43 Testing insert operation on source bucket
2021/12/15 18:56:43 Deploying app: TestAdvancedInsertOps. Item count src bucket: 0 dst bucket: 0
2021/12/15 18:56:45 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedInsertOps' in metakv",
  "warnings": null
 }
}
2021/12/15 18:56:50 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/15 18:56:55 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/15 18:57:00 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/15 18:57:05 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/15 18:57:05 App: TestAdvancedInsertOps got deployed
2021/12/15 18:57:05 Starting bucket ops to source bucket
2021/12/15 18:57:11 Waiting for dst bucket item count to get to: 2048 curr count: 0
2021/12/15 18:57:16 Waiting for dst bucket item count to get to: 2048 curr count: 1024
2021/12/15 18:57:16 src & dst bucket item count matched up. src bucket count: 2048 dst bucket count: 2048
2021/12/15 18:57:16 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 18:57:16 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/15 18:57:16 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 18:57:21 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/15 18:57:26 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/15 18:57:31 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/15 18:57:36 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/15 18:57:36 App: TestAdvancedInsertOps got un-deployed
2021/12/15 18:57:36 No eventing-con process running
2021/12/15 18:57:36 Function: TestAdvancedInsertOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedInsertOps deleting in the background"
}
2021/12/15 18:57:38 Context: Bucket flush, response code: 200 dump: 
2021/12/15 18:57:48 Waiting for dst bucket item count to get to: 0 curr count: 1043
2021/12/15 18:57:53 Waiting for dst bucket item count to get to: 0 curr count: 19
2021/12/15 18:57:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 18:57:54 Context: Bucket flush, response code: 200 dump: 
2021/12/15 18:57:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedInsertOps (161.86s)
=== RUN   TestAdvancedUpsertOps
2021/12/15 18:57:59 Starting bucket ops to source bucket
2021/12/15 18:58:00 Deploying app: TestAdvancedUpsertOps. Item count src bucket: 0 dst bucket: 0
2021/12/15 18:58:01 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedUpsertOps' in metakv",
  "warnings": null
 }
}
2021/12/15 18:58:06 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/15 18:58:11 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/15 18:58:16 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/15 18:58:21 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/15 18:58:21 App: TestAdvancedUpsertOps got deployed
2021/12/15 18:58:21 Starting bucket ops to source bucket
2021/12/15 18:58:27 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/15 18:58:32 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/15 18:58:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 18:58:32 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 18:58:32 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/15 18:58:32 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 18:58:37 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/15 18:58:42 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/15 18:58:47 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/15 18:58:52 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/15 18:58:52 App: TestAdvancedUpsertOps got un-deployed
2021/12/15 18:58:52 No eventing-con process running
2021/12/15 18:58:52 Function: TestAdvancedUpsertOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedUpsertOps deleting in the background"
}
2021/12/15 18:58:54 Context: Bucket flush, response code: 200 dump: 
2021/12/15 18:59:04 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/15 18:59:09 Waiting for dst bucket item count to get to: 0 curr count: 512
2021/12/15 18:59:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 18:59:11 Context: Bucket flush, response code: 200 dump: 
2021/12/15 18:59:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 18:59:16 Testing upsert operation on source bucket
2021/12/15 18:59:16 Deploying app: TestAdvancedUpsertOps. Item count src bucket: 0 dst bucket: 0
2021/12/15 18:59:17 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedUpsertOps' in metakv",
  "warnings": null
 }
}
2021/12/15 18:59:22 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/15 18:59:27 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/15 18:59:32 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/15 18:59:37 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/15 18:59:37 App: TestAdvancedUpsertOps got deployed
2021/12/15 18:59:37 Starting bucket ops to source bucket
2021/12/15 18:59:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 18:59:38 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 18:59:38 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/15 18:59:38 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 18:59:43 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/15 18:59:48 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/15 18:59:53 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/15 18:59:58 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/15 18:59:58 App: TestAdvancedUpsertOps got un-deployed
2021/12/15 18:59:58 No eventing-con process running
2021/12/15 18:59:58 Function: TestAdvancedUpsertOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedUpsertOps deleting in the background"
}
2021/12/15 19:00:00 Context: Bucket flush, response code: 200 dump: 
2021/12/15 19:00:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 19:00:07 Context: Bucket flush, response code: 200 dump: 
2021/12/15 19:00:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedUpsertOps (133.04s)
=== RUN   TestAdvancedReplaceOps
2021/12/15 19:00:12 Starting bucket ops to source bucket
2021/12/15 19:00:13 Deploying app: TestAdvancedReplaceOps. Item count src bucket: 0 dst bucket: 0
2021/12/15 19:00:15 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedReplaceOps' in metakv",
  "warnings": null
 }
}
2021/12/15 19:00:20 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/15 19:00:25 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/15 19:00:30 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/15 19:00:35 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/15 19:00:35 App: TestAdvancedReplaceOps got deployed
2021/12/15 19:00:35 Starting bucket ops to source bucket
2021/12/15 19:00:41 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/15 19:00:46 Waiting for dst bucket item count to get to: 0 curr count: 512
2021/12/15 19:00:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 19:00:46 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 19:00:46 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/15 19:00:46 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 19:00:51 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/15 19:00:56 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/15 19:01:01 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/15 19:01:06 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/15 19:01:06 App: TestAdvancedReplaceOps got un-deployed
2021/12/15 19:01:06 No eventing-con process running
2021/12/15 19:01:06 Function: TestAdvancedReplaceOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedReplaceOps deleting in the background"
}
2021/12/15 19:01:08 Context: Bucket flush, response code: 200 dump: 
2021/12/15 19:01:18 Waiting for dst bucket item count to get to: 0 curr count: 512
2021/12/15 19:01:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 19:01:19 Context: Bucket flush, response code: 200 dump: 
2021/12/15 19:01:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 19:01:24 Testing upsert operation on source bucket
2021/12/15 19:01:24 Deploying app: TestAdvancedReplaceOps. Item count src bucket: 0 dst bucket: 0
2021/12/15 19:01:26 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedReplaceOps' in metakv",
  "warnings": null
 }
}
2021/12/15 19:01:31 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/15 19:01:36 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/15 19:01:41 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/15 19:01:46 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/15 19:01:46 App: TestAdvancedReplaceOps got deployed
2021/12/15 19:01:46 Starting bucket ops to source bucket
2021/12/15 19:01:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 19:01:47 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 19:01:47 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/15 19:01:47 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 19:01:52 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/15 19:01:57 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/15 19:02:02 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/15 19:02:07 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/15 19:02:07 App: TestAdvancedReplaceOps got un-deployed
2021/12/15 19:02:07 No eventing-con process running
2021/12/15 19:02:07 Function: TestAdvancedReplaceOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedReplaceOps deleting in the background"
}
2021/12/15 19:02:09 Context: Bucket flush, response code: 200 dump: 
2021/12/15 19:02:19 Waiting for dst bucket item count to get to: 0 curr count: 247
2021/12/15 19:02:24 Waiting for dst bucket item count to get to: 0 curr count: 123
2021/12/15 19:02:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 19:02:25 Context: Bucket flush, response code: 200 dump: 
2021/12/15 19:02:30 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedReplaceOps (137.86s)
=== RUN   TestAdvancedDeleteOps
2021/12/15 19:02:30 Starting bucket ops to source bucket
2021/12/15 19:02:30 Deploying app: TestAdvancedDeleteOps. Item count src bucket: 0 dst bucket: 0
2021/12/15 19:02:32 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedDeleteOps' in metakv",
  "warnings": null
 }
}
2021/12/15 19:02:37 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/15 19:02:42 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/15 19:02:47 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/15 19:02:52 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/15 19:02:52 App: TestAdvancedDeleteOps got deployed
2021/12/15 19:02:52 Starting bucket ops to source bucket
2021/12/15 19:02:57 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/15 19:03:02 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/15 19:03:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 19:03:02 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 19:03:02 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/15 19:03:02 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 19:03:07 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/15 19:03:12 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/15 19:03:17 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/15 19:03:22 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/15 19:03:22 App: TestAdvancedDeleteOps got un-deployed
2021/12/15 19:03:22 No eventing-con process running
2021/12/15 19:03:22 Function: TestAdvancedDeleteOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedDeleteOps deleting in the background"
}
2021/12/15 19:03:24 Context: Bucket flush, response code: 200 dump: 
2021/12/15 19:03:34 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/15 19:03:39 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/15 19:03:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 19:03:40 Context: Bucket flush, response code: 200 dump: 
2021/12/15 19:03:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 19:03:45 Testing delete operation on source bucket
2021/12/15 19:03:45 Deploying app: TestAdvancedDeleteOps. Item count src bucket: 0 dst bucket: 0
2021/12/15 19:03:47 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedDeleteOps' in metakv",
  "warnings": null
 }
}
2021/12/15 19:03:52 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/15 19:03:57 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/15 19:04:02 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/15 19:04:07 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/15 19:04:07 App: TestAdvancedDeleteOps got deployed
2021/12/15 19:04:07 Starting bucket ops to source bucket
2021/12/15 19:04:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 19:04:07 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 19:04:07 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/15 19:04:07 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 19:04:12 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/15 19:04:17 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/15 19:04:22 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/15 19:04:27 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/15 19:04:27 App: TestAdvancedDeleteOps got un-deployed
2021/12/15 19:04:27 No eventing-con process running
2021/12/15 19:04:27 Function: TestAdvancedDeleteOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedDeleteOps deleting in the background"
}
2021/12/15 19:04:29 Context: Bucket flush, response code: 200 dump: 
2021/12/15 19:04:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 19:04:36 Context: Bucket flush, response code: 200 dump: 
2021/12/15 19:04:41 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedDeleteOps (130.79s)
=== RUN   TestEnoentAdvancedGet
2021/12/15 19:04:41 Deploying app: TestEnoentAdvancedGet. Item count src bucket: 0 dst bucket: 0
2021/12/15 19:04:42 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedGet' in metakv",
  "warnings": null
 }
}
2021/12/15 19:04:47 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/15 19:04:52 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/15 19:04:57 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/15 19:05:02 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/15 19:05:02 App: TestEnoentAdvancedGet got deployed
2021/12/15 19:05:02 Starting bucket ops to source bucket
2021/12/15 19:05:07 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/15 19:05:12 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/15 19:05:12 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/15 19:05:12 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 19:05:12 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/15 19:05:13 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 19:05:18 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/15 19:05:23 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/15 19:05:28 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/15 19:05:33 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/15 19:05:33 App: TestEnoentAdvancedGet got un-deployed
2021/12/15 19:05:33 No eventing-con process running
2021/12/15 19:05:33 Function: TestEnoentAdvancedGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedGet deleting in the background"
}
2021/12/15 19:05:34 Context: Bucket flush, response code: 200 dump: 
2021/12/15 19:05:44 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/15 19:05:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 19:05:46 Context: Bucket flush, response code: 200 dump: 
2021/12/15 19:05:56 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/15 19:05:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 19:05:56 Testing get enoent operation on source bucket
2021/12/15 19:05:56 Deploying app: TestEnoentAdvancedGet. Item count src bucket: 0 dst bucket: 0
2021/12/15 19:05:57 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedGet' in metakv",
  "warnings": null
 }
}
2021/12/15 19:06:02 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/15 19:06:07 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/15 19:06:12 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/15 19:06:17 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/15 19:06:17 App: TestEnoentAdvancedGet got deployed
2021/12/15 19:06:17 Starting bucket ops to source bucket
2021/12/15 19:06:22 Waiting for dst bucket item count to get to: 200 curr count: 0
2021/12/15 19:06:27 Waiting for dst bucket item count to get to: 200 curr count: 100
2021/12/15 19:06:32 Waiting for dst bucket item count to get to: 200 curr count: 100
2021/12/15 19:06:32 src & dst bucket item count matched up. src bucket count: 200 dst bucket count: 200
2021/12/15 19:06:33 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 19:06:33 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/15 19:06:33 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 19:06:38 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/15 19:06:43 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/15 19:06:48 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/15 19:06:53 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/15 19:06:53 App: TestEnoentAdvancedGet got un-deployed
2021/12/15 19:06:53 No eventing-con process running
2021/12/15 19:06:53 Function: TestEnoentAdvancedGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedGet deleting in the background"
}
2021/12/15 19:06:54 Context: Bucket flush, response code: 200 dump: 
2021/12/15 19:07:04 Waiting for dst bucket item count to get to: 0 curr count: 200
2021/12/15 19:07:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 19:07:06 Context: Bucket flush, response code: 200 dump: 
2021/12/15 19:07:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEnoentAdvancedGet (149.99s)
=== RUN   TestEnoentAdvancedReplace
2021/12/15 19:07:11 Deploying app: TestEnoentAdvancedReplace. Item count src bucket: 0 dst bucket: 0
2021/12/15 19:07:12 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedReplace' in metakv",
  "warnings": null
 }
}
2021/12/15 19:07:17 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/15 19:07:22 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/15 19:07:27 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/15 19:07:32 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/15 19:07:32 App: TestEnoentAdvancedReplace got deployed
2021/12/15 19:07:32 Starting bucket ops to source bucket
2021/12/15 19:07:37 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/15 19:07:42 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/15 19:07:42 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/15 19:07:42 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 19:07:42 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/15 19:07:43 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 19:07:48 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/15 19:07:53 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/15 19:07:58 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/15 19:08:03 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/15 19:08:03 App: TestEnoentAdvancedReplace got un-deployed
2021/12/15 19:08:03 No eventing-con process running
2021/12/15 19:08:03 Function: TestEnoentAdvancedReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedReplace deleting in the background"
}
2021/12/15 19:08:04 Context: Bucket flush, response code: 200 dump: 
2021/12/15 19:08:14 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/15 19:08:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 19:08:16 Context: Bucket flush, response code: 200 dump: 
2021/12/15 19:08:26 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/15 19:08:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 19:08:26 Testing get enoent operation on source bucket
2021/12/15 19:08:26 Deploying app: TestEnoentAdvancedReplace. Item count src bucket: 0 dst bucket: 0
2021/12/15 19:08:27 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedReplace' in metakv",
  "warnings": null
 }
}
2021/12/15 19:08:32 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/15 19:08:37 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/15 19:08:42 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/15 19:08:47 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/15 19:08:47 App: TestEnoentAdvancedReplace got deployed
2021/12/15 19:08:47 Starting bucket ops to source bucket
2021/12/15 19:08:53 Waiting for dst bucket item count to get to: 200 curr count: 0
2021/12/15 19:08:58 Waiting for dst bucket item count to get to: 200 curr count: 100
2021/12/15 19:09:03 Waiting for dst bucket item count to get to: 200 curr count: 100
2021/12/15 19:09:03 src & dst bucket item count matched up. src bucket count: 200 dst bucket count: 200
2021/12/15 19:09:03 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 19:09:03 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/15 19:09:03 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 19:09:08 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/15 19:09:13 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/15 19:09:18 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/15 19:09:23 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/15 19:09:23 App: TestEnoentAdvancedReplace got un-deployed
2021/12/15 19:09:23 No eventing-con process running
2021/12/15 19:09:23 Function: TestEnoentAdvancedReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedReplace deleting in the background"
}
2021/12/15 19:09:24 Context: Bucket flush, response code: 200 dump: 
2021/12/15 19:09:34 Waiting for dst bucket item count to get to: 0 curr count: 200
2021/12/15 19:09:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 19:09:36 Context: Bucket flush, response code: 200 dump: 
2021/12/15 19:09:41 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEnoentAdvancedReplace (149.99s)
=== RUN   TestEnoentAdvancedDelete
2021/12/15 19:09:41 Deploying app: TestEnoentAdvancedDelete. Item count src bucket: 0 dst bucket: 0
2021/12/15 19:09:42 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedDelete' in metakv",
  "warnings": null
 }
}
2021/12/15 19:09:47 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/15 19:09:52 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/15 19:09:57 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/15 19:10:02 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/15 19:10:02 App: TestEnoentAdvancedDelete got deployed
2021/12/15 19:10:02 Starting bucket ops to source bucket
2021/12/15 19:10:07 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/15 19:10:12 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/15 19:10:12 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/15 19:10:12 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 19:10:12 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/15 19:10:12 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 19:10:17 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/15 19:10:23 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/15 19:10:28 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/15 19:10:33 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/15 19:10:33 App: TestEnoentAdvancedDelete got un-deployed
2021/12/15 19:10:33 No eventing-con process running
2021/12/15 19:10:33 Function: TestEnoentAdvancedDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedDelete deleting in the background"
}
2021/12/15 19:10:34 Context: Bucket flush, response code: 200 dump: 
2021/12/15 19:10:44 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/15 19:10:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 19:10:46 Context: Bucket flush, response code: 200 dump: 
2021/12/15 19:10:56 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/15 19:10:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 19:10:56 Testing get enoent operation on source bucket
2021/12/15 19:10:56 Deploying app: TestEnoentAdvancedDelete. Item count src bucket: 0 dst bucket: 0
2021/12/15 19:10:57 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedDelete' in metakv",
  "warnings": null
 }
}
2021/12/15 19:11:02 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/15 19:11:07 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/15 19:11:12 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/15 19:11:17 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/15 19:11:17 App: TestEnoentAdvancedDelete got deployed
2021/12/15 19:11:17 Starting bucket ops to source bucket
2021/12/15 19:11:22 Waiting for dst bucket item count to get to: 200 curr count: 0
2021/12/15 19:11:27 Waiting for dst bucket item count to get to: 200 curr count: 100
2021/12/15 19:11:32 Waiting for dst bucket item count to get to: 200 curr count: 100
2021/12/15 19:11:32 src & dst bucket item count matched up. src bucket count: 200 dst bucket count: 200
2021/12/15 19:11:32 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 19:11:33 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/15 19:11:33 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 19:11:38 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/15 19:11:43 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/15 19:11:48 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/15 19:11:53 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/15 19:11:53 App: TestEnoentAdvancedDelete got un-deployed
2021/12/15 19:11:53 No eventing-con process running
2021/12/15 19:11:53 Function: TestEnoentAdvancedDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedDelete deleting in the background"
}
2021/12/15 19:11:54 Context: Bucket flush, response code: 200 dump: 
2021/12/15 19:12:04 Waiting for dst bucket item count to get to: 0 curr count: 200
2021/12/15 19:12:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 19:12:06 Context: Bucket flush, response code: 200 dump: 
2021/12/15 19:12:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEnoentAdvancedDelete (149.98s)
=== RUN   TestAdvancedInsertKeyExist
2021/12/15 19:12:11 Deploying app: TestAdvancedInsertKeyExist. Item count src bucket: 0 dst bucket: 0
2021/12/15 19:12:12 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedInsertKeyExist' in metakv",
  "warnings": null
 }
}
2021/12/15 19:12:17 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/15 19:12:22 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/15 19:12:27 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/15 19:12:32 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/15 19:12:32 App: TestAdvancedInsertKeyExist got deployed
2021/12/15 19:12:32 Starting bucket ops to source bucket
2021/12/15 19:12:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 19:12:33 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 19:12:33 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/15 19:12:33 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 19:12:38 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/15 19:12:43 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/15 19:12:48 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/15 19:12:53 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/15 19:12:53 App: TestAdvancedInsertKeyExist got un-deployed
2021/12/15 19:12:53 No eventing-con process running
2021/12/15 19:12:53 Function: TestAdvancedInsertKeyExist delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedInsertKeyExist deleting in the background"
}
2021/12/15 19:12:54 Context: Bucket flush, response code: 200 dump: 
2021/12/15 19:12:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 19:13:02 Context: Bucket flush, response code: 200 dump: 
2021/12/15 19:13:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 19:13:07 Pumping document in the destination bucket
2021/12/15 19:13:07 Starting bucket ops to source bucket
2021/12/15 19:13:07 Deploying app: TestAdvancedInsertKeyExist. Item count src bucket: 0 dst bucket: 0
2021/12/15 19:13:08 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedInsertKeyExist' in metakv",
  "warnings": null
 }
}
2021/12/15 19:13:13 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/15 19:13:18 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/15 19:13:23 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/15 19:13:28 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/15 19:13:28 App: TestAdvancedInsertKeyExist got deployed
2021/12/15 19:13:28 Starting bucket ops to source bucket
2021/12/15 19:13:33 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/15 19:13:38 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/15 19:13:43 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/15 19:13:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 19:13:43 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 19:13:43 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/15 19:13:43 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 19:13:48 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/15 19:13:53 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/15 19:13:58 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/15 19:14:03 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/15 19:14:03 App: TestAdvancedInsertKeyExist got un-deployed
2021/12/15 19:14:03 No eventing-con process running
2021/12/15 19:14:03 Function: TestAdvancedInsertKeyExist delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedInsertKeyExist deleting in the background"
}
2021/12/15 19:14:05 Context: Bucket flush, response code: 200 dump: 
2021/12/15 19:14:15 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/15 19:14:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 19:14:17 Context: Bucket flush, response code: 200 dump: 
2021/12/15 19:14:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedInsertKeyExist (130.75s)
=== RUN   TestExipryGet
2021/12/15 19:14:22 Deploying app: TestExipryGet. Item count src bucket: 0 dst bucket: 0
2021/12/15 19:14:23 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryGet' in metakv",
  "warnings": null
 }
}
2021/12/15 19:14:28 Waiting for app: TestExipryGet to get deployed
2021/12/15 19:14:33 Waiting for app: TestExipryGet to get deployed
2021/12/15 19:14:38 Waiting for app: TestExipryGet to get deployed
2021/12/15 19:14:43 Waiting for app: TestExipryGet to get deployed
2021/12/15 19:14:43 App: TestExipryGet got deployed
2021/12/15 19:14:43 Starting bucket ops to source bucket
2021/12/15 19:15:28 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/15 19:15:33 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/15 19:15:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 19:15:33 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 19:15:34 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/15 19:15:34 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 19:15:39 Waiting for app: TestExipryGet to get un-deployed
2021/12/15 19:15:44 Waiting for app: TestExipryGet to get un-deployed
2021/12/15 19:15:49 Waiting for app: TestExipryGet to get un-deployed
2021/12/15 19:15:54 Waiting for app: TestExipryGet to get un-deployed
2021/12/15 19:15:54 App: TestExipryGet got un-deployed
2021/12/15 19:15:54 No eventing-con process running
2021/12/15 19:15:54 Function: TestExipryGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryGet deleting in the background"
}
2021/12/15 19:15:55 Context: Bucket flush, response code: 200 dump: 
2021/12/15 19:16:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 19:16:03 Context: Bucket flush, response code: 200 dump: 
2021/12/15 19:16:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 19:16:08 Pumping document in the destination bucket for expiry get
2021/12/15 19:16:08 Starting bucket ops to source bucket
2021/12/15 19:16:08 Deploying app: TestExipryGet. Item count src bucket: 0 dst bucket: 0
2021/12/15 19:16:09 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryGet' in metakv",
  "warnings": null
 }
}
2021/12/15 19:16:14 Waiting for app: TestExipryGet to get deployed
2021/12/15 19:16:19 Waiting for app: TestExipryGet to get deployed
2021/12/15 19:16:24 Waiting for app: TestExipryGet to get deployed
2021/12/15 19:16:29 Waiting for app: TestExipryGet to get deployed
2021/12/15 19:16:29 App: TestExipryGet got deployed
2021/12/15 19:16:29 Starting bucket ops to source bucket
2021/12/15 19:17:14 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/15 19:17:19 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/15 19:17:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 19:17:19 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 19:17:19 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/15 19:17:19 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 19:17:24 Waiting for app: TestExipryGet to get un-deployed
2021/12/15 19:17:29 Waiting for app: TestExipryGet to get un-deployed
2021/12/15 19:17:34 Waiting for app: TestExipryGet to get un-deployed
2021/12/15 19:17:39 Waiting for app: TestExipryGet to get un-deployed
2021/12/15 19:17:39 App: TestExipryGet got un-deployed
2021/12/15 19:17:39 No eventing-con process running
2021/12/15 19:17:39 Function: TestExipryGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryGet deleting in the background"
}
2021/12/15 19:17:41 Context: Bucket flush, response code: 200 dump: 
2021/12/15 19:17:51 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/15 19:17:56 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/15 19:17:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 19:17:58 Context: Bucket flush, response code: 200 dump: 
2021/12/15 19:18:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestExipryGet (221.03s)
=== RUN   TestExipryInsert
2021/12/15 19:18:03 Deploying app: TestExipryInsert. Item count src bucket: 0 dst bucket: 0
2021/12/15 19:18:04 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryInsert' in metakv",
  "warnings": null
 }
}
2021/12/15 19:18:09 Waiting for app: TestExipryInsert to get deployed
2021/12/15 19:18:14 Waiting for app: TestExipryInsert to get deployed
2021/12/15 19:18:19 Waiting for app: TestExipryInsert to get deployed
2021/12/15 19:18:24 Waiting for app: TestExipryInsert to get deployed
2021/12/15 19:18:24 App: TestExipryInsert got deployed
2021/12/15 19:18:24 Starting bucket ops to source bucket
2021/12/15 19:19:09 Waiting for dst bucket item count to get to: 0 curr count: 200
2021/12/15 19:19:14 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/15 19:19:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 19:19:14 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 19:19:14 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/15 19:19:14 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 19:19:19 Waiting for app: TestExipryInsert to get un-deployed
2021/12/15 19:19:24 Waiting for app: TestExipryInsert to get un-deployed
2021/12/15 19:19:29 Waiting for app: TestExipryInsert to get un-deployed
2021/12/15 19:19:34 Waiting for app: TestExipryInsert to get un-deployed
2021/12/15 19:19:34 App: TestExipryInsert got un-deployed
2021/12/15 19:19:35 No eventing-con process running
2021/12/15 19:19:35 Function: TestExipryInsert delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryInsert deleting in the background"
}
2021/12/15 19:19:36 Context: Bucket flush, response code: 200 dump: 
2021/12/15 19:19:41 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 19:19:44 Context: Bucket flush, response code: 200 dump: 
2021/12/15 19:19:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 19:19:49 Testing on the destination bucket for expiry insert
2021/12/15 19:19:49 Deploying app: TestExipryInsert. Item count src bucket: 0 dst bucket: 0
2021/12/15 19:19:50 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryInsert' in metakv",
  "warnings": null
 }
}
2021/12/15 19:19:55 Waiting for app: TestExipryInsert to get deployed
2021/12/15 19:20:00 Waiting for app: TestExipryInsert to get deployed
2021/12/15 19:20:05 Waiting for app: TestExipryInsert to get deployed
2021/12/15 19:20:10 Waiting for app: TestExipryInsert to get deployed
2021/12/15 19:20:10 App: TestExipryInsert got deployed
2021/12/15 19:20:10 Starting bucket ops to source bucket
2021/12/15 19:20:55 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/15 19:21:00 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/15 19:21:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 19:21:00 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 19:21:00 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/15 19:21:00 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 19:21:05 Waiting for app: TestExipryInsert to get un-deployed
2021/12/15 19:21:10 Waiting for app: TestExipryInsert to get un-deployed
2021/12/15 19:21:15 Waiting for app: TestExipryInsert to get un-deployed
2021/12/15 19:21:20 Waiting for app: TestExipryInsert to get un-deployed
2021/12/15 19:21:20 App: TestExipryInsert got un-deployed
2021/12/15 19:21:20 No eventing-con process running
2021/12/15 19:21:20 Function: TestExipryInsert delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryInsert deleting in the background"
}
2021/12/15 19:21:22 Context: Bucket flush, response code: 200 dump: 
2021/12/15 19:21:32 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/15 19:21:37 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/15 19:21:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 19:21:39 Context: Bucket flush, response code: 200 dump: 
2021/12/15 19:21:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestExipryInsert (220.75s)
=== RUN   TestExipryReplace
2021/12/15 19:21:44 Deploying app: TestExipryReplace. Item count src bucket: 0 dst bucket: 0
2021/12/15 19:21:45 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryReplace' in metakv",
  "warnings": null
 }
}
2021/12/15 19:21:50 Waiting for app: TestExipryReplace to get deployed
2021/12/15 19:21:55 Waiting for app: TestExipryReplace to get deployed
2021/12/15 19:22:00 Waiting for app: TestExipryReplace to get deployed
2021/12/15 19:22:05 Waiting for app: TestExipryReplace to get deployed
2021/12/15 19:22:05 App: TestExipryReplace got deployed
2021/12/15 19:22:05 Starting bucket ops to source bucket
2021/12/15 19:22:50 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/15 19:22:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 19:22:50 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 19:22:50 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/15 19:22:50 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 19:22:55 Waiting for app: TestExipryReplace to get un-deployed
2021/12/15 19:23:00 Waiting for app: TestExipryReplace to get un-deployed
2021/12/15 19:23:05 Waiting for app: TestExipryReplace to get un-deployed
2021/12/15 19:23:10 Waiting for app: TestExipryReplace to get un-deployed
2021/12/15 19:23:10 App: TestExipryReplace got un-deployed
2021/12/15 19:23:10 No eventing-con process running
2021/12/15 19:23:10 Function: TestExipryReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryReplace deleting in the background"
}
2021/12/15 19:23:12 Context: Bucket flush, response code: 200 dump: 
2021/12/15 19:23:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 19:23:19 Context: Bucket flush, response code: 200 dump: 
2021/12/15 19:23:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 19:23:24 Testing on the destination bucket for expiry insert
2021/12/15 19:23:24 Deploying app: TestExipryReplace. Item count src bucket: 0 dst bucket: 0
2021/12/15 19:23:25 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryReplace' in metakv",
  "warnings": null
 }
}
2021/12/15 19:23:30 Waiting for app: TestExipryReplace to get deployed
2021/12/15 19:23:35 Waiting for app: TestExipryReplace to get deployed
2021/12/15 19:23:40 Waiting for app: TestExipryReplace to get deployed
2021/12/15 19:23:45 Waiting for app: TestExipryReplace to get deployed
2021/12/15 19:23:45 App: TestExipryReplace got deployed
2021/12/15 19:23:45 Starting bucket ops to source bucket
2021/12/15 19:24:31 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/15 19:24:31 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 19:24:31 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 19:24:31 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/15 19:24:31 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 19:24:36 Waiting for app: TestExipryReplace to get un-deployed
2021/12/15 19:24:41 Waiting for app: TestExipryReplace to get un-deployed
2021/12/15 19:24:46 Waiting for app: TestExipryReplace to get un-deployed
2021/12/15 19:24:51 Waiting for app: TestExipryReplace to get un-deployed
2021/12/15 19:24:51 App: TestExipryReplace got un-deployed
2021/12/15 19:24:51 No eventing-con process running
2021/12/15 19:24:51 Function: TestExipryReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryReplace deleting in the background"
}
2021/12/15 19:24:53 Context: Bucket flush, response code: 200 dump: 
2021/12/15 19:25:03 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/15 19:25:08 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/15 19:25:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 19:25:09 Context: Bucket flush, response code: 200 dump: 
2021/12/15 19:25:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestExipryReplace (210.64s)
=== RUN   TestExipryUpsert
2021/12/15 19:25:14 Deploying app: TestExipryUpsert. Item count src bucket: 0 dst bucket: 0
2021/12/15 19:25:15 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryUpsert' in metakv",
  "warnings": null
 }
}
2021/12/15 19:25:20 Waiting for app: TestExipryUpsert to get deployed
2021/12/15 19:25:25 Waiting for app: TestExipryUpsert to get deployed
2021/12/15 19:25:30 Waiting for app: TestExipryUpsert to get deployed
2021/12/15 19:25:35 Waiting for app: TestExipryUpsert to get deployed
2021/12/15 19:25:35 App: TestExipryUpsert got deployed
2021/12/15 19:25:35 Starting bucket ops to source bucket
2021/12/15 19:26:21 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/15 19:26:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 19:26:21 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 19:26:21 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/15 19:26:21 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 19:26:26 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/15 19:26:31 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/15 19:26:36 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/15 19:26:41 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/15 19:26:41 App: TestExipryUpsert got un-deployed
2021/12/15 19:26:41 No eventing-con process running
2021/12/15 19:26:41 Function: TestExipryUpsert delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryUpsert deleting in the background"
}
2021/12/15 19:26:43 Context: Bucket flush, response code: 200 dump: 
2021/12/15 19:26:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 19:26:50 Context: Bucket flush, response code: 200 dump: 
2021/12/15 19:26:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 19:26:55 Testing on the destination bucket for expiry insert
2021/12/15 19:26:55 Deploying app: TestExipryUpsert. Item count src bucket: 0 dst bucket: 0
2021/12/15 19:26:56 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryUpsert' in metakv",
  "warnings": null
 }
}
2021/12/15 19:27:01 Waiting for app: TestExipryUpsert to get deployed
2021/12/15 19:27:06 Waiting for app: TestExipryUpsert to get deployed
2021/12/15 19:27:11 Waiting for app: TestExipryUpsert to get deployed
2021/12/15 19:27:16 Waiting for app: TestExipryUpsert to get deployed
2021/12/15 19:27:16 App: TestExipryUpsert got deployed
2021/12/15 19:27:16 Starting bucket ops to source bucket
2021/12/15 19:28:01 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/15 19:28:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 19:28:01 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 19:28:01 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/15 19:28:01 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 19:28:06 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/15 19:28:11 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/15 19:28:16 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/15 19:28:21 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/15 19:28:21 App: TestExipryUpsert got un-deployed
2021/12/15 19:28:21 No eventing-con process running
2021/12/15 19:28:21 Function: TestExipryUpsert delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryUpsert deleting in the background"
}
2021/12/15 19:28:23 Context: Bucket flush, response code: 200 dump: 
2021/12/15 19:28:33 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/15 19:28:38 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/15 19:28:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 19:28:40 Context: Bucket flush, response code: 200 dump: 
2021/12/15 19:28:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestExipryUpsert (210.66s)
=== RUN   TestCasReplace
2021/12/15 19:28:45 Deploying app: TestCasReplace. Item count src bucket: 0 dst bucket: 0
2021/12/15 19:28:46 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCasReplace' in metakv",
  "warnings": null
 }
}
2021/12/15 19:28:51 Waiting for app: TestCasReplace to get deployed
2021/12/15 19:28:56 Waiting for app: TestCasReplace to get deployed
2021/12/15 19:29:01 Waiting for app: TestCasReplace to get deployed
2021/12/15 19:29:06 Waiting for app: TestCasReplace to get deployed
2021/12/15 19:29:06 App: TestCasReplace got deployed
2021/12/15 19:29:06 Starting bucket ops to source bucket
2021/12/15 19:29:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 19:29:06 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 19:29:06 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/15 19:29:06 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 19:29:11 Waiting for app: TestCasReplace to get un-deployed
2021/12/15 19:29:16 Waiting for app: TestCasReplace to get un-deployed
2021/12/15 19:29:21 Waiting for app: TestCasReplace to get un-deployed
2021/12/15 19:29:26 Waiting for app: TestCasReplace to get un-deployed
2021/12/15 19:29:26 App: TestCasReplace got un-deployed
2021/12/15 19:29:26 No eventing-con process running
2021/12/15 19:29:27 Function: TestCasReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCasReplace deleting in the background"
}
2021/12/15 19:29:28 Context: Bucket flush, response code: 200 dump: 
2021/12/15 19:29:38 Waiting for dst bucket item count to get to: 0 curr count: 2
2021/12/15 19:29:43 Waiting for dst bucket item count to get to: 0 curr count: 1
2021/12/15 19:29:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 19:29:45 Context: Bucket flush, response code: 200 dump: 
2021/12/15 19:29:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 19:29:50 Testing on the destination bucket for upsert with cas
2021/12/15 19:29:50 Starting bucket ops to source bucket
2021/12/15 19:29:50 Deploying app: TestCasReplace. Item count src bucket: 0 dst bucket: 0
2021/12/15 19:29:51 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCasReplace' in metakv",
  "warnings": null
 }
}
2021/12/15 19:29:56 Waiting for app: TestCasReplace to get deployed
2021/12/15 19:30:01 Waiting for app: TestCasReplace to get deployed
2021/12/15 19:30:06 Waiting for app: TestCasReplace to get deployed
2021/12/15 19:30:11 Waiting for app: TestCasReplace to get deployed
2021/12/15 19:30:11 App: TestCasReplace got deployed
2021/12/15 19:30:11 Starting bucket ops to source bucket
2021/12/15 19:30:16 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/15 19:30:21 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/15 19:30:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 19:30:22 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 19:30:22 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/15 19:30:22 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 19:30:27 Waiting for app: TestCasReplace to get un-deployed
2021/12/15 19:30:32 Waiting for app: TestCasReplace to get un-deployed
2021/12/15 19:30:37 Waiting for app: TestCasReplace to get un-deployed
2021/12/15 19:30:42 Waiting for app: TestCasReplace to get un-deployed
2021/12/15 19:30:42 App: TestCasReplace got un-deployed
2021/12/15 19:30:42 No eventing-con process running
2021/12/15 19:30:42 Function: TestCasReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCasReplace deleting in the background"
}
2021/12/15 19:30:43 Context: Bucket flush, response code: 200 dump: 
2021/12/15 19:30:53 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/15 19:30:58 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/15 19:30:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 19:31:00 Context: Bucket flush, response code: 200 dump: 
2021/12/15 19:31:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCasReplace (140.19s)
=== RUN   TestCasDelete
2021/12/15 19:31:05 Deploying app: TestCasDelete. Item count src bucket: 0 dst bucket: 0
2021/12/15 19:31:06 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCasDelete' in metakv",
  "warnings": null
 }
}
2021/12/15 19:31:11 Waiting for app: TestCasDelete to get deployed
2021/12/15 19:31:16 Waiting for app: TestCasDelete to get deployed
2021/12/15 19:31:21 Waiting for app: TestCasDelete to get deployed
2021/12/15 19:31:26 Waiting for app: TestCasDelete to get deployed
2021/12/15 19:31:26 App: TestCasDelete got deployed
2021/12/15 19:31:26 Starting bucket ops to source bucket
2021/12/15 19:31:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 19:31:27 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 19:31:27 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/15 19:31:27 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 19:31:32 Waiting for app: TestCasDelete to get un-deployed
2021/12/15 19:31:37 Waiting for app: TestCasDelete to get un-deployed
2021/12/15 19:31:42 Waiting for app: TestCasDelete to get un-deployed
2021/12/15 19:31:47 Waiting for app: TestCasDelete to get un-deployed
2021/12/15 19:31:47 App: TestCasDelete got un-deployed
2021/12/15 19:31:47 No eventing-con process running
2021/12/15 19:31:47 Function: TestCasDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCasDelete deleting in the background"
}
2021/12/15 19:31:48 Context: Bucket flush, response code: 200 dump: 
2021/12/15 19:31:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 19:31:56 Context: Bucket flush, response code: 200 dump: 
2021/12/15 19:32:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 19:32:01 Testing on the destination bucket for delete with cas
2021/12/15 19:32:01 Starting bucket ops to source bucket
2021/12/15 19:32:01 Deploying app: TestCasDelete. Item count src bucket: 0 dst bucket: 0
2021/12/15 19:32:02 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCasDelete' in metakv",
  "warnings": null
 }
}
2021/12/15 19:32:07 Waiting for app: TestCasDelete to get deployed
2021/12/15 19:32:12 Waiting for app: TestCasDelete to get deployed
2021/12/15 19:32:17 Waiting for app: TestCasDelete to get deployed
2021/12/15 19:32:22 Waiting for app: TestCasDelete to get deployed
2021/12/15 19:32:22 App: TestCasDelete got deployed
2021/12/15 19:32:22 Starting bucket ops to source bucket
2021/12/15 19:32:27 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/15 19:32:32 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/15 19:32:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 19:32:32 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 19:32:32 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/15 19:32:32 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 19:32:37 Waiting for app: TestCasDelete to get un-deployed
2021/12/15 19:32:42 Waiting for app: TestCasDelete to get un-deployed
2021/12/15 19:32:47 Waiting for app: TestCasDelete to get un-deployed
2021/12/15 19:32:52 Waiting for app: TestCasDelete to get un-deployed
2021/12/15 19:32:52 App: TestCasDelete got un-deployed
2021/12/15 19:32:52 No eventing-con process running
2021/12/15 19:32:52 Function: TestCasDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCasDelete deleting in the background"
}
2021/12/15 19:32:54 Context: Bucket flush, response code: 200 dump: 
2021/12/15 19:33:04 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/15 19:33:09 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/15 19:33:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 19:33:11 Context: Bucket flush, response code: 200 dump: 
2021/12/15 19:33:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCasDelete (130.77s)
=== RUN   TestCountersIncrement
2021/12/15 19:33:16 Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/15 19:33:21 Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/15 19:33:22 Removing node(s): [] from the cluster
2021/12/15 19:33:27 Rebalance progress: 100
2021/12/15 19:33:27 Deploying app: TestCountersIncrement. Item count src bucket: 0 dst bucket: 0
2021/12/15 19:33:28 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCountersIncrement' in metakv",
  "warnings": null
 }
}
2021/12/15 19:33:33 Waiting for app: TestCountersIncrement to get deployed
2021/12/15 19:33:38 Waiting for app: TestCountersIncrement to get deployed
2021/12/15 19:33:43 Waiting for app: TestCountersIncrement to get deployed
2021/12/15 19:33:48 Waiting for app: TestCountersIncrement to get deployed
2021/12/15 19:33:48 App: TestCountersIncrement got deployed
2021/12/15 19:33:48 Starting bucket ops to source bucket
2021/12/15 19:33:54 Waiting for dst bucket item count to get to: 1024 curr count: 0
2021/12/15 19:34:00 Waiting for dst bucket item count to get to: 1024 curr count: 417
2021/12/15 19:34:05 Waiting for dst bucket item count to get to: 1024 curr count: 417
2021/12/15 19:34:05 src & dst bucket item count matched up. src bucket count: 1024 dst bucket count: 1024
2021/12/15 19:34:05 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 19:34:05 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 19:34:10 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/15 19:34:15 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/15 19:34:20 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/15 19:34:25 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/15 19:34:25 App: TestCountersIncrement got un-deployed
2021/12/15 19:34:25 No eventing-con process running
2021/12/15 19:34:25 Function: TestCountersIncrement delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCountersIncrement deleting in the background"
}
2021/12/15 19:34:26 Context: Bucket flush, response code: 200 dump: 
2021/12/15 19:34:36 Waiting for dst bucket item count to get to: 0 curr count: 1025
2021/12/15 19:34:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 19:34:38 Context: Bucket flush, response code: 200 dump: 
2021/12/15 19:34:48 Waiting for dst bucket item count to get to: 0 curr count: 817
2021/12/15 19:34:53 Waiting for dst bucket item count to get to: 0 curr count: 305
2021/12/15 19:34:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 19:34:53 Deploying app: TestCountersIncrement. Item count src bucket: 0 dst bucket: 0
2021/12/15 19:34:54 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCountersIncrement' in metakv",
  "warnings": null
 }
}
2021/12/15 19:34:59 Waiting for app: TestCountersIncrement to get deployed
2021/12/15 19:35:04 Waiting for app: TestCountersIncrement to get deployed
2021/12/15 19:35:09 Waiting for app: TestCountersIncrement to get deployed
2021/12/15 19:35:14 Waiting for app: TestCountersIncrement to get deployed
2021/12/15 19:35:14 App: TestCountersIncrement got deployed
2021/12/15 19:35:14 Starting bucket ops to source bucket
2021/12/15 19:35:21 Waiting for dst bucket item count to get to: 2048 curr count: 0
2021/12/15 19:35:26 Waiting for dst bucket item count to get to: 2048 curr count: 1024
2021/12/15 19:35:26 src & dst bucket item count matched up. src bucket count: 2048 dst bucket count: 2048
2021/12/15 19:35:26 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 19:35:26 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 19:35:31 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/15 19:35:36 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/15 19:35:41 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/15 19:35:46 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/15 19:35:46 App: TestCountersIncrement got un-deployed
2021/12/15 19:35:46 No eventing-con process running
2021/12/15 19:35:46 Function: TestCountersIncrement delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCountersIncrement deleting in the background"
}
2021/12/15 19:35:48 Context: Bucket flush, response code: 200 dump: 
2021/12/15 19:35:58 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/15 19:35:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 19:35:59 Context: Bucket flush, response code: 200 dump: 
2021/12/15 19:36:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 19:36:04 Removing node(s): [http://127.0.0.1:19002 http://127.0.0.1:19003] from the cluster
2021/12/15 19:36:09 Rebalance progress: 100
--- PASS: TestCountersIncrement (173.58s)
=== RUN   TestCountersDecrement
2021/12/15 19:36:09 Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/15 19:36:10 Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/15 19:36:10 Removing node(s): [] from the cluster
2021/12/15 19:36:15 Rebalance progress: 100
2021/12/15 19:36:15 Deploying app: TestCountersDecrement. Item count src bucket: 0 dst bucket: 0
2021/12/15 19:36:17 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCountersDecrement' in metakv",
  "warnings": null
 }
}
2021/12/15 19:36:22 Waiting for app: TestCountersDecrement to get deployed
2021/12/15 19:36:27 Waiting for app: TestCountersDecrement to get deployed
2021/12/15 19:36:32 Waiting for app: TestCountersDecrement to get deployed
2021/12/15 19:36:37 Waiting for app: TestCountersDecrement to get deployed
2021/12/15 19:36:37 App: TestCountersDecrement got deployed
2021/12/15 19:36:37 Starting bucket ops to source bucket
2021/12/15 19:36:43 Waiting for dst bucket item count to get to: 1024 curr count: 0
2021/12/15 19:36:48 Waiting for dst bucket item count to get to: 1024 curr count: 794
2021/12/15 19:36:53 Waiting for dst bucket item count to get to: 1024 curr count: 794
2021/12/15 19:36:53 src & dst bucket item count matched up. src bucket count: 1024 dst bucket count: 1024
2021/12/15 19:36:53 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 19:36:53 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 19:36:58 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/15 19:37:03 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/15 19:37:08 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/15 19:37:13 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/15 19:37:13 App: TestCountersDecrement got un-deployed
2021/12/15 19:37:13 No eventing-con process running
2021/12/15 19:37:13 Function: TestCountersDecrement delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCountersDecrement deleting in the background"
}
2021/12/15 19:37:15 Context: Bucket flush, response code: 200 dump: 
2021/12/15 19:37:25 Waiting for dst bucket item count to get to: 0 curr count: 1025
2021/12/15 19:37:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 19:37:26 Context: Bucket flush, response code: 200 dump: 
2021/12/15 19:37:36 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/15 19:37:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 19:37:36 Deploying app: TestCountersDecrement. Item count src bucket: 0 dst bucket: 0
2021/12/15 19:37:38 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCountersDecrement' in metakv",
  "warnings": null
 }
}
2021/12/15 19:37:43 Waiting for app: TestCountersDecrement to get deployed
2021/12/15 19:37:48 Waiting for app: TestCountersDecrement to get deployed
2021/12/15 19:37:53 Waiting for app: TestCountersDecrement to get deployed
2021/12/15 19:37:58 Waiting for app: TestCountersDecrement to get deployed
2021/12/15 19:37:58 App: TestCountersDecrement got deployed
2021/12/15 19:37:58 Starting bucket ops to source bucket
2021/12/15 19:38:04 Waiting for dst bucket item count to get to: 2048 curr count: 0
2021/12/15 19:38:09 Waiting for dst bucket item count to get to: 2048 curr count: 1024
2021/12/15 19:38:14 Waiting for dst bucket item count to get to: 2048 curr count: 1024
2021/12/15 19:38:14 src & dst bucket item count matched up. src bucket count: 2048 dst bucket count: 2048
2021/12/15 19:38:14 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 19:38:14 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 19:38:19 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/15 19:38:24 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/15 19:38:29 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/15 19:38:34 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/15 19:38:34 App: TestCountersDecrement got un-deployed
2021/12/15 19:38:34 No eventing-con process running
2021/12/15 19:38:34 Function: TestCountersDecrement delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCountersDecrement deleting in the background"
}
2021/12/15 19:38:36 Context: Bucket flush, response code: 200 dump: 
2021/12/15 19:38:46 Waiting for dst bucket item count to get to: 0 curr count: 2048
2021/12/15 19:38:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 19:38:48 Context: Bucket flush, response code: 200 dump: 
2021/12/15 19:38:58 Waiting for dst bucket item count to get to: 0 curr count: 1
2021/12/15 19:38:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 19:38:58 Removing node(s): [http://127.0.0.1:19002 http://127.0.0.1:19003] from the cluster
2021/12/15 19:39:03 Rebalance progress: 100
--- PASS: TestCountersDecrement (173.14s)
=== RUN   TestCacheGet
2021/12/15 19:39:03 Deploying app: TestCacheGet. Item count src bucket: 0 dst bucket: 0
2021/12/15 19:39:04 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCacheGet' in metakv",
  "warnings": null
 }
}
2021/12/15 19:39:09 Waiting for app: TestCacheGet to get deployed
2021/12/15 19:39:14 Waiting for app: TestCacheGet to get deployed
2021/12/15 19:39:19 Waiting for app: TestCacheGet to get deployed
2021/12/15 19:39:24 Waiting for app: TestCacheGet to get deployed
2021/12/15 19:39:24 App: TestCacheGet got deployed
2021/12/15 19:39:24 Starting bucket ops to source bucket
2021/12/15 19:39:29 Waiting for dst bucket item count to get to: 20 curr count: 0
2021/12/15 19:39:34 Waiting for dst bucket item count to get to: 20 curr count: 0
2021/12/15 19:39:39 Waiting for dst bucket item count to get to: 20 curr count: 0
2021/12/15 19:39:44 Waiting for dst bucket item count to get to: 20 curr count: 8
2021/12/15 19:39:49 Waiting for dst bucket item count to get to: 20 curr count: 16
2021/12/15 19:39:54 Waiting for dst bucket item count to get to: 20 curr count: 18
2021/12/15 19:39:54 src & dst bucket item count matched up. src bucket count: 20 dst bucket count: 20
2021/12/15 19:39:54 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 19:39:54 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 19:39:59 Waiting for app: TestCacheGet to get un-deployed
2021/12/15 19:40:04 Waiting for app: TestCacheGet to get un-deployed
2021/12/15 19:40:09 Waiting for app: TestCacheGet to get un-deployed
2021/12/15 19:40:14 Waiting for app: TestCacheGet to get un-deployed
2021/12/15 19:40:14 App: TestCacheGet got un-deployed
2021/12/15 19:40:14 No eventing-con process running
2021/12/15 19:40:14 Function: TestCacheGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCacheGet deleting in the background"
}
2021/12/15 19:40:16 Context: Bucket flush, response code: 200 dump: 
2021/12/15 19:40:26 Waiting for dst bucket item count to get to: 0 curr count: 20
2021/12/15 19:40:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 19:40:28 Context: Bucket flush, response code: 200 dump: 
2021/12/15 19:40:38 Waiting for dst bucket item count to get to: 0 curr count: 12
2021/12/15 19:40:43 Waiting for dst bucket item count to get to: 0 curr count: 2
2021/12/15 19:40:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCacheGet (100.24s)
=== RUN   TestCacheRYOW
2021/12/15 19:40:43 Deploying app: TestCacheRYOW. Item count src bucket: 0 dst bucket: 0
2021/12/15 19:40:44 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCacheRYOW' in metakv",
  "warnings": null
 }
}
2021/12/15 19:40:49 Waiting for app: TestCacheRYOW to get deployed
2021/12/15 19:40:54 Waiting for app: TestCacheRYOW to get deployed
2021/12/15 19:40:59 Waiting for app: TestCacheRYOW to get deployed
2021/12/15 19:41:04 Waiting for app: TestCacheRYOW to get deployed
2021/12/15 19:41:04 App: TestCacheRYOW got deployed
2021/12/15 19:41:04 Starting bucket ops to source bucket
2021/12/15 19:41:09 Waiting for dst bucket item count to get to: 50 curr count: 0
2021/12/15 19:41:14 Waiting for dst bucket item count to get to: 50 curr count: 25
2021/12/15 19:41:14 src & dst bucket item count matched up. src bucket count: 50 dst bucket count: 50
2021/12/15 19:41:14 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 19:41:14 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 19:41:19 Waiting for app: TestCacheRYOW to get un-deployed
2021/12/15 19:41:24 Waiting for app: TestCacheRYOW to get un-deployed
2021/12/15 19:41:29 Waiting for app: TestCacheRYOW to get un-deployed
2021/12/15 19:41:34 Waiting for app: TestCacheRYOW to get un-deployed
2021/12/15 19:41:34 App: TestCacheRYOW got un-deployed
2021/12/15 19:41:34 No eventing-con process running
2021/12/15 19:41:35 Function: TestCacheRYOW delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCacheRYOW deleting in the background"
}
2021/12/15 19:41:36 Context: Bucket flush, response code: 200 dump: 
2021/12/15 19:41:46 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/15 19:41:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 19:41:48 Context: Bucket flush, response code: 200 dump: 
2021/12/15 19:41:58 Waiting for dst bucket item count to get to: 0 curr count: 35
2021/12/15 19:42:03 Waiting for dst bucket item count to get to: 0 curr count: 10
2021/12/15 19:42:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCacheRYOW (80.24s)
=== RUN   TestCacheExpiry
2021/12/15 19:42:03 Deploying app: TestCacheExpiry. Item count src bucket: 0 dst bucket: 0
2021/12/15 19:42:04 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCacheExpiry' in metakv",
  "warnings": null
 }
}
2021/12/15 19:42:09 Waiting for app: TestCacheExpiry to get deployed
2021/12/15 19:42:14 Waiting for app: TestCacheExpiry to get deployed
2021/12/15 19:42:19 Waiting for app: TestCacheExpiry to get deployed
2021/12/15 19:42:24 Waiting for app: TestCacheExpiry to get deployed
2021/12/15 19:42:24 App: TestCacheExpiry got deployed
2021/12/15 19:42:24 Starting bucket ops to source bucket
2021/12/15 19:42:30 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/15 19:42:35 Waiting for dst bucket item count to get to: 100 curr count: 14
2021/12/15 19:42:40 Waiting for dst bucket item count to get to: 100 curr count: 37
2021/12/15 19:42:45 Waiting for dst bucket item count to get to: 100 curr count: 72
2021/12/15 19:42:50 Waiting for dst bucket item count to get to: 100 curr count: 99
2021/12/15 19:42:50 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/15 19:42:50 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 19:42:50 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 19:42:55 Waiting for app: TestCacheExpiry to get un-deployed
2021/12/15 19:43:00 Waiting for app: TestCacheExpiry to get un-deployed
2021/12/15 19:43:05 Waiting for app: TestCacheExpiry to get un-deployed
2021/12/15 19:43:10 Waiting for app: TestCacheExpiry to get un-deployed
2021/12/15 19:43:10 App: TestCacheExpiry got un-deployed
2021/12/15 19:43:10 No eventing-con process running
2021/12/15 19:43:10 Function: TestCacheExpiry delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCacheExpiry deleting in the background"
}
2021/12/15 19:43:12 Context: Bucket flush, response code: 200 dump: 
2021/12/15 19:43:22 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/15 19:43:27 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/15 19:43:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 19:43:28 Context: Bucket flush, response code: 200 dump: 
2021/12/15 19:43:38 Waiting for dst bucket item count to get to: 0 curr count: 81
2021/12/15 19:43:43 Waiting for dst bucket item count to get to: 0 curr count: 31
2021/12/15 19:43:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCacheExpiry (100.28s)
=== RUN   TestCacheExpiryCustom
2021/12/15 19:43:43 Deploying app: TestCacheExpiryCustom. Item count src bucket: 0 dst bucket: 0
2021/12/15 19:43:45 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCacheExpiryCustom' in metakv",
  "warnings": null
 }
}
2021/12/15 19:43:50 Waiting for app: TestCacheExpiryCustom to get deployed
2021/12/15 19:43:55 Waiting for app: TestCacheExpiryCustom to get deployed
2021/12/15 19:44:00 Waiting for app: TestCacheExpiryCustom to get deployed
2021/12/15 19:44:05 Waiting for app: TestCacheExpiryCustom to get deployed
2021/12/15 19:44:05 App: TestCacheExpiryCustom got deployed
2021/12/15 19:44:05 Starting bucket ops to source bucket
2021/12/15 19:44:10 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/15 19:44:15 Waiting for dst bucket item count to get to: 100 curr count: 14
2021/12/15 19:44:20 Waiting for dst bucket item count to get to: 100 curr count: 37
2021/12/15 19:44:25 Waiting for dst bucket item count to get to: 100 curr count: 72
2021/12/15 19:44:30 Waiting for dst bucket item count to get to: 100 curr count: 99
2021/12/15 19:44:30 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/15 19:44:30 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 19:44:30 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 19:44:35 Waiting for app: TestCacheExpiryCustom to get un-deployed
2021/12/15 19:44:40 Waiting for app: TestCacheExpiryCustom to get un-deployed
2021/12/15 19:44:45 Waiting for app: TestCacheExpiryCustom to get un-deployed
2021/12/15 19:44:50 Waiting for app: TestCacheExpiryCustom to get un-deployed
2021/12/15 19:44:50 App: TestCacheExpiryCustom got un-deployed
2021/12/15 19:44:50 No eventing-con process running
2021/12/15 19:44:50 Function: TestCacheExpiryCustom delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCacheExpiryCustom deleting in the background"
}
2021/12/15 19:44:52 Context: Bucket flush, response code: 200 dump: 
2021/12/15 19:45:02 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/15 19:45:07 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/15 19:45:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 19:45:09 Context: Bucket flush, response code: 200 dump: 
2021/12/15 19:45:19 Waiting for dst bucket item count to get to: 0 curr count: 85
2021/12/15 19:45:24 Waiting for dst bucket item count to get to: 0 curr count: 35
2021/12/15 19:45:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCacheExpiryCustom (100.24s)
=== RUN   TestCacheOverflowCustom
2021/12/15 19:45:24 Deploying app: TestCacheOverflowCustom. Item count src bucket: 0 dst bucket: 0
2021/12/15 19:45:25 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCacheOverflowCustom' in metakv",
  "warnings": null
 }
}
2021/12/15 19:45:30 Waiting for app: TestCacheOverflowCustom to get deployed
2021/12/15 19:45:35 Waiting for app: TestCacheOverflowCustom to get deployed
2021/12/15 19:45:40 Waiting for app: TestCacheOverflowCustom to get deployed
2021/12/15 19:45:45 Waiting for app: TestCacheOverflowCustom to get deployed
2021/12/15 19:45:45 App: TestCacheOverflowCustom got deployed
2021/12/15 19:45:45 Starting bucket ops to source bucket
2021/12/15 19:45:50 Waiting for dst bucket item count to get to: 35 curr count: 0
2021/12/15 19:45:55 Waiting for dst bucket item count to get to: 35 curr count: 6
2021/12/15 19:46:00 Waiting for dst bucket item count to get to: 35 curr count: 15
2021/12/15 19:46:05 Waiting for dst bucket item count to get to: 35 curr count: 23
2021/12/15 19:46:10 Waiting for dst bucket item count to get to: 35 curr count: 27
2021/12/15 19:46:10 src & dst bucket item count matched up. src bucket count: 35 dst bucket count: 35
2021/12/15 19:46:10 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 19:46:10 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 19:46:15 Waiting for app: TestCacheOverflowCustom to get un-deployed
2021/12/15 19:46:20 Waiting for app: TestCacheOverflowCustom to get un-deployed
2021/12/15 19:46:25 Waiting for app: TestCacheOverflowCustom to get un-deployed
2021/12/15 19:46:30 Waiting for app: TestCacheOverflowCustom to get un-deployed
2021/12/15 19:46:30 App: TestCacheOverflowCustom got un-deployed
2021/12/15 19:46:30 No eventing-con process running
2021/12/15 19:46:30 Function: TestCacheOverflowCustom delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCacheOverflowCustom deleting in the background"
}
2021/12/15 19:46:32 Context: Bucket flush, response code: 200 dump: 
2021/12/15 19:46:42 Waiting for dst bucket item count to get to: 0 curr count: 5
2021/12/15 19:46:47 Waiting for dst bucket item count to get to: 0 curr count: 5
2021/12/15 19:46:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 19:46:49 Context: Bucket flush, response code: 200 dump: 
2021/12/15 19:46:59 Waiting for dst bucket item count to get to: 0 curr count: 34
2021/12/15 19:47:04 Waiting for dst bucket item count to get to: 0 curr count: 16
2021/12/15 19:47:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCacheOverflowCustom (100.28s)
=== RUN   TestStrictMode
2021/12/15 19:47:04 Deploying app: TestStrictMode. Item count src bucket: 0 dst bucket: 0
2021/12/15 19:47:05 Context: Post to main store, response code: 422 dump: {
 "name": "ERR_HANDLER_COMPILATION",
 "code": 27,
 "description": "Function compilation failed",
 "attributes": null,
 "runtime_info": {
  "code": 27,
  "info": {
   "area": "handlerCode",
   "column_number": 63,
   "compile_success": false,
   "description": "        destination[meta.id] = 00000000000000000000000000000101;\n                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nSyntaxError: Octal literals are not allowed in strict mode.\n",
   "index": 132,
   "language": "JavaScript",
   "line_number": 4
  }
 }
}
--- PASS: TestStrictMode (1.06s)
=== RUN   TestDataTypes
2021/12/15 19:47:05 Deploying app: TestDataTypes. Item count src bucket: 0 dst bucket: 0
2021/12/15 19:47:06 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDataTypes' in metakv",
  "warnings": null
 }
}
2021/12/15 19:47:11 Waiting for app: TestDataTypes to get deployed
2021/12/15 19:47:16 Waiting for app: TestDataTypes to get deployed
2021/12/15 19:47:21 Waiting for app: TestDataTypes to get deployed
2021/12/15 19:47:26 Waiting for app: TestDataTypes to get deployed
2021/12/15 19:47:26 App: TestDataTypes got deployed
2021/12/15 19:47:26 Starting bucket ops to source bucket
2021/12/15 19:47:31 Waiting for dst bucket item count to get to: 18 curr count: 0
2021/12/15 19:47:31 src & dst bucket item count matched up. src bucket count: 18 dst bucket count: 18
2021/12/15 19:47:31 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 19:47:31 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 19:47:36 Waiting for app: TestDataTypes to get un-deployed
2021/12/15 19:47:41 Waiting for app: TestDataTypes to get un-deployed
2021/12/15 19:47:46 Waiting for app: TestDataTypes to get un-deployed
2021/12/15 19:47:51 Waiting for app: TestDataTypes to get un-deployed
2021/12/15 19:47:51 App: TestDataTypes got un-deployed
2021/12/15 19:47:51 No eventing-con process running
2021/12/15 19:47:51 Function: TestDataTypes delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDataTypes deleting in the background"
}
2021/12/15 19:47:53 Context: Bucket flush, response code: 200 dump: 
2021/12/15 19:48:03 Waiting for dst bucket item count to get to: 0 curr count: 1
2021/12/15 19:48:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 19:48:05 Context: Bucket flush, response code: 200 dump: 
2021/12/15 19:48:15 Waiting for dst bucket item count to get to: 0 curr count: 18
2021/12/15 19:48:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDataTypes (69.99s)
=== RUN   TestEnoentGet
2021/12/15 19:48:15 Deploying app: TestEnoentGet. Item count src bucket: 0 dst bucket: 0
2021/12/15 19:48:16 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentGet' in metakv",
  "warnings": null
 }
}
2021/12/15 19:48:21 Waiting for app: TestEnoentGet to get deployed
2021/12/15 19:48:26 Waiting for app: TestEnoentGet to get deployed
2021/12/15 19:48:31 Waiting for app: TestEnoentGet to get deployed
2021/12/15 19:48:36 Waiting for app: TestEnoentGet to get deployed
2021/12/15 19:48:36 App: TestEnoentGet got deployed
2021/12/15 19:48:36 Starting bucket ops to source bucket
2021/12/15 19:48:41 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/15 19:48:41 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/15 19:48:41 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 19:48:42 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 19:48:47 Waiting for app: TestEnoentGet to get un-deployed
2021/12/15 19:48:52 Waiting for app: TestEnoentGet to get un-deployed
2021/12/15 19:48:57 Waiting for app: TestEnoentGet to get un-deployed
2021/12/15 19:49:02 Waiting for app: TestEnoentGet to get un-deployed
2021/12/15 19:49:02 App: TestEnoentGet got un-deployed
2021/12/15 19:49:02 No eventing-con process running
2021/12/15 19:49:02 Function: TestEnoentGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentGet deleting in the background"
}
2021/12/15 19:49:03 Context: Bucket flush, response code: 200 dump: 
2021/12/15 19:49:13 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/15 19:49:18 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/15 19:49:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 19:49:20 Context: Bucket flush, response code: 200 dump: 
2021/12/15 19:49:30 Waiting for dst bucket item count to get to: 0 curr count: 74
2021/12/15 19:49:35 Waiting for dst bucket item count to get to: 0 curr count: 74
2021/12/15 19:49:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 19:49:35 Deploying app: TestEnoentGet. Item count src bucket: 0 dst bucket: 0
2021/12/15 19:49:36 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentGet' in metakv",
  "warnings": null
 }
}
2021/12/15 19:49:41 Waiting for app: TestEnoentGet to get deployed
2021/12/15 19:49:46 Waiting for app: TestEnoentGet to get deployed
2021/12/15 19:49:51 Waiting for app: TestEnoentGet to get deployed
2021/12/15 19:49:56 Waiting for app: TestEnoentGet to get deployed
2021/12/15 19:49:56 App: TestEnoentGet got deployed
2021/12/15 19:49:56 Starting bucket ops to source bucket
2021/12/15 19:50:02 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/15 19:50:02 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/15 19:50:02 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 19:50:02 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 19:50:07 Waiting for app: TestEnoentGet to get un-deployed
2021/12/15 19:50:12 Waiting for app: TestEnoentGet to get un-deployed
2021/12/15 19:50:17 Waiting for app: TestEnoentGet to get un-deployed
2021/12/15 19:50:22 Waiting for app: TestEnoentGet to get un-deployed
2021/12/15 19:50:22 App: TestEnoentGet got un-deployed
2021/12/15 19:50:22 No eventing-con process running
2021/12/15 19:50:22 Function: TestEnoentGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentGet deleting in the background"
}
2021/12/15 19:50:24 Context: Bucket flush, response code: 200 dump: 
2021/12/15 19:50:34 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/15 19:50:39 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/15 19:50:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 19:50:40 Context: Bucket flush, response code: 200 dump: 
2021/12/15 19:50:50 Waiting for dst bucket item count to get to: 0 curr count: 79
2021/12/15 19:50:55 Waiting for dst bucket item count to get to: 0 curr count: 79
2021/12/15 19:50:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEnoentGet (160.49s)
=== RUN   TestEnoentDelete
2021/12/15 19:50:55 Deploying app: TestEnoentDelete. Item count src bucket: 0 dst bucket: 0
2021/12/15 19:50:57 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentDelete' in metakv",
  "warnings": null
 }
}
2021/12/15 19:51:02 Waiting for app: TestEnoentDelete to get deployed
2021/12/15 19:51:07 Waiting for app: TestEnoentDelete to get deployed
2021/12/15 19:51:12 Waiting for app: TestEnoentDelete to get deployed
2021/12/15 19:51:17 Waiting for app: TestEnoentDelete to get deployed
2021/12/15 19:51:17 App: TestEnoentDelete got deployed
2021/12/15 19:51:17 Starting bucket ops to source bucket
2021/12/15 19:51:22 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/15 19:51:22 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/15 19:51:22 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 19:51:22 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 19:51:27 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/15 19:51:32 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/15 19:51:37 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/15 19:51:42 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/15 19:51:42 App: TestEnoentDelete got un-deployed
2021/12/15 19:51:42 No eventing-con process running
2021/12/15 19:51:42 Function: TestEnoentDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentDelete deleting in the background"
}
2021/12/15 19:51:44 Context: Bucket flush, response code: 200 dump: 
2021/12/15 19:51:54 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/15 19:51:59 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/15 19:51:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 19:52:01 Context: Bucket flush, response code: 200 dump: 
2021/12/15 19:52:11 Waiting for dst bucket item count to get to: 0 curr count: 81
2021/12/15 19:52:16 Waiting for dst bucket item count to get to: 0 curr count: 81
2021/12/15 19:52:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 19:52:16 Deploying app: TestEnoentDelete. Item count src bucket: 0 dst bucket: 0
2021/12/15 19:52:17 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentDelete' in metakv",
  "warnings": null
 }
}
2021/12/15 19:52:22 Waiting for app: TestEnoentDelete to get deployed
2021/12/15 19:52:27 Waiting for app: TestEnoentDelete to get deployed
2021/12/15 19:52:32 Waiting for app: TestEnoentDelete to get deployed
2021/12/15 19:52:37 Waiting for app: TestEnoentDelete to get deployed
2021/12/15 19:52:37 App: TestEnoentDelete got deployed
2021/12/15 19:52:37 Starting bucket ops to source bucket
2021/12/15 19:52:42 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/15 19:52:42 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/15 19:52:42 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 19:52:42 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 19:52:47 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/15 19:52:52 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/15 19:52:57 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/15 19:53:02 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/15 19:53:02 App: TestEnoentDelete got un-deployed
2021/12/15 19:53:02 No eventing-con process running
2021/12/15 19:53:02 Function: TestEnoentDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentDelete deleting in the background"
}
2021/12/15 19:53:04 Context: Bucket flush, response code: 200 dump: 
2021/12/15 19:53:14 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/15 19:53:19 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/15 19:53:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 19:53:21 Context: Bucket flush, response code: 200 dump: 
2021/12/15 19:53:31 Waiting for dst bucket item count to get to: 0 curr count: 92
2021/12/15 19:53:36 Waiting for dst bucket item count to get to: 0 curr count: 92
2021/12/15 19:53:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEnoentDelete (160.41s)
=== RUN   TestError
2021/12/15 19:53:46 Waiting for app: TestError to get un-deployed
2021/12/15 19:53:46 App: TestError got un-deployed
2021/12/15 19:53:46 No eventing-con process running
2021/12/15 19:53:46 Function: TestError delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestError not found"
 }
}
2021/12/15 19:53:48 Context: Bucket flush, response code: 200 dump: 
2021/12/15 19:53:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 19:53:54 Context: Bucket flush, response code: 200 dump: 
2021/12/15 19:53:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 19:53:59 Deploying app: TestError. Item count src bucket: 0 dst bucket: 0
2021/12/15 19:54:01 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestError' in metakv",
  "warnings": null
 }
}
2021/12/15 19:54:06 Waiting for app: TestError to get deployed
2021/12/15 19:54:11 Waiting for app: TestError to get deployed
2021/12/15 19:54:16 Waiting for app: TestError to get deployed
2021/12/15 19:54:21 Waiting for app: TestError to get deployed
2021/12/15 19:54:21 App: TestError got deployed
2021/12/15 19:54:21 Starting bucket ops to source bucket
2021/12/15 19:54:26 Waiting for dst bucket item count to get to: 300 curr count: 0
2021/12/15 19:54:31 Waiting for dst bucket item count to get to: 300 curr count: 0
2021/12/15 19:54:31 src & dst bucket item count matched up. src bucket count: 300 dst bucket count: 300
2021/12/15 19:54:31 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 19:54:31 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 19:54:36 Waiting for app: TestError to get un-deployed
2021/12/15 19:54:41 Waiting for app: TestError to get un-deployed
2021/12/15 19:54:46 Waiting for app: TestError to get un-deployed
2021/12/15 19:54:51 Waiting for app: TestError to get un-deployed
2021/12/15 19:54:51 App: TestError got un-deployed
2021/12/15 19:54:51 No eventing-con process running
2021/12/15 19:54:51 Function: TestError delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestError deleting in the background"
}
2021/12/15 19:54:53 Context: Bucket flush, response code: 200 dump: 
2021/12/15 19:55:03 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/15 19:55:08 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/15 19:55:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 19:55:10 Context: Bucket flush, response code: 200 dump: 
2021/12/15 19:55:20 Waiting for dst bucket item count to get to: 0 curr count: 156
2021/12/15 19:55:25 Waiting for dst bucket item count to get to: 0 curr count: 156
2021/12/15 19:55:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestError (108.75s)
=== RUN   TestCRLF
2021/12/15 19:55:35 Waiting for app: TestCRLF to get un-deployed
2021/12/15 19:55:35 App: TestCRLF got un-deployed
2021/12/15 19:55:35 No eventing-con process running
2021/12/15 19:55:35 Function: TestCRLF delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestCRLF not found"
 }
}
2021/12/15 19:55:36 Context: Bucket flush, response code: 200 dump: 
2021/12/15 19:55:41 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 19:55:43 Context: Bucket flush, response code: 200 dump: 
2021/12/15 19:55:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 19:55:48 Deploying app: TestCRLF. Item count src bucket: 0 dst bucket: 0
2021/12/15 19:55:49 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCRLF' in metakv",
  "warnings": null
 }
}
2021/12/15 19:55:54 Waiting for app: TestCRLF to get deployed
2021/12/15 19:55:59 Waiting for app: TestCRLF to get deployed
2021/12/15 19:56:04 Waiting for app: TestCRLF to get deployed
2021/12/15 19:56:09 Waiting for app: TestCRLF to get deployed
2021/12/15 19:56:09 App: TestCRLF got deployed
2021/12/15 19:56:09 Starting bucket ops to source bucket
2021/12/15 19:56:15 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/15 19:56:20 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/15 19:56:25 Waiting for dst bucket item count to get to: 100 curr count: 54
2021/12/15 19:56:25 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/15 19:56:25 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 19:56:25 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 19:56:30 Waiting for app: TestCRLF to get un-deployed
2021/12/15 19:56:35 Waiting for app: TestCRLF to get un-deployed
2021/12/15 19:56:40 Waiting for app: TestCRLF to get un-deployed
2021/12/15 19:56:45 Waiting for app: TestCRLF to get un-deployed
2021/12/15 19:56:45 App: TestCRLF got un-deployed
2021/12/15 19:56:45 No eventing-con process running
2021/12/15 19:56:45 Function: TestCRLF delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCRLF deleting in the background"
}
2021/12/15 19:56:47 Context: Bucket flush, response code: 200 dump: 
2021/12/15 19:56:57 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/15 19:56:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 19:56:58 Context: Bucket flush, response code: 200 dump: 
2021/12/15 19:57:09 Waiting for dst bucket item count to get to: 0 curr count: 81
2021/12/15 19:57:14 Waiting for dst bucket item count to get to: 0 curr count: 35
2021/12/15 19:57:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCRLF (108.94s)
=== RUN   TestImportExport
2021/12/15 19:57:24 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/15 19:57:24 App: TestImportExport_n1ql_insert_on_update got un-deployed
2021/12/15 19:57:24 No eventing-con process running
2021/12/15 19:57:24 Function: TestImportExport_n1ql_insert_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestImportExport_n1ql_insert_on_update not found"
 }
}
2021/12/15 19:57:25 Context: Bucket flush, response code: 200 dump: 
2021/12/15 19:57:30 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 19:57:32 Context: Bucket flush, response code: 200 dump: 
2021/12/15 19:57:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 19:57:42 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/15 19:57:42 App: TestImportExport_bucket_op_on_update got un-deployed
2021/12/15 19:57:42 No eventing-con process running
2021/12/15 19:57:42 Function: TestImportExport_bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestImportExport_bucket_op_on_update not found"
 }
}
2021/12/15 19:57:44 Context: Bucket flush, response code: 200 dump: 
2021/12/15 19:57:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 19:57:51 Context: Bucket flush, response code: 200 dump: 
2021/12/15 19:57:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 19:57:56 Deploying app: TestImportExport_n1ql_insert_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/15 19:57:57 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestImportExport_n1ql_insert_on_update' in metakv",
  "warnings": null
 }
}
2021/12/15 19:57:57 Deploying app: TestImportExport_bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/15 19:57:58 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestImportExport_bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/15 19:58:03 Waiting for app: TestImportExport_n1ql_insert_on_update to get deployed
2021/12/15 19:58:08 Waiting for app: TestImportExport_n1ql_insert_on_update to get deployed
2021/12/15 19:58:13 Waiting for app: TestImportExport_n1ql_insert_on_update to get deployed
2021/12/15 19:58:18 Waiting for app: TestImportExport_n1ql_insert_on_update to get deployed
2021/12/15 19:58:18 App: TestImportExport_n1ql_insert_on_update got deployed
2021/12/15 19:58:23 Waiting for app: TestImportExport_bucket_op_on_update to get deployed
2021/12/15 19:58:28 Waiting for app: TestImportExport_bucket_op_on_update to get deployed
2021/12/15 19:58:33 Waiting for app: TestImportExport_bucket_op_on_update to get deployed
2021/12/15 19:58:33 App: TestImportExport_bucket_op_on_update got deployed
2021/12/15 19:58:33 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 19:58:38 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/15 19:58:43 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/15 19:58:48 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/15 19:58:53 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/15 19:58:53 App: TestImportExport_n1ql_insert_on_update got un-deployed
2021/12/15 19:58:53 No eventing-con process running
2021/12/15 19:58:53 Function: TestImportExport_n1ql_insert_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestImportExport_n1ql_insert_on_update deleting in the background"
}
2021/12/15 19:58:54 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 19:58:59 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/15 19:59:04 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/15 19:59:09 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/15 19:59:14 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/15 19:59:14 App: TestImportExport_bucket_op_on_update got un-deployed
2021/12/15 19:59:14 No eventing-con process running
2021/12/15 19:59:14 Function: TestImportExport_bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestImportExport_bucket_op_on_update deleting in the background"
}
2021/12/15 19:59:31 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/15 19:59:31 App: TestImportExport_bucket_op_on_update got un-deployed
2021/12/15 19:59:31 No eventing-con process running
2021/12/15 19:59:31 Function: TestImportExport_bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestImportExport_bucket_op_on_update deleting in the background"
}
2021/12/15 19:59:33 Context: Bucket flush, response code: 200 dump: 
2021/12/15 19:59:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 19:59:40 Context: Bucket flush, response code: 200 dump: 
2021/12/15 19:59:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 19:59:50 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/15 19:59:50 App: TestImportExport_n1ql_insert_on_update got un-deployed
2021/12/15 19:59:50 No eventing-con process running
2021/12/15 19:59:50 Function: TestImportExport_n1ql_insert_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestImportExport_n1ql_insert_on_update deleting in the background"
}
2021/12/15 19:59:51 Context: Bucket flush, response code: 200 dump: 
2021/12/15 19:59:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 19:59:58 Context: Bucket flush, response code: 200 dump: 
2021/12/15 20:00:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestImportExport (169.64s)
=== RUN   TestDeployUndeployLoopNonDefaultSettings
2021/12/15 20:00:13 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/15 20:00:13 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/15 20:00:13 No eventing-con process running
2021/12/15 20:00:13 Function: TestDeployUndeployLoopNonDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDeployUndeployLoopNonDefaultSettings not found"
 }
}
2021/12/15 20:00:15 Context: Bucket flush, response code: 200 dump: 
2021/12/15 20:00:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 20:00:22 Context: Bucket flush, response code: 200 dump: 
2021/12/15 20:00:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 20:00:27 Deploying app: TestDeployUndeployLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/15 20:00:28 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/15 20:00:28 Starting bucket ops to source bucket
2021/12/15 20:00:36 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 20:00:41 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 20:00:46 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 20:00:51 Waiting for dst bucket item count to get to: 5000 curr count: 241
2021/12/15 20:00:56 Waiting for dst bucket item count to get to: 5000 curr count: 241
2021/12/15 20:00:56 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/15 20:01:01 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get deployed
2021/12/15 20:01:01 App: TestDeployUndeployLoopNonDefaultSettings got deployed
2021/12/15 20:01:01 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 20:01:01 Undeploying app: bucket_op_on_update
2021/12/15 20:01:01 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 20:01:06 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/15 20:01:11 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/15 20:01:16 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/15 20:01:21 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/15 20:01:21 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/15 20:01:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 20:01:21 No eventing-con process running
2021/12/15 20:01:23 Context: Bucket flush, response code: 200 dump: 
2021/12/15 20:01:30 Context: Bucket flush, response code: 200 dump: 
2021/12/15 20:02:05 Deploying app: TestDeployUndeployLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/15 20:02:07 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/15 20:02:07 Starting bucket ops to source bucket
2021/12/15 20:02:15 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 20:02:20 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 20:02:25 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 20:02:30 Waiting for dst bucket item count to get to: 5000 curr count: 1493
2021/12/15 20:02:35 Waiting for dst bucket item count to get to: 5000 curr count: 1493
2021/12/15 20:02:35 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/15 20:02:40 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get deployed
2021/12/15 20:02:40 App: TestDeployUndeployLoopNonDefaultSettings got deployed
2021/12/15 20:02:40 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 20:02:40 Undeploying app: bucket_op_on_update
2021/12/15 20:02:40 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 20:02:45 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/15 20:02:50 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/15 20:02:55 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/15 20:03:00 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/15 20:03:00 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/15 20:03:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 20:03:00 No eventing-con process running
2021/12/15 20:03:02 Context: Bucket flush, response code: 200 dump: 
2021/12/15 20:03:09 Context: Bucket flush, response code: 200 dump: 
2021/12/15 20:03:44 Deploying app: TestDeployUndeployLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/15 20:03:45 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/15 20:03:45 Starting bucket ops to source bucket
2021/12/15 20:03:53 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 20:03:58 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 20:04:03 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 20:04:08 Waiting for dst bucket item count to get to: 5000 curr count: 3326
2021/12/15 20:04:13 Waiting for dst bucket item count to get to: 5000 curr count: 3326
2021/12/15 20:04:13 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/15 20:04:18 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get deployed
2021/12/15 20:04:18 App: TestDeployUndeployLoopNonDefaultSettings got deployed
2021/12/15 20:04:18 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 20:04:18 Undeploying app: bucket_op_on_update
2021/12/15 20:04:18 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 20:04:23 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/15 20:04:28 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/15 20:04:33 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/15 20:04:38 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/15 20:04:38 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/15 20:04:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 20:04:38 No eventing-con process running
2021/12/15 20:04:40 Context: Bucket flush, response code: 200 dump: 
2021/12/15 20:04:47 Context: Bucket flush, response code: 200 dump: 
2021/12/15 20:05:22 Deploying app: TestDeployUndeployLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/15 20:05:23 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/15 20:05:23 Starting bucket ops to source bucket
2021/12/15 20:05:31 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 20:05:36 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 20:05:41 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 20:05:46 Waiting for dst bucket item count to get to: 5000 curr count: 2466
2021/12/15 20:05:51 Waiting for dst bucket item count to get to: 5000 curr count: 4966
2021/12/15 20:05:51 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/15 20:05:56 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get deployed
2021/12/15 20:05:56 App: TestDeployUndeployLoopNonDefaultSettings got deployed
2021/12/15 20:05:56 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 20:05:56 Undeploying app: bucket_op_on_update
2021/12/15 20:05:57 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 20:06:02 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/15 20:06:07 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/15 20:06:12 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/15 20:06:17 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/15 20:06:17 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/15 20:06:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 20:06:17 No eventing-con process running
2021/12/15 20:06:18 Context: Bucket flush, response code: 200 dump: 
2021/12/15 20:06:26 Context: Bucket flush, response code: 200 dump: 
2021/12/15 20:07:01 Deploying app: TestDeployUndeployLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/15 20:07:02 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/15 20:07:02 Starting bucket ops to source bucket
2021/12/15 20:07:10 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 20:07:15 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 20:07:20 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 20:07:25 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 20:07:25 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/15 20:07:30 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get deployed
2021/12/15 20:07:30 App: TestDeployUndeployLoopNonDefaultSettings got deployed
2021/12/15 20:07:30 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 20:07:30 Undeploying app: bucket_op_on_update
2021/12/15 20:07:30 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 20:07:35 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/15 20:07:40 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/15 20:07:45 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/15 20:07:50 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/15 20:07:50 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/15 20:07:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 20:07:50 No eventing-con process running
2021/12/15 20:07:52 Context: Bucket flush, response code: 200 dump: 
2021/12/15 20:07:59 Context: Bucket flush, response code: 200 dump: 
2021/12/15 20:08:34 Function: TestDeployUndeployLoopNonDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDeployUndeployLoopNonDefaultSettings deleting in the background"
}
--- PASS: TestDeployUndeployLoopNonDefaultSettings (510.96s)
=== RUN   TestOnUpdateN1QLOp
2021/12/15 20:08:44 Waiting for app: TestOnUpdateN1QLOp to get un-deployed
2021/12/15 20:08:44 App: TestOnUpdateN1QLOp got un-deployed
2021/12/15 20:08:44 No eventing-con process running
2021/12/15 20:08:44 Function: TestOnUpdateN1QLOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateN1QLOp not found"
 }
}
2021/12/15 20:08:46 Context: Bucket flush, response code: 200 dump: 
2021/12/15 20:08:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 20:08:53 Context: Bucket flush, response code: 200 dump: 
2021/12/15 20:08:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 20:08:58 Deploying app: TestOnUpdateN1QLOp. Item count src bucket: 0 dst bucket: 0
2021/12/15 20:08:59 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateN1QLOp' in metakv",
  "warnings": null
 }
}
2021/12/15 20:08:59 Starting bucket ops to source bucket
2021/12/15 20:09:07 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 20:09:12 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 20:09:17 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 20:09:22 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 20:09:27 Waiting for dst bucket item count to get to: 5000 curr count: 2512
2021/12/15 20:09:27 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/15 20:09:27 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 20:09:27 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 20:09:32 Waiting for app: TestOnUpdateN1QLOp to get un-deployed
2021/12/15 20:09:37 Waiting for app: TestOnUpdateN1QLOp to get un-deployed
2021/12/15 20:09:42 Waiting for app: TestOnUpdateN1QLOp to get un-deployed
2021/12/15 20:09:47 Waiting for app: TestOnUpdateN1QLOp to get un-deployed
2021/12/15 20:09:47 App: TestOnUpdateN1QLOp got un-deployed
2021/12/15 20:09:47 No eventing-con process running
2021/12/15 20:09:47 Function: TestOnUpdateN1QLOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateN1QLOp deleting in the background"
}
2021/12/15 20:09:49 Context: Bucket flush, response code: 200 dump: 
2021/12/15 20:09:59 Waiting for dst bucket item count to get to: 0 curr count: 4981
2021/12/15 20:10:04 Waiting for dst bucket item count to get to: 0 curr count: 2481
2021/12/15 20:10:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 20:10:06 Context: Bucket flush, response code: 200 dump: 
2021/12/15 20:10:16 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/15 20:10:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateN1QLOp (101.60s)
=== RUN   TestOnUpdateBucketOpDefaultSettings
2021/12/15 20:10:26 Waiting for app: TestOnUpdateBucketOpDefaultSettings to get un-deployed
2021/12/15 20:10:26 App: TestOnUpdateBucketOpDefaultSettings got un-deployed
2021/12/15 20:10:26 No eventing-con process running
2021/12/15 20:10:26 Function: TestOnUpdateBucketOpDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateBucketOpDefaultSettings not found"
 }
}
2021/12/15 20:10:27 Context: Bucket flush, response code: 200 dump: 
2021/12/15 20:10:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 20:10:34 Context: Bucket flush, response code: 200 dump: 
2021/12/15 20:10:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 20:10:39 Deploying app: TestOnUpdateBucketOpDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/15 20:10:40 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateBucketOpDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/15 20:10:40 Starting bucket ops to source bucket
2021/12/15 20:10:48 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 20:10:54 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 20:10:59 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 20:11:04 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 20:11:04 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/15 20:11:04 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 20:11:04 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 20:11:09 Waiting for app: TestOnUpdateBucketOpDefaultSettings to get un-deployed
2021/12/15 20:11:14 Waiting for app: TestOnUpdateBucketOpDefaultSettings to get un-deployed
2021/12/15 20:11:19 Waiting for app: TestOnUpdateBucketOpDefaultSettings to get un-deployed
2021/12/15 20:11:24 Waiting for app: TestOnUpdateBucketOpDefaultSettings to get un-deployed
2021/12/15 20:11:24 App: TestOnUpdateBucketOpDefaultSettings got un-deployed
2021/12/15 20:11:24 No eventing-con process running
2021/12/15 20:11:24 Function: TestOnUpdateBucketOpDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateBucketOpDefaultSettings deleting in the background"
}
2021/12/15 20:11:26 Context: Bucket flush, response code: 200 dump: 
2021/12/15 20:11:36 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/15 20:11:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 20:11:37 Context: Bucket flush, response code: 200 dump: 
2021/12/15 20:11:47 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/15 20:11:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateBucketOpDefaultSettings (91.63s)
=== RUN   TestOnUpdateBucketOpNonDefaultSettings
2021/12/15 20:11:57 Waiting for app: TestOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/15 20:11:57 App: TestOnUpdateBucketOpNonDefaultSettings got un-deployed
2021/12/15 20:11:57 No eventing-con process running
2021/12/15 20:11:57 Function: TestOnUpdateBucketOpNonDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateBucketOpNonDefaultSettings not found"
 }
}
2021/12/15 20:11:59 Context: Bucket flush, response code: 200 dump: 
2021/12/15 20:12:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 20:12:06 Context: Bucket flush, response code: 200 dump: 
2021/12/15 20:12:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 20:12:11 Deploying app: TestOnUpdateBucketOpNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/15 20:12:12 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateBucketOpNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/15 20:12:12 Starting bucket ops to source bucket
2021/12/15 20:12:20 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 20:12:25 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 20:12:30 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 20:12:35 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 20:12:35 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/15 20:12:35 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 20:12:35 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 20:12:40 Waiting for app: TestOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/15 20:12:45 Waiting for app: TestOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/15 20:12:50 Waiting for app: TestOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/15 20:12:55 Waiting for app: TestOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/15 20:12:55 App: TestOnUpdateBucketOpNonDefaultSettings got un-deployed
2021/12/15 20:12:55 No eventing-con process running
2021/12/15 20:12:56 Function: TestOnUpdateBucketOpNonDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateBucketOpNonDefaultSettings deleting in the background"
}
2021/12/15 20:12:57 Context: Bucket flush, response code: 200 dump: 
2021/12/15 20:13:07 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/15 20:13:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 20:13:09 Context: Bucket flush, response code: 200 dump: 
2021/12/15 20:13:19 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/15 20:13:24 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/15 20:13:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateBucketOpNonDefaultSettings (96.74s)
=== RUN   TestOnUpdateBucketOpDefaultSettings10K
2021/12/15 20:13:34 Waiting for app: TestOnUpdateBucketOpDefaultSettings10K to get un-deployed
2021/12/15 20:13:34 App: TestOnUpdateBucketOpDefaultSettings10K got un-deployed
2021/12/15 20:13:34 No eventing-con process running
2021/12/15 20:13:34 Function: TestOnUpdateBucketOpDefaultSettings10K delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateBucketOpDefaultSettings10K not found"
 }
}
2021/12/15 20:13:36 Context: Bucket flush, response code: 200 dump: 
2021/12/15 20:13:41 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 20:13:43 Context: Bucket flush, response code: 200 dump: 
2021/12/15 20:13:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 20:13:48 Deploying app: TestOnUpdateBucketOpDefaultSettings10K. Item count src bucket: 0 dst bucket: 0
2021/12/15 20:13:49 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateBucketOpDefaultSettings10K' in metakv",
  "warnings": null
 }
}
2021/12/15 20:13:49 Starting bucket ops to source bucket
2021/12/15 20:13:57 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 20:14:02 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 20:14:07 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 20:14:12 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 20:14:17 Waiting for dst bucket item count to get to: 5000 curr count: 2500
2021/12/15 20:14:17 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/15 20:14:17 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 20:14:17 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 20:14:22 Waiting for app: TestOnUpdateBucketOpDefaultSettings10K to get un-deployed
2021/12/15 20:14:27 Waiting for app: TestOnUpdateBucketOpDefaultSettings10K to get un-deployed
2021/12/15 20:14:32 Waiting for app: TestOnUpdateBucketOpDefaultSettings10K to get un-deployed
2021/12/15 20:14:37 Waiting for app: TestOnUpdateBucketOpDefaultSettings10K to get un-deployed
2021/12/15 20:14:37 App: TestOnUpdateBucketOpDefaultSettings10K got un-deployed
2021/12/15 20:14:37 No eventing-con process running
2021/12/15 20:14:37 Function: TestOnUpdateBucketOpDefaultSettings10K delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateBucketOpDefaultSettings10K deleting in the background"
}
2021/12/15 20:14:39 Context: Bucket flush, response code: 200 dump: 
2021/12/15 20:14:49 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/15 20:14:54 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/15 20:14:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 20:14:56 Context: Bucket flush, response code: 200 dump: 
2021/12/15 20:15:06 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/15 20:15:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateBucketOpDefaultSettings10K (101.64s)
=== RUN   TestOnUpdateBucketOpDefaultSettings100K
2021/12/15 20:15:16 Waiting for app: TestOnUpdateBucketOpDefaultSettings100K to get un-deployed
2021/12/15 20:15:16 App: TestOnUpdateBucketOpDefaultSettings100K got un-deployed
2021/12/15 20:15:16 No eventing-con process running
2021/12/15 20:15:16 Function: TestOnUpdateBucketOpDefaultSettings100K delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateBucketOpDefaultSettings100K not found"
 }
}
2021/12/15 20:15:18 Context: Bucket flush, response code: 200 dump: 
2021/12/15 20:15:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 20:15:24 Context: Bucket flush, response code: 200 dump: 
2021/12/15 20:15:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 20:15:29 Deploying app: TestOnUpdateBucketOpDefaultSettings100K. Item count src bucket: 0 dst bucket: 0
2021/12/15 20:15:31 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateBucketOpDefaultSettings100K' in metakv",
  "warnings": null
 }
}
2021/12/15 20:15:31 Starting bucket ops to source bucket
2021/12/15 20:15:39 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 20:15:44 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 20:15:49 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 20:15:54 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 20:15:54 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/15 20:15:54 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 20:15:54 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 20:15:59 Waiting for app: TestOnUpdateBucketOpDefaultSettings100K to get un-deployed
2021/12/15 20:16:04 Waiting for app: TestOnUpdateBucketOpDefaultSettings100K to get un-deployed
2021/12/15 20:16:09 Waiting for app: TestOnUpdateBucketOpDefaultSettings100K to get un-deployed
2021/12/15 20:16:14 Waiting for app: TestOnUpdateBucketOpDefaultSettings100K to get un-deployed
2021/12/15 20:16:14 App: TestOnUpdateBucketOpDefaultSettings100K got un-deployed
2021/12/15 20:16:14 No eventing-con process running
2021/12/15 20:16:14 Function: TestOnUpdateBucketOpDefaultSettings100K delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateBucketOpDefaultSettings100K deleting in the background"
}
2021/12/15 20:16:16 Context: Bucket flush, response code: 200 dump: 
2021/12/15 20:16:26 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/15 20:16:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 20:16:28 Context: Bucket flush, response code: 200 dump: 
2021/12/15 20:16:38 Waiting for dst bucket item count to get to: 0 curr count: 3026
2021/12/15 20:16:43 Waiting for dst bucket item count to get to: 0 curr count: 526
2021/12/15 20:16:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateBucketOpDefaultSettings100K (97.07s)
=== RUN   TestOnDeleteBucketOp
2021/12/15 20:16:53 Waiting for app: TestOnDeleteBucketOp to get un-deployed
2021/12/15 20:16:53 App: TestOnDeleteBucketOp got un-deployed
2021/12/15 20:16:53 No eventing-con process running
2021/12/15 20:16:53 Function: TestOnDeleteBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnDeleteBucketOp not found"
 }
}
2021/12/15 20:16:55 Context: Bucket flush, response code: 200 dump: 
2021/12/15 20:17:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 20:17:01 Context: Bucket flush, response code: 200 dump: 
2021/12/15 20:17:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 20:17:06 Deploying app: TestOnDeleteBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/15 20:17:08 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnDeleteBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/15 20:17:08 Starting bucket ops to source bucket
2021/12/15 20:17:18 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 20:17:23 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 20:17:28 Waiting for dst bucket item count to get to: 5000 curr count: 519
2021/12/15 20:17:33 Waiting for dst bucket item count to get to: 5000 curr count: 519
2021/12/15 20:17:33 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/15 20:17:33 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 20:17:34 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 20:17:39 Waiting for app: TestOnDeleteBucketOp to get un-deployed
2021/12/15 20:17:44 Waiting for app: TestOnDeleteBucketOp to get un-deployed
2021/12/15 20:17:49 Waiting for app: TestOnDeleteBucketOp to get un-deployed
2021/12/15 20:17:54 Waiting for app: TestOnDeleteBucketOp to get un-deployed
2021/12/15 20:17:54 App: TestOnDeleteBucketOp got un-deployed
2021/12/15 20:17:54 No eventing-con process running
2021/12/15 20:17:54 Function: TestOnDeleteBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnDeleteBucketOp deleting in the background"
}
2021/12/15 20:17:55 Context: Bucket flush, response code: 200 dump: 
2021/12/15 20:18:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 20:18:03 Context: Bucket flush, response code: 200 dump: 
2021/12/15 20:18:13 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/15 20:18:18 Waiting for dst bucket item count to get to: 0 curr count: 2496
2021/12/15 20:18:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnDeleteBucketOp (94.96s)
=== RUN   TestOnDeleteBucketOp5K
2021/12/15 20:18:28 Waiting for app: TestOnDeleteBucketOp5K to get un-deployed
2021/12/15 20:18:28 App: TestOnDeleteBucketOp5K got un-deployed
2021/12/15 20:18:28 No eventing-con process running
2021/12/15 20:18:28 Function: TestOnDeleteBucketOp5K delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnDeleteBucketOp5K not found"
 }
}
2021/12/15 20:18:30 Context: Bucket flush, response code: 200 dump: 
2021/12/15 20:18:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 20:18:36 Context: Bucket flush, response code: 200 dump: 
2021/12/15 20:18:41 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 20:18:41 Deploying app: TestOnDeleteBucketOp5K. Item count src bucket: 0 dst bucket: 0
2021/12/15 20:18:43 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnDeleteBucketOp5K' in metakv",
  "warnings": null
 }
}
2021/12/15 20:18:43 Starting bucket ops to source bucket
2021/12/15 20:18:54 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 20:18:59 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 20:19:04 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 20:19:04 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/15 20:19:04 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 20:19:04 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 20:19:09 Waiting for app: TestOnDeleteBucketOp5K to get un-deployed
2021/12/15 20:19:14 Waiting for app: TestOnDeleteBucketOp5K to get un-deployed
2021/12/15 20:19:19 Waiting for app: TestOnDeleteBucketOp5K to get un-deployed
2021/12/15 20:19:24 Waiting for app: TestOnDeleteBucketOp5K to get un-deployed
2021/12/15 20:19:24 App: TestOnDeleteBucketOp5K got un-deployed
2021/12/15 20:19:24 No eventing-con process running
2021/12/15 20:19:24 Function: TestOnDeleteBucketOp5K delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnDeleteBucketOp5K deleting in the background"
}
2021/12/15 20:19:26 Context: Bucket flush, response code: 200 dump: 
2021/12/15 20:19:31 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 20:19:33 Context: Bucket flush, response code: 200 dump: 
2021/12/15 20:19:43 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/15 20:19:48 Waiting for dst bucket item count to get to: 0 curr count: 2496
2021/12/15 20:19:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnDeleteBucketOp5K (90.09s)
=== RUN   TestTimerBucketOp
2021/12/15 20:19:58 Waiting for app: TestTimerBucketOp to get un-deployed
2021/12/15 20:19:58 App: TestTimerBucketOp got un-deployed
2021/12/15 20:19:58 No eventing-con process running
2021/12/15 20:19:58 Function: TestTimerBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestTimerBucketOp not found"
 }
}
2021/12/15 20:20:00 Context: Bucket flush, response code: 200 dump: 
2021/12/15 20:20:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 20:20:06 Context: Bucket flush, response code: 200 dump: 
2021/12/15 20:20:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 20:20:11 Deploying app: TestTimerBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/15 20:20:13 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestTimerBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/15 20:20:13 Starting bucket ops to source bucket
2021/12/15 20:20:21 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 20:20:26 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 20:20:31 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 20:20:36 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 20:20:41 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 20:20:46 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 20:20:51 Waiting for dst bucket item count to get to: 5000 curr count: 458
2021/12/15 20:20:56 Waiting for dst bucket item count to get to: 5000 curr count: 458
2021/12/15 20:20:56 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/15 20:20:56 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 20:20:56 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 20:21:01 Waiting for app: TestTimerBucketOp to get un-deployed
2021/12/15 20:21:06 Waiting for app: TestTimerBucketOp to get un-deployed
2021/12/15 20:21:11 Waiting for app: TestTimerBucketOp to get un-deployed
2021/12/15 20:21:16 Waiting for app: TestTimerBucketOp to get un-deployed
2021/12/15 20:21:16 App: TestTimerBucketOp got un-deployed
2021/12/15 20:21:16 No eventing-con process running
2021/12/15 20:21:16 Function: TestTimerBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestTimerBucketOp deleting in the background"
}
2021/12/15 20:21:18 Context: Bucket flush, response code: 200 dump: 
2021/12/15 20:21:28 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/15 20:21:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 20:21:29 Context: Bucket flush, response code: 200 dump: 
2021/12/15 20:21:39 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/15 20:21:44 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/15 20:21:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestTimerBucketOp (116.60s)
=== RUN   TestTimerInPastBucketOp
2021/12/15 20:21:54 Waiting for app: TestTimerInPastBucketOp to get un-deployed
2021/12/15 20:21:54 App: TestTimerInPastBucketOp got un-deployed
2021/12/15 20:21:54 No eventing-con process running
2021/12/15 20:21:54 Function: TestTimerInPastBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestTimerInPastBucketOp not found"
 }
}
2021/12/15 20:21:56 Context: Bucket flush, response code: 200 dump: 
2021/12/15 20:22:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 20:22:03 Context: Bucket flush, response code: 200 dump: 
2021/12/15 20:22:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 20:22:08 Deploying app: TestTimerInPastBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/15 20:22:09 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestTimerInPastBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/15 20:22:09 Starting bucket ops to source bucket
2021/12/15 20:22:17 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 20:22:22 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 20:22:27 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 20:22:32 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 20:22:37 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 20:22:42 Waiting for dst bucket item count to get to: 5000 curr count: 539
2021/12/15 20:22:47 Waiting for dst bucket item count to get to: 5000 curr count: 2579
2021/12/15 20:22:52 Waiting for dst bucket item count to get to: 5000 curr count: 4101
2021/12/15 20:22:57 Waiting for dst bucket item count to get to: 5000 curr count: 4561
2021/12/15 20:22:57 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/15 20:22:57 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 20:22:57 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 20:23:02 Waiting for app: TestTimerInPastBucketOp to get un-deployed
2021/12/15 20:23:07 Waiting for app: TestTimerInPastBucketOp to get un-deployed
2021/12/15 20:23:12 Waiting for app: TestTimerInPastBucketOp to get un-deployed
2021/12/15 20:23:17 Waiting for app: TestTimerInPastBucketOp to get un-deployed
2021/12/15 20:23:17 App: TestTimerInPastBucketOp got un-deployed
2021/12/15 20:23:17 No eventing-con process running
2021/12/15 20:23:17 Function: TestTimerInPastBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestTimerInPastBucketOp deleting in the background"
}
2021/12/15 20:23:19 Context: Bucket flush, response code: 200 dump: 
2021/12/15 20:23:29 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/15 20:23:34 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/15 20:23:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 20:23:36 Context: Bucket flush, response code: 200 dump: 
2021/12/15 20:23:46 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/15 20:23:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestTimerInPastBucketOp (121.56s)
=== RUN   TestDeployUndeployLoopTimer
2021/12/15 20:23:56 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/15 20:23:56 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/15 20:23:56 No eventing-con process running
2021/12/15 20:23:56 Function: TestDeployUndeployLoopTimer delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDeployUndeployLoopTimer not found"
 }
}
2021/12/15 20:23:58 Context: Bucket flush, response code: 200 dump: 
2021/12/15 20:24:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 20:24:05 Context: Bucket flush, response code: 200 dump: 
2021/12/15 20:24:10 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 20:24:10 Deploying app: TestDeployUndeployLoopTimer. Item count src bucket: 0 dst bucket: 0
2021/12/15 20:24:11 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopTimer' in metakv",
  "warnings": null
 }
}
2021/12/15 20:24:11 Starting bucket ops to source bucket
2021/12/15 20:24:19 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 20:24:24 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 20:24:29 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 20:24:34 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 20:24:39 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 20:24:44 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 20:24:49 Waiting for dst bucket item count to get to: 5000 curr count: 2473
2021/12/15 20:24:54 Waiting for dst bucket item count to get to: 5000 curr count: 2473
2021/12/15 20:24:54 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/15 20:24:59 Waiting for app: TestDeployUndeployLoopTimer to get deployed
2021/12/15 20:24:59 App: TestDeployUndeployLoopTimer got deployed
2021/12/15 20:24:59 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 20:24:59 Undeploying app: bucket_op_with_timer
2021/12/15 20:24:59 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 20:25:04 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/15 20:25:09 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/15 20:25:14 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/15 20:25:19 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/15 20:25:19 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/15 20:25:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 20:25:19 No eventing-con process running
2021/12/15 20:25:21 Context: Bucket flush, response code: 200 dump: 
2021/12/15 20:25:28 Context: Bucket flush, response code: 200 dump: 
2021/12/15 20:26:03 Deploying app: TestDeployUndeployLoopTimer. Item count src bucket: 0 dst bucket: 0
2021/12/15 20:26:04 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopTimer' in metakv",
  "warnings": null
 }
}
2021/12/15 20:26:04 Starting bucket ops to source bucket
2021/12/15 20:26:12 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 20:26:17 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 20:26:22 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 20:26:27 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 20:26:32 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 20:26:37 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 20:26:42 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 20:26:47 Waiting for dst bucket item count to get to: 5000 curr count: 2500
2021/12/15 20:26:47 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/15 20:26:52 Waiting for app: TestDeployUndeployLoopTimer to get deployed
2021/12/15 20:26:52 App: TestDeployUndeployLoopTimer got deployed
2021/12/15 20:26:52 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 20:26:52 Undeploying app: bucket_op_with_timer
2021/12/15 20:26:52 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 20:26:57 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/15 20:27:02 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/15 20:27:07 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/15 20:27:12 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/15 20:27:12 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/15 20:27:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 20:27:12 No eventing-con process running
2021/12/15 20:27:14 Context: Bucket flush, response code: 200 dump: 
2021/12/15 20:27:21 Context: Bucket flush, response code: 200 dump: 
2021/12/15 20:27:56 Deploying app: TestDeployUndeployLoopTimer. Item count src bucket: 0 dst bucket: 0
2021/12/15 20:27:57 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopTimer' in metakv",
  "warnings": null
 }
}
2021/12/15 20:27:57 Starting bucket ops to source bucket
2021/12/15 20:28:05 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 20:28:10 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 20:28:15 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 20:28:20 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 20:28:25 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 20:28:30 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 20:28:35 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 20:28:35 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/15 20:28:40 Waiting for app: TestDeployUndeployLoopTimer to get deployed
2021/12/15 20:28:40 App: TestDeployUndeployLoopTimer got deployed
2021/12/15 20:28:40 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 20:28:40 Undeploying app: bucket_op_with_timer
2021/12/15 20:28:41 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 20:28:46 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/15 20:28:51 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/15 20:28:56 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/15 20:29:01 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/15 20:29:01 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/15 20:29:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 20:29:01 No eventing-con process running
2021/12/15 20:29:02 Context: Bucket flush, response code: 200 dump: 
2021/12/15 20:29:10 Context: Bucket flush, response code: 200 dump: 
2021/12/15 20:29:45 Deploying app: TestDeployUndeployLoopTimer. Item count src bucket: 0 dst bucket: 0
2021/12/15 20:29:46 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopTimer' in metakv",
  "warnings": null
 }
}
2021/12/15 20:29:46 Starting bucket ops to source bucket
2021/12/15 20:29:54 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 20:29:59 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 20:30:04 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 20:30:09 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 20:30:14 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 20:30:19 Waiting for dst bucket item count to get to: 5000 curr count: 362
2021/12/15 20:30:24 Waiting for dst bucket item count to get to: 5000 curr count: 362
2021/12/15 20:30:24 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/15 20:30:29 Waiting for app: TestDeployUndeployLoopTimer to get deployed
2021/12/15 20:30:29 App: TestDeployUndeployLoopTimer got deployed
2021/12/15 20:30:29 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 20:30:29 Undeploying app: bucket_op_with_timer
2021/12/15 20:30:29 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 20:30:34 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/15 20:30:39 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/15 20:30:44 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/15 20:30:49 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/15 20:30:49 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/15 20:30:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 20:30:49 No eventing-con process running
2021/12/15 20:30:51 Context: Bucket flush, response code: 200 dump: 
2021/12/15 20:30:58 Context: Bucket flush, response code: 200 dump: 
2021/12/15 20:31:33 Deploying app: TestDeployUndeployLoopTimer. Item count src bucket: 0 dst bucket: 0
2021/12/15 20:31:34 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopTimer' in metakv",
  "warnings": null
 }
}
2021/12/15 20:31:34 Starting bucket ops to source bucket
2021/12/15 20:31:42 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 20:31:47 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 20:31:52 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 20:31:57 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 20:32:02 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 20:32:07 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 20:32:12 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 20:32:17 Waiting for dst bucket item count to get to: 5000 curr count: 2500
2021/12/15 20:32:17 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/15 20:32:22 Waiting for app: TestDeployUndeployLoopTimer to get deployed
2021/12/15 20:32:22 App: TestDeployUndeployLoopTimer got deployed
2021/12/15 20:32:22 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 20:32:22 Undeploying app: bucket_op_with_timer
2021/12/15 20:32:23 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 20:32:28 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/15 20:32:33 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/15 20:32:38 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/15 20:32:43 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/15 20:32:43 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/15 20:32:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 20:32:43 No eventing-con process running
2021/12/15 20:32:44 Context: Bucket flush, response code: 200 dump: 
2021/12/15 20:32:52 Context: Bucket flush, response code: 200 dump: 
2021/12/15 20:33:27 Function: TestDeployUndeployLoopTimer delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDeployUndeployLoopTimer deleting in the background"
}
--- PASS: TestDeployUndeployLoopTimer (580.73s)
=== RUN   TestMultipleHandlers
2021/12/15 20:33:37 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get un-deployed
2021/12/15 20:33:37 App: TestMultipleHandlers_bucket_op_on_update got un-deployed
2021/12/15 20:33:37 No eventing-con process running
2021/12/15 20:33:37 Function: TestMultipleHandlers_bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMultipleHandlers_bucket_op_on_update not found"
 }
}
2021/12/15 20:33:39 Context: Bucket flush, response code: 200 dump: 
2021/12/15 20:33:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 20:33:45 Context: Bucket flush, response code: 200 dump: 
2021/12/15 20:33:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 20:33:55 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get un-deployed
2021/12/15 20:33:55 App: TestMultipleHandlers_bucket_op_on_delete got un-deployed
2021/12/15 20:33:55 No eventing-con process running
2021/12/15 20:33:55 Function: TestMultipleHandlers_bucket_op_on_delete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMultipleHandlers_bucket_op_on_delete not found"
 }
}
2021/12/15 20:33:57 Context: Bucket flush, response code: 200 dump: 
2021/12/15 20:34:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 20:34:04 Context: Bucket flush, response code: 200 dump: 
2021/12/15 20:34:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 20:34:09 Deploying app: TestMultipleHandlers_bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/15 20:34:10 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMultipleHandlers_bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/15 20:34:10 Deploying app: TestMultipleHandlers_bucket_op_on_delete. Item count src bucket: 0 dst bucket: 0
2021/12/15 20:34:11 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMultipleHandlers_bucket_op_on_delete' in metakv",
  "warnings": null
 }
}
2021/12/15 20:34:16 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get deployed
2021/12/15 20:34:21 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get deployed
2021/12/15 20:34:26 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get deployed
2021/12/15 20:34:31 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get deployed
2021/12/15 20:34:31 App: TestMultipleHandlers_bucket_op_on_update got deployed
2021/12/15 20:34:36 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get deployed
2021/12/15 20:34:41 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get deployed
2021/12/15 20:34:46 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get deployed
2021/12/15 20:34:46 App: TestMultipleHandlers_bucket_op_on_delete got deployed
2021/12/15 20:34:46 Starting bucket ops to source bucket
2021/12/15 20:34:56 Waiting for dst bucket item count to get to: 5000 curr count: 1949
2021/12/15 20:35:01 Waiting for dst bucket item count to get to: 5000 curr count: 1949
2021/12/15 20:35:01 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/15 20:35:01 Starting bucket ops to source bucket
2021/12/15 20:35:16 Waiting for dst bucket item count to get to: 10000 curr count: 7162
2021/12/15 20:35:21 Waiting for dst bucket item count to get to: 10000 curr count: 7162
2021/12/15 20:35:21 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/15 20:35:21 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 20:35:21 Function: TestMultipleHandlers_bucket_op_on_update update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/15 20:35:21 Function: TestMultipleHandlers_bucket_op_on_delete update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/15 20:35:21 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 20:35:26 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get un-deployed
2021/12/15 20:35:31 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get un-deployed
2021/12/15 20:35:36 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get un-deployed
2021/12/15 20:35:41 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get un-deployed
2021/12/15 20:35:41 App: TestMultipleHandlers_bucket_op_on_update got un-deployed
2021/12/15 20:35:41 No eventing-con process running
2021/12/15 20:35:42 Function: TestMultipleHandlers_bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMultipleHandlers_bucket_op_on_update deleting in the background"
}
2021/12/15 20:35:43 Context: Bucket flush, response code: 200 dump: 
2021/12/15 20:35:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 20:35:51 Context: Bucket flush, response code: 200 dump: 
2021/12/15 20:36:01 Waiting for dst bucket item count to get to: 0 curr count: 7554
2021/12/15 20:36:06 Waiting for dst bucket item count to get to: 0 curr count: 7554
2021/12/15 20:36:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 20:36:06 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 20:36:11 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get un-deployed
2021/12/15 20:36:16 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get un-deployed
2021/12/15 20:36:21 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get un-deployed
2021/12/15 20:36:26 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get un-deployed
2021/12/15 20:36:26 App: TestMultipleHandlers_bucket_op_on_delete got un-deployed
2021/12/15 20:36:26 No eventing-con process running
2021/12/15 20:36:26 Function: TestMultipleHandlers_bucket_op_on_delete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMultipleHandlers_bucket_op_on_delete deleting in the background"
}
2021/12/15 20:36:28 Context: Bucket flush, response code: 200 dump: 
2021/12/15 20:36:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 20:36:34 Context: Bucket flush, response code: 200 dump: 
2021/12/15 20:36:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestMultipleHandlers (192.61s)
=== RUN   TestPauseResumeLoopDefaultSettings
2021/12/15 20:36:49 Waiting for app: TestPauseResumeLoopDefaultSettings to get un-deployed
2021/12/15 20:36:49 App: TestPauseResumeLoopDefaultSettings got un-deployed
2021/12/15 20:36:49 No eventing-con process running
2021/12/15 20:36:49 Function: TestPauseResumeLoopDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestPauseResumeLoopDefaultSettings not found"
 }
}
2021/12/15 20:36:51 Context: Bucket flush, response code: 200 dump: 
2021/12/15 20:36:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 20:36:58 Context: Bucket flush, response code: 200 dump: 
2021/12/15 20:37:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 20:37:03 Deploying app: TestPauseResumeLoopDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/15 20:37:04 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestPauseResumeLoopDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/15 20:37:04 Starting bucket ops to source bucket
2021/12/15 20:37:12 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 20:37:17 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 20:37:22 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 20:37:27 Waiting for dst bucket item count to get to: 5000 curr count: 1861
2021/12/15 20:37:32 Waiting for dst bucket item count to get to: 5000 curr count: 4361
2021/12/15 20:37:32 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/15 20:37:32 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 20:37:37 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to deployed
2021/12/15 20:37:37 Function: TestPauseResumeLoopDefaultSettings status changed to deployed
2021/12/15 20:37:37 Pausing app: bucket_op_on_update
2021/12/15 20:37:37 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/15 20:37:42 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to paused
2021/12/15 20:37:42 Function: TestPauseResumeLoopDefaultSettings status changed to paused
2021/12/15 20:37:42 Resuming app: bucket_op_on_update
2021/12/15 20:37:43 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/15 20:37:43 Starting bucket ops to source bucket
2021/12/15 20:37:51 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/15 20:37:56 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/15 20:38:01 Waiting for dst bucket item count to get to: 10000 curr count: 5056
2021/12/15 20:38:06 Waiting for dst bucket item count to get to: 10000 curr count: 7556
2021/12/15 20:38:06 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/15 20:38:06 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 20:38:12 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to deployed
2021/12/15 20:38:12 Function: TestPauseResumeLoopDefaultSettings status changed to deployed
2021/12/15 20:38:12 Pausing app: bucket_op_on_update
2021/12/15 20:38:12 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/15 20:38:17 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to paused
2021/12/15 20:38:17 Function: TestPauseResumeLoopDefaultSettings status changed to paused
2021/12/15 20:38:17 Resuming app: bucket_op_on_update
2021/12/15 20:38:18 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/15 20:38:18 Starting bucket ops to source bucket
2021/12/15 20:38:26 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/15 20:38:31 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/15 20:38:31 src & dst bucket item count matched up. src bucket count: 15000 dst bucket count: 15000
2021/12/15 20:38:31 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 20:38:36 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to deployed
2021/12/15 20:38:36 Function: TestPauseResumeLoopDefaultSettings status changed to deployed
2021/12/15 20:38:36 Pausing app: bucket_op_on_update
2021/12/15 20:38:36 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/15 20:38:41 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to paused
2021/12/15 20:38:41 Function: TestPauseResumeLoopDefaultSettings status changed to paused
2021/12/15 20:38:41 Resuming app: bucket_op_on_update
2021/12/15 20:38:42 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/15 20:38:42 Starting bucket ops to source bucket
2021/12/15 20:38:50 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/15 20:38:55 Waiting for dst bucket item count to get to: 20000 curr count: 15664
2021/12/15 20:39:00 Waiting for dst bucket item count to get to: 20000 curr count: 15664
2021/12/15 20:39:00 src & dst bucket item count matched up. src bucket count: 20000 dst bucket count: 20000
2021/12/15 20:39:00 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 20:39:05 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to deployed
2021/12/15 20:39:05 Function: TestPauseResumeLoopDefaultSettings status changed to deployed
2021/12/15 20:39:05 Pausing app: bucket_op_on_update
2021/12/15 20:39:05 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/15 20:39:10 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to paused
2021/12/15 20:39:10 Function: TestPauseResumeLoopDefaultSettings status changed to paused
2021/12/15 20:39:10 Resuming app: bucket_op_on_update
2021/12/15 20:39:11 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/15 20:39:11 Starting bucket ops to source bucket
2021/12/15 20:39:19 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/15 20:39:24 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/15 20:39:29 Waiting for dst bucket item count to get to: 25000 curr count: 21981
2021/12/15 20:39:34 Waiting for dst bucket item count to get to: 25000 curr count: 21981
2021/12/15 20:39:34 src & dst bucket item count matched up. src bucket count: 25000 dst bucket count: 25000
2021/12/15 20:39:34 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 20:39:39 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to deployed
2021/12/15 20:39:39 Function: TestPauseResumeLoopDefaultSettings status changed to deployed
2021/12/15 20:39:39 Pausing app: bucket_op_on_update
2021/12/15 20:39:40 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/15 20:39:45 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to paused
2021/12/15 20:39:45 Function: TestPauseResumeLoopDefaultSettings status changed to paused
2021/12/15 20:39:45 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 20:39:50 Waiting for app: TestPauseResumeLoopDefaultSettings to get un-deployed
2021/12/15 20:39:55 Waiting for app: TestPauseResumeLoopDefaultSettings to get un-deployed
2021/12/15 20:40:00 Waiting for app: TestPauseResumeLoopDefaultSettings to get un-deployed
2021/12/15 20:40:05 Waiting for app: TestPauseResumeLoopDefaultSettings to get un-deployed
2021/12/15 20:40:05 App: TestPauseResumeLoopDefaultSettings got un-deployed
2021/12/15 20:40:05 No eventing-con process running
2021/12/15 20:40:05 Function: TestPauseResumeLoopDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestPauseResumeLoopDefaultSettings deleting in the background"
}
2021/12/15 20:40:07 Context: Bucket flush, response code: 200 dump: 
2021/12/15 20:40:17 Waiting for dst bucket item count to get to: 0 curr count: 25000
2021/12/15 20:40:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 20:40:19 Context: Bucket flush, response code: 200 dump: 
2021/12/15 20:40:29 Waiting for dst bucket item count to get to: 0 curr count: 21867
2021/12/15 20:40:34 Waiting for dst bucket item count to get to: 0 curr count: 9367
2021/12/15 20:40:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestPauseResumeLoopDefaultSettings (234.21s)
=== RUN   TestPauseResumeLoopNonDefaultSettings
2021/12/15 20:40:44 Waiting for app: TestPauseResumeLoopNonDefaultSettings to get un-deployed
2021/12/15 20:40:44 App: TestPauseResumeLoopNonDefaultSettings got un-deployed
2021/12/15 20:40:44 No eventing-con process running
2021/12/15 20:40:44 Function: TestPauseResumeLoopNonDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestPauseResumeLoopNonDefaultSettings not found"
 }
}
2021/12/15 20:40:45 Context: Bucket flush, response code: 200 dump: 
2021/12/15 20:40:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 20:40:52 Context: Bucket flush, response code: 200 dump: 
2021/12/15 20:40:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 20:40:57 Deploying app: TestPauseResumeLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/15 20:40:58 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestPauseResumeLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/15 20:40:58 Starting bucket ops to source bucket
2021/12/15 20:41:06 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 20:41:11 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 20:41:16 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 20:41:21 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 20:41:26 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 20:41:31 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 20:41:36 Waiting for dst bucket item count to get to: 5000 curr count: 2500
2021/12/15 20:41:36 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/15 20:41:36 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 20:41:41 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to deployed
2021/12/15 20:41:41 Function: TestPauseResumeLoopNonDefaultSettings status changed to deployed
2021/12/15 20:41:41 Pausing app: bucket_op_with_timer
2021/12/15 20:41:41 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/15 20:41:46 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to paused
2021/12/15 20:41:47 Function: TestPauseResumeLoopNonDefaultSettings status changed to paused
2021/12/15 20:41:47 Resuming app: bucket_op_with_timer
2021/12/15 20:41:48 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/15 20:41:48 Starting bucket ops to source bucket
2021/12/15 20:41:56 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/15 20:42:01 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/15 20:42:06 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/15 20:42:11 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/15 20:42:16 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/15 20:42:21 Waiting for dst bucket item count to get to: 10000 curr count: 5773
2021/12/15 20:42:26 Waiting for dst bucket item count to get to: 10000 curr count: 8273
2021/12/15 20:42:26 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/15 20:42:26 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 20:42:31 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to deployed
2021/12/15 20:42:31 Function: TestPauseResumeLoopNonDefaultSettings status changed to deployed
2021/12/15 20:42:31 Pausing app: bucket_op_with_timer
2021/12/15 20:42:31 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/15 20:42:36 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to paused
2021/12/15 20:42:36 Function: TestPauseResumeLoopNonDefaultSettings status changed to paused
2021/12/15 20:42:36 Resuming app: bucket_op_with_timer
2021/12/15 20:42:37 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/15 20:42:37 Starting bucket ops to source bucket
2021/12/15 20:42:45 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/15 20:42:50 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/15 20:42:55 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/15 20:43:00 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/15 20:43:05 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/15 20:43:10 Waiting for dst bucket item count to get to: 15000 curr count: 11389
2021/12/15 20:43:15 Waiting for dst bucket item count to get to: 15000 curr count: 13889
2021/12/15 20:43:15 src & dst bucket item count matched up. src bucket count: 15000 dst bucket count: 15000
2021/12/15 20:43:15 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 20:43:20 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to deployed
2021/12/15 20:43:20 Function: TestPauseResumeLoopNonDefaultSettings status changed to deployed
2021/12/15 20:43:20 Pausing app: bucket_op_with_timer
2021/12/15 20:43:20 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/15 20:43:25 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to paused
2021/12/15 20:43:25 Function: TestPauseResumeLoopNonDefaultSettings status changed to paused
2021/12/15 20:43:25 Resuming app: bucket_op_with_timer
2021/12/15 20:43:26 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/15 20:43:26 Starting bucket ops to source bucket
2021/12/15 20:43:34 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/15 20:43:39 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/15 20:43:44 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/15 20:43:49 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/15 20:43:54 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/15 20:43:59 Waiting for dst bucket item count to get to: 20000 curr count: 17308
2021/12/15 20:44:04 Waiting for dst bucket item count to get to: 20000 curr count: 17308
2021/12/15 20:44:05 src & dst bucket item count matched up. src bucket count: 20000 dst bucket count: 20000
2021/12/15 20:44:05 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 20:44:10 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to deployed
2021/12/15 20:44:10 Function: TestPauseResumeLoopNonDefaultSettings status changed to deployed
2021/12/15 20:44:10 Pausing app: bucket_op_with_timer
2021/12/15 20:44:10 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/15 20:44:15 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to paused
2021/12/15 20:44:15 Function: TestPauseResumeLoopNonDefaultSettings status changed to paused
2021/12/15 20:44:15 Resuming app: bucket_op_with_timer
2021/12/15 20:44:16 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/15 20:44:16 Starting bucket ops to source bucket
2021/12/15 20:44:24 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/15 20:44:29 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/15 20:44:34 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/15 20:44:39 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/15 20:44:44 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/15 20:44:49 Waiting for dst bucket item count to get to: 25000 curr count: 22719
2021/12/15 20:44:54 Waiting for dst bucket item count to get to: 25000 curr count: 22719
2021/12/15 20:44:54 src & dst bucket item count matched up. src bucket count: 25000 dst bucket count: 25000
2021/12/15 20:44:54 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 20:44:59 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to deployed
2021/12/15 20:44:59 Function: TestPauseResumeLoopNonDefaultSettings status changed to deployed
2021/12/15 20:44:59 Pausing app: bucket_op_with_timer
2021/12/15 20:44:59 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/15 20:45:04 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to paused
2021/12/15 20:45:04 Function: TestPauseResumeLoopNonDefaultSettings status changed to paused
2021/12/15 20:45:04 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 20:45:09 Waiting for app: TestPauseResumeLoopNonDefaultSettings to get un-deployed
2021/12/15 20:45:14 Waiting for app: TestPauseResumeLoopNonDefaultSettings to get un-deployed
2021/12/15 20:45:19 Waiting for app: TestPauseResumeLoopNonDefaultSettings to get un-deployed
2021/12/15 20:45:24 Waiting for app: TestPauseResumeLoopNonDefaultSettings to get un-deployed
2021/12/15 20:45:24 App: TestPauseResumeLoopNonDefaultSettings got un-deployed
2021/12/15 20:45:24 No eventing-con process running
2021/12/15 20:45:24 Function: TestPauseResumeLoopNonDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestPauseResumeLoopNonDefaultSettings deleting in the background"
}
2021/12/15 20:45:26 Context: Bucket flush, response code: 200 dump: 
2021/12/15 20:45:36 Waiting for dst bucket item count to get to: 0 curr count: 25000
2021/12/15 20:45:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 20:45:38 Context: Bucket flush, response code: 200 dump: 
2021/12/15 20:45:48 Waiting for dst bucket item count to get to: 0 curr count: 16074
2021/12/15 20:45:53 Waiting for dst bucket item count to get to: 0 curr count: 3574
2021/12/15 20:45:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestPauseResumeLoopNonDefaultSettings (319.29s)
=== RUN   TestPauseAndResumeWithWorkerCountChange
2021/12/15 20:46:03 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get un-deployed
2021/12/15 20:46:03 App: TestPauseAndResumeWithWorkerCountChange got un-deployed
2021/12/15 20:46:03 No eventing-con process running
2021/12/15 20:46:03 Function: TestPauseAndResumeWithWorkerCountChange delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestPauseAndResumeWithWorkerCountChange not found"
 }
}
2021/12/15 20:46:05 Context: Bucket flush, response code: 200 dump: 
2021/12/15 20:46:10 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 20:46:11 Context: Bucket flush, response code: 200 dump: 
2021/12/15 20:46:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 20:46:16 Deploying app: TestPauseAndResumeWithWorkerCountChange. Item count src bucket: 0 dst bucket: 0
2021/12/15 20:46:18 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestPauseAndResumeWithWorkerCountChange' in metakv",
  "warnings": null
 }
}
2021/12/15 20:46:23 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get deployed
2021/12/15 20:46:28 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get deployed
2021/12/15 20:46:33 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get deployed
2021/12/15 20:46:38 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get deployed
2021/12/15 20:46:38 App: TestPauseAndResumeWithWorkerCountChange got deployed
2021/12/15 20:46:38 Starting bucket ops to source bucket
2021/12/15 20:46:47 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 20:46:52 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 20:46:57 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 20:47:02 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 20:47:07 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 20:47:12 Waiting for dst bucket item count to get to: 5000 curr count: 2272
2021/12/15 20:47:17 Waiting for dst bucket item count to get to: 5000 curr count: 2272
2021/12/15 20:47:17 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/15 20:47:17 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 20:47:17 Pausing app: bucket_op_with_timer
2021/12/15 20:47:17 Function: TestPauseAndResumeWithWorkerCountChange update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/15 20:47:22 Waiting for function: TestPauseAndResumeWithWorkerCountChange status to change to paused
2021/12/15 20:47:22 Function: TestPauseAndResumeWithWorkerCountChange status changed to paused
2021/12/15 20:47:22 Resuming app: bucket_op_with_timer
2021/12/15 20:47:23 Function: TestPauseAndResumeWithWorkerCountChange update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:6] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/15 20:47:23 Starting bucket ops to source bucket
2021/12/15 20:47:35 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/15 20:47:40 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/15 20:47:45 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/15 20:47:50 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/15 20:47:55 Waiting for dst bucket item count to get to: 10000 curr count: 6612
2021/12/15 20:48:00 Waiting for dst bucket item count to get to: 10000 curr count: 9112
2021/12/15 20:48:00 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/15 20:48:00 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 20:48:05 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get un-deployed
2021/12/15 20:48:10 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get un-deployed
2021/12/15 20:48:15 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get un-deployed
2021/12/15 20:48:20 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get un-deployed
2021/12/15 20:48:20 App: TestPauseAndResumeWithWorkerCountChange got un-deployed
2021/12/15 20:48:20 No eventing-con process running
2021/12/15 20:48:20 Function: TestPauseAndResumeWithWorkerCountChange delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestPauseAndResumeWithWorkerCountChange deleting in the background"
}
2021/12/15 20:48:22 Context: Bucket flush, response code: 200 dump: 
2021/12/15 20:48:32 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/15 20:48:37 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/15 20:48:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 20:48:39 Context: Bucket flush, response code: 200 dump: 
2021/12/15 20:48:49 Waiting for dst bucket item count to get to: 0 curr count: 9310
2021/12/15 20:48:54 Waiting for dst bucket item count to get to: 0 curr count: 4310
2021/12/15 20:48:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestPauseAndResumeWithWorkerCountChange (180.87s)
=== RUN   TestPauseResumeWithEventingReb
2021/12/15 20:49:04 Waiting for app: TestPauseResumeWithEventingReb to get un-deployed
2021/12/15 20:49:04 App: TestPauseResumeWithEventingReb got un-deployed
2021/12/15 20:49:04 No eventing-con process running
2021/12/15 20:49:04 Function: TestPauseResumeWithEventingReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestPauseResumeWithEventingReb not found"
 }
}
2021/12/15 20:49:06 Context: Bucket flush, response code: 200 dump: 
2021/12/15 20:49:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 20:49:12 Context: Bucket flush, response code: 200 dump: 
2021/12/15 20:49:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 20:49:17 Deploying app: TestPauseResumeWithEventingReb. Item count src bucket: 0 dst bucket: 0
2021/12/15 20:49:18 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestPauseResumeWithEventingReb' in metakv",
  "warnings": null
 }
}
2021/12/15 20:49:23 Waiting for app: TestPauseResumeWithEventingReb to get deployed
2021/12/15 20:49:28 Waiting for app: TestPauseResumeWithEventingReb to get deployed
2021/12/15 20:49:33 Waiting for app: TestPauseResumeWithEventingReb to get deployed
2021/12/15 20:49:38 Waiting for app: TestPauseResumeWithEventingReb to get deployed
2021/12/15 20:49:38 App: TestPauseResumeWithEventingReb got deployed
2021/12/15 20:49:38 Starting bucket ops to source bucket
2021/12/15 20:49:48 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 20:49:53 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 20:49:58 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 20:50:03 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 20:50:08 Waiting for dst bucket item count to get to: 5000 curr count: 1078
2021/12/15 20:50:13 Waiting for dst bucket item count to get to: 5000 curr count: 2781
2021/12/15 20:50:18 Waiting for dst bucket item count to get to: 5000 curr count: 4203
2021/12/15 20:50:18 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/15 20:50:18 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 20:50:18 Pausing app: bucket_op_with_timer
2021/12/15 20:50:18 Function: TestPauseResumeWithEventingReb update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/15 20:50:23 Waiting for function: TestPauseResumeWithEventingReb status to change to paused
2021/12/15 20:50:23 Function: TestPauseResumeWithEventingReb status changed to paused
2021/12/15 20:50:23 Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/15 20:50:23 Removing node(s): [] from the cluster
2021/12/15 20:50:28 Rebalance progress: 29.16666666666666
2021/12/15 20:50:33 Rebalance progress: 29.16666666666666
2021/12/15 20:50:38 Rebalance progress: 100
2021/12/15 20:50:38 Resuming app: bucket_op_with_timer
2021/12/15 20:50:40 Function: TestPauseResumeWithEventingReb update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:6] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/15 20:50:40 Starting bucket ops to source bucket
2021/12/15 20:50:51 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/15 20:50:56 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/15 20:51:01 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/15 20:51:06 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/15 20:51:11 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/15 20:51:16 Waiting for dst bucket item count to get to: 10000 curr count: 7500
2021/12/15 20:51:16 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/15 20:51:16 Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/15 20:51:21 Rebalance progress: 29.16666666666666
2021/12/15 20:51:26 Rebalance progress: 29.16666666666666
2021/12/15 20:51:31 Rebalance progress: 100

VB distribution across eventing nodes and their workers::

2021/12/15 20:51:37 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:66} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/15 20:51:52 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:66} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/15 20:52:07 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:66} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/15 20:52:22 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:66} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/15 20:52:37 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:66} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/15 20:52:52 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:66} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/15 20:53:07 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:66} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/15 20:53:22 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:66} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/15 20:53:37 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:66} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/15 20:53:52 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:66} {Name:#primary Bucket:hello-world Completion:100}]

DCP Stream statuses:
	stream status: running
	len: 1024
	vb list dump: [0-1023]

vbucket curr owner:
Producer node: 172.31.2.134:9300	Node UUID: 397db3716346d2b10c24acbfeda15c2c
	workerID: worker_TestPauseResumeWithEventingReb_4
	len: 57
	v: [911-967]
	workerID: worker_TestPauseResumeWithEventingReb_5
	len: 56
	v: [968-1023]
	workerID: worker_TestPauseResumeWithEventingReb_0
	len: 57
	v: [683-739]
	workerID: worker_TestPauseResumeWithEventingReb_1
	len: 57
	v: [740-796]
	workerID: worker_TestPauseResumeWithEventingReb_2
	len: 57
	v: [797-853]
	workerID: worker_TestPauseResumeWithEventingReb_3
	len: 57
	v: [854-910]
Producer node: 127.0.0.1:9303	Node UUID: 201f4904ab605131d94b62bdd8e3b413
	workerID: worker_TestPauseResumeWithEventingReb_2
	len: 57
	v: [456-512]
	workerID: worker_TestPauseResumeWithEventingReb_3
	len: 57
	v: [513-569]
	workerID: worker_TestPauseResumeWithEventingReb_4
	len: 57
	v: [570-626]
	workerID: worker_TestPauseResumeWithEventingReb_5
	len: 56
	v: [627-682]
	workerID: worker_TestPauseResumeWithEventingReb_0
	len: 57
	v: [342-398]
	workerID: worker_TestPauseResumeWithEventingReb_1
	len: 57
	v: [399-455]
Producer node: 127.0.0.1:9302	Node UUID: 71dd74e37b11698f0577664c1e8e9b69
	workerID: worker_TestPauseResumeWithEventingReb_4
	len: 57
	v: [228-284]
	workerID: worker_TestPauseResumeWithEventingReb_5
	len: 57
	v: [285-341]
	workerID: worker_TestPauseResumeWithEventingReb_0
	len: 57
	v: [0-56]
	workerID: worker_TestPauseResumeWithEventingReb_1
	len: 57
	v: [57-113]
	workerID: worker_TestPauseResumeWithEventingReb_2
	len: 57
	v: [114-170]
	workerID: worker_TestPauseResumeWithEventingReb_3
	len: 57
	v: [171-227]


2021/12/15 20:54:07 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 20:54:07 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 20:54:12 clusterSetup.go:397: Waiting for app: TestPauseResumeWithEventingReb to get un-deployed
2021/12/15 20:54:17 clusterSetup.go:397: Waiting for app: TestPauseResumeWithEventingReb to get un-deployed
2021/12/15 20:54:22 clusterSetup.go:397: Waiting for app: TestPauseResumeWithEventingReb to get un-deployed
2021/12/15 20:54:27 clusterSetup.go:397: Waiting for app: TestPauseResumeWithEventingReb to get un-deployed
2021/12/15 20:54:27 clusterSetup.go:405: App: TestPauseResumeWithEventingReb got un-deployed
2021/12/15 20:54:27 eventingSetup.go:46: No eventing-con process running
2021/12/15 20:54:27 eventingSetup.go:418: Function: TestPauseResumeWithEventingReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestPauseResumeWithEventingReb deleting in the background"
}
2021/12/15 20:54:29 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 20:54:39 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 9611
2021/12/15 20:54:44 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4611
2021/12/15 20:54:44 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 20:54:46 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 20:54:56 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/15 20:54:56 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestPauseResumeWithEventingReb (361.89s)
=== RUN   TestBucketDeleteAfterPause
2021/12/15 20:55:06 clusterSetup.go:397: Waiting for app: TestBucketDeleteAfterPause to get un-deployed
2021/12/15 20:55:06 clusterSetup.go:405: App: TestBucketDeleteAfterPause got un-deployed
2021/12/15 20:55:06 eventingSetup.go:46: No eventing-con process running
2021/12/15 20:55:06 eventingSetup.go:418: Function: TestBucketDeleteAfterPause delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestBucketDeleteAfterPause not found"
 }
}
2021/12/15 20:55:07 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 20:55:12 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 20:55:14 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 20:55:19 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 20:55:19 eventingSetup.go:125: Deploying app: TestBucketDeleteAfterPause. Item count src bucket: 0 dst bucket: 0
2021/12/15 20:55:20 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestBucketDeleteAfterPause' in metakv",
  "warnings": null
 }
}
2021/12/15 20:55:25 clusterSetup.go:346: Waiting for app: TestBucketDeleteAfterPause to get deployed
2021/12/15 20:55:30 clusterSetup.go:346: Waiting for app: TestBucketDeleteAfterPause to get deployed
2021/12/15 20:55:35 clusterSetup.go:346: Waiting for app: TestBucketDeleteAfterPause to get deployed
2021/12/15 20:55:40 clusterSetup.go:346: Waiting for app: TestBucketDeleteAfterPause to get deployed
2021/12/15 20:55:40 clusterSetup.go:354: App: TestBucketDeleteAfterPause got deployed
2021/12/15 20:55:40 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/15 20:55:50 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 20:55:55 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 20:55:55 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/15 20:55:55 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 20:55:55 handler_test.go:712: Pausing app: bucket_op_on_update
2021/12/15 20:55:55 eventingSetup.go:270: Function: TestBucketDeleteAfterPause update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/15 20:56:00 eventingSetup.go:719: Waiting for function: TestBucketDeleteAfterPause status to change to paused
2021/12/15 20:56:00 eventingSetup.go:727: Function: TestBucketDeleteAfterPause status changed to paused
2021/12/15 20:56:00 handler_test.go:716: Deleting source bucket: default
2021/12/15 20:56:06 eventingSetup.go:719: Waiting for function: TestBucketDeleteAfterPause status to change to undeployed
2021/12/15 20:56:06 eventingSetup.go:727: Function: TestBucketDeleteAfterPause status changed to undeployed
2021/12/15 20:56:11 clusterSetup.go:397: Waiting for app: TestBucketDeleteAfterPause to get un-deployed
2021/12/15 20:56:11 clusterSetup.go:405: App: TestBucketDeleteAfterPause got un-deployed
2021/12/15 20:56:11 eventingSetup.go:46: No eventing-con process running
2021/12/15 20:56:11 eventingSetup.go:418: Function: TestBucketDeleteAfterPause delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestBucketDeleteAfterPause deleting in the background"
}
2021/12/15 20:56:13 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 20:56:18 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 20:56:20 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 20:56:30 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/15 20:56:35 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/15 20:56:35 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestBucketDeleteAfterPause (98.96s)
=== RUN   TestChangeFnCodeBetweenPauseResume
2021/12/15 20:56:45 clusterSetup.go:397: Waiting for app: TestChangeFnCodeBetweenPauseResume to get un-deployed
2021/12/15 20:56:45 clusterSetup.go:405: App: TestChangeFnCodeBetweenPauseResume got un-deployed
2021/12/15 20:56:45 eventingSetup.go:46: No eventing-con process running
2021/12/15 20:56:45 eventingSetup.go:418: Function: TestChangeFnCodeBetweenPauseResume delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestChangeFnCodeBetweenPauseResume not found"
 }
}
2021/12/15 20:56:46 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 20:56:51 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 20:56:53 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 20:56:58 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 20:56:58 eventingSetup.go:125: Deploying app: TestChangeFnCodeBetweenPauseResume. Item count src bucket: 0 dst bucket: 0
2021/12/15 20:56:59 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestChangeFnCodeBetweenPauseResume' in metakv",
  "warnings": null
 }
}
2021/12/15 20:57:04 clusterSetup.go:346: Waiting for app: TestChangeFnCodeBetweenPauseResume to get deployed
2021/12/15 20:57:09 clusterSetup.go:346: Waiting for app: TestChangeFnCodeBetweenPauseResume to get deployed
2021/12/15 20:57:14 clusterSetup.go:346: Waiting for app: TestChangeFnCodeBetweenPauseResume to get deployed
2021/12/15 20:57:19 clusterSetup.go:346: Waiting for app: TestChangeFnCodeBetweenPauseResume to get deployed
2021/12/15 20:57:19 clusterSetup.go:354: App: TestChangeFnCodeBetweenPauseResume got deployed
2021/12/15 20:57:19 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/15 20:57:54 handler_test.go:739: Pausing function: TestChangeFnCodeBetweenPauseResume
2021/12/15 20:57:54 eventingSetup.go:270: Function: TestChangeFnCodeBetweenPauseResume update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/15 20:57:59 eventingSetup.go:719: Waiting for function: TestChangeFnCodeBetweenPauseResume status to change to paused
2021/12/15 20:57:59 eventingSetup.go:727: Function: TestChangeFnCodeBetweenPauseResume status changed to paused
2021/12/15 21:00:59 handler_test.go:748: Resuming function: TestChangeFnCodeBetweenPauseResume
2021/12/15 21:00:59 eventingSetup.go:125: Deploying app: TestChangeFnCodeBetweenPauseResume. Item count src bucket: 5000 dst bucket: 0
2021/12/15 21:01:00 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestChangeFnCodeBetweenPauseResume' in metakv",
  "warnings": null
 }
}
2021/12/15 21:01:05 eventingSetup.go:795: Waiting for function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter to get to 5000
2021/12/15 21:01:05 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 21:01:05 eventingSetup.go:799: Function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter got to 0 expected 5000
2021/12/15 21:01:10 eventingSetup.go:795: Waiting for function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter to get to 5000
2021/12/15 21:01:10 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 21:01:10 eventingSetup.go:799: Function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter got to 3208 expected 5000
2021/12/15 21:01:15 eventingSetup.go:795: Waiting for function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter to get to 5000
2021/12/15 21:01:15 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 21:01:15 eventingSetup.go:799: Function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter got to 3901 expected 5000
2021/12/15 21:01:20 eventingSetup.go:795: Waiting for function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter to get to 5000
2021/12/15 21:01:20 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 21:01:20 eventingSetup.go:804: Function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter got to 5000
2021/12/15 21:01:20 handler_test.go:753: Undeploying function: TestChangeFnCodeBetweenPauseResume
2021/12/15 21:01:20 eventingSetup.go:270: Function: TestChangeFnCodeBetweenPauseResume update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/15 21:01:30 clusterSetup.go:397: Waiting for app: TestChangeFnCodeBetweenPauseResume to get un-deployed
2021/12/15 21:01:35 clusterSetup.go:397: Waiting for app: TestChangeFnCodeBetweenPauseResume to get un-deployed
2021/12/15 21:01:40 clusterSetup.go:397: Waiting for app: TestChangeFnCodeBetweenPauseResume to get un-deployed
2021/12/15 21:01:40 clusterSetup.go:405: App: TestChangeFnCodeBetweenPauseResume got un-deployed
2021/12/15 21:01:40 eventingSetup.go:46: No eventing-con process running
2021/12/15 21:01:40 eventingSetup.go:418: Function: TestChangeFnCodeBetweenPauseResume delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestChangeFnCodeBetweenPauseResume deleting in the background"
}
2021/12/15 21:01:42 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 21:01:52 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/15 21:01:57 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/15 21:01:57 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 21:01:59 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 21:02:04 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestChangeFnCodeBetweenPauseResume (328.95s)
=== RUN   TestCommentUnCommentOnDelete
2021/12/15 21:02:14 clusterSetup.go:397: Waiting for app: TestCommentUnCommentOnDelete to get un-deployed
2021/12/15 21:02:14 clusterSetup.go:405: App: TestCommentUnCommentOnDelete got un-deployed
2021/12/15 21:02:14 eventingSetup.go:46: No eventing-con process running
2021/12/15 21:02:14 eventingSetup.go:418: Function: TestCommentUnCommentOnDelete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestCommentUnCommentOnDelete not found"
 }
}
2021/12/15 21:02:15 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 21:02:20 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 21:02:22 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 21:02:27 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 21:02:27 eventingSetup.go:125: Deploying app: TestCommentUnCommentOnDelete. Item count src bucket: 0 dst bucket: 0
2021/12/15 21:02:28 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCommentUnCommentOnDelete' in metakv",
  "warnings": null
 }
}
2021/12/15 21:02:33 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/15 21:02:38 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/15 21:02:43 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/15 21:02:48 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/15 21:02:48 clusterSetup.go:354: App: TestCommentUnCommentOnDelete got deployed
2021/12/15 21:02:48 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/15 21:02:57 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 21:03:02 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 826
2021/12/15 21:03:07 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 826
2021/12/15 21:03:07 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/15 21:03:07 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 21:03:07 handler_test.go:780: Undeploying app: TestCommentUnCommentOnDelete
2021/12/15 21:03:07 eventingSetup.go:270: Function: TestCommentUnCommentOnDelete update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/15 21:03:37 eventingSetup.go:125: Deploying app: TestCommentUnCommentOnDelete. Item count src bucket: 5000 dst bucket: 5000
2021/12/15 21:03:38 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCommentUnCommentOnDelete' in metakv",
  "warnings": null
 }
}
2021/12/15 21:03:43 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/15 21:03:48 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/15 21:03:53 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/15 21:03:58 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/15 21:03:58 clusterSetup.go:354: App: TestCommentUnCommentOnDelete got deployed
2021/12/15 21:03:58 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/15 21:04:11 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/15 21:04:16 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/15 21:04:16 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 21:04:16 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 21:04:16 handler_test.go:799: Undeploying app: TestCommentUnCommentOnDelete
2021/12/15 21:04:16 eventingSetup.go:270: Function: TestCommentUnCommentOnDelete update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/15 21:04:26 clusterSetup.go:397: Waiting for app: TestCommentUnCommentOnDelete to get un-deployed
2021/12/15 21:04:31 clusterSetup.go:397: Waiting for app: TestCommentUnCommentOnDelete to get un-deployed
2021/12/15 21:04:36 clusterSetup.go:397: Waiting for app: TestCommentUnCommentOnDelete to get un-deployed
2021/12/15 21:04:36 clusterSetup.go:405: App: TestCommentUnCommentOnDelete got un-deployed
2021/12/15 21:04:36 eventingSetup.go:46: No eventing-con process running
2021/12/15 21:04:36 eventingSetup.go:418: Function: TestCommentUnCommentOnDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCommentUnCommentOnDelete deleting in the background"
}
2021/12/15 21:04:37 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 21:04:42 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 21:04:44 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 21:04:49 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCommentUnCommentOnDelete (165.67s)
=== RUN   TestCPPWorkerCleanup
2021/12/15 21:04:59 clusterSetup.go:397: Waiting for app: TestCPPWorkerCleanup to get un-deployed
2021/12/15 21:04:59 clusterSetup.go:405: App: TestCPPWorkerCleanup got un-deployed
2021/12/15 21:04:59 eventingSetup.go:46: No eventing-con process running
2021/12/15 21:04:59 eventingSetup.go:418: Function: TestCPPWorkerCleanup delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestCPPWorkerCleanup not found"
 }
}
2021/12/15 21:05:01 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 21:05:06 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 21:05:07 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 21:05:12 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 21:05:12 eventingSetup.go:125: Deploying app: TestCPPWorkerCleanup. Item count src bucket: 0 dst bucket: 0
2021/12/15 21:05:13 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCPPWorkerCleanup' in metakv",
  "warnings": null
 }
}
2021/12/15 21:05:13 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/15 21:05:21 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 21:05:26 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 21:05:31 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 21:05:36 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 174
2021/12/15 21:05:41 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 2165
2021/12/15 21:05:46 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 4491
2021/12/15 21:05:46 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/15 21:05:46 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 21:05:46 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 21:05:51 clusterSetup.go:397: Waiting for app: TestCPPWorkerCleanup to get un-deployed
2021/12/15 21:05:56 clusterSetup.go:397: Waiting for app: TestCPPWorkerCleanup to get un-deployed
2021/12/15 21:06:01 clusterSetup.go:397: Waiting for app: TestCPPWorkerCleanup to get un-deployed
2021/12/15 21:06:06 clusterSetup.go:397: Waiting for app: TestCPPWorkerCleanup to get un-deployed
2021/12/15 21:06:06 clusterSetup.go:405: App: TestCPPWorkerCleanup got un-deployed
2021/12/15 21:06:06 eventingSetup.go:46: No eventing-con process running
2021/12/15 21:06:06 eventingSetup.go:418: Function: TestCPPWorkerCleanup delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCPPWorkerCleanup deleting in the background"
}
2021/12/15 21:06:07 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 21:06:17 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/15 21:06:17 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 21:06:19 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 21:06:29 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/15 21:06:34 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/15 21:06:34 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCPPWorkerCleanup (134.89s)
=== RUN   TestWithUserXattrs
2021/12/15 21:07:14 clusterSetup.go:397: Waiting for app: TestWithUserXattrs to get un-deployed
2021/12/15 21:07:14 clusterSetup.go:405: App: TestWithUserXattrs got un-deployed
2021/12/15 21:07:14 eventingSetup.go:46: No eventing-con process running
2021/12/15 21:07:14 eventingSetup.go:418: Function: TestWithUserXattrs delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestWithUserXattrs not found"
 }
}
2021/12/15 21:07:15 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 21:07:20 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 21:07:22 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 21:07:27 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 21:07:27 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/15 21:07:29 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/15 21:07:31 eventingSetup.go:125: Deploying app: TestWithUserXattrs. Item count src bucket: 0 dst bucket: 0
2021/12/15 21:07:32 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestWithUserXattrs' in metakv",
  "warnings": null
 }
}
2021/12/15 21:07:37 clusterSetup.go:346: Waiting for app: TestWithUserXattrs to get deployed
2021/12/15 21:07:42 clusterSetup.go:346: Waiting for app: TestWithUserXattrs to get deployed
2021/12/15 21:07:47 clusterSetup.go:346: Waiting for app: TestWithUserXattrs to get deployed
2021/12/15 21:07:52 clusterSetup.go:346: Waiting for app: TestWithUserXattrs to get deployed
2021/12/15 21:07:52 clusterSetup.go:354: App: TestWithUserXattrs got deployed
2021/12/15 21:07:52 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/15 21:08:02 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 21:08:07 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 2500
2021/12/15 21:08:07 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/15 21:08:07 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 21:08:07 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 21:08:12 clusterSetup.go:397: Waiting for app: TestWithUserXattrs to get un-deployed
2021/12/15 21:08:17 clusterSetup.go:397: Waiting for app: TestWithUserXattrs to get un-deployed
2021/12/15 21:08:22 clusterSetup.go:397: Waiting for app: TestWithUserXattrs to get un-deployed
2021/12/15 21:08:27 clusterSetup.go:397: Waiting for app: TestWithUserXattrs to get un-deployed
2021/12/15 21:08:27 clusterSetup.go:405: App: TestWithUserXattrs got un-deployed
2021/12/15 21:08:27 eventingSetup.go:46: No eventing-con process running
2021/12/15 21:08:27 eventingSetup.go:418: Function: TestWithUserXattrs delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestWithUserXattrs deleting in the background"
}
2021/12/15 21:08:28 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 21:08:38 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4693
2021/12/15 21:08:43 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2193
2021/12/15 21:08:43 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 21:08:45 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 21:08:55 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/15 21:08:55 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestWithUserXattrs (110.97s)
=== RUN   TestEventProcessingPostBucketFlush
2021/12/15 21:09:05 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/15 21:09:05 clusterSetup.go:405: App: TestEventProcessingPostBucketFlush got un-deployed
2021/12/15 21:09:05 eventingSetup.go:46: No eventing-con process running
2021/12/15 21:09:05 eventingSetup.go:418: Function: TestEventProcessingPostBucketFlush delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventProcessingPostBucketFlush not found"
 }
}
2021/12/15 21:09:06 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 21:09:11 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 21:09:13 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 21:09:18 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 21:09:18 eventingSetup.go:125: Deploying app: TestEventProcessingPostBucketFlush. Item count src bucket: 0 dst bucket: 0
2021/12/15 21:09:19 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventProcessingPostBucketFlush' in metakv",
  "warnings": null
 }
}
2021/12/15 21:09:24 clusterSetup.go:346: Waiting for app: TestEventProcessingPostBucketFlush to get deployed
2021/12/15 21:09:29 clusterSetup.go:346: Waiting for app: TestEventProcessingPostBucketFlush to get deployed
2021/12/15 21:09:34 clusterSetup.go:346: Waiting for app: TestEventProcessingPostBucketFlush to get deployed
2021/12/15 21:09:39 clusterSetup.go:346: Waiting for app: TestEventProcessingPostBucketFlush to get deployed
2021/12/15 21:09:39 clusterSetup.go:354: App: TestEventProcessingPostBucketFlush got deployed
2021/12/15 21:09:39 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/15 21:09:49 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 21:09:54 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 21:09:59 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 2500
2021/12/15 21:09:59 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/15 21:09:59 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 21:10:03 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 21:10:13 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4294
2021/12/15 21:10:18 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1794
2021/12/15 21:10:18 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 21:10:19 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 21:10:29 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/15 21:10:34 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/15 21:10:34 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 21:10:34 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/15 21:11:00 eventingSetup.go:441: Waiting for dst bucket item count to get to: 25000 curr count: 17003
2021/12/15 21:11:05 eventingSetup.go:441: Waiting for dst bucket item count to get to: 25000 curr count: 17003
2021/12/15 21:11:05 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 25000 dst bucket count: 25000
2021/12/15 21:11:05 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 21:11:05 eventingSetup.go:270: Function: TestEventProcessingPostBucketFlush update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/15 21:11:10 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/15 21:11:15 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/15 21:11:20 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/15 21:11:25 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/15 21:11:25 clusterSetup.go:405: App: TestEventProcessingPostBucketFlush got un-deployed
2021/12/15 21:11:25 eventingSetup.go:46: No eventing-con process running
2021/12/15 21:11:30 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/15 21:11:30 clusterSetup.go:405: App: TestEventProcessingPostBucketFlush got un-deployed
2021/12/15 21:11:30 eventingSetup.go:46: No eventing-con process running
2021/12/15 21:11:30 eventingSetup.go:418: Function: TestEventProcessingPostBucketFlush delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventProcessingPostBucketFlush deleting in the background"
}
2021/12/15 21:11:31 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 21:11:41 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 25000
2021/12/15 21:11:46 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 25000
2021/12/15 21:11:46 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 21:11:48 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 21:11:58 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 16648
2021/12/15 21:12:03 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4148
2021/12/15 21:12:03 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventProcessingPostBucketFlush (187.88s)
=== RUN   TestMetaBucketDelete
2021/12/15 21:12:13 clusterSetup.go:397: Waiting for app: TestMetaBucketDelete to get un-deployed
2021/12/15 21:12:13 clusterSetup.go:405: App: TestMetaBucketDelete got un-deployed
2021/12/15 21:12:13 eventingSetup.go:46: No eventing-con process running
2021/12/15 21:12:13 eventingSetup.go:418: Function: TestMetaBucketDelete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMetaBucketDelete not found"
 }
}
2021/12/15 21:12:14 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 21:12:19 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 21:12:21 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 21:12:26 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 21:12:26 eventingSetup.go:125: Deploying app: TestMetaBucketDelete. Item count src bucket: 0 dst bucket: 0
2021/12/15 21:12:27 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMetaBucketDelete' in metakv",
  "warnings": null
 }
}
2021/12/15 21:12:32 clusterSetup.go:346: Waiting for app: TestMetaBucketDelete to get deployed
2021/12/15 21:12:37 clusterSetup.go:346: Waiting for app: TestMetaBucketDelete to get deployed
2021/12/15 21:12:42 clusterSetup.go:346: Waiting for app: TestMetaBucketDelete to get deployed
2021/12/15 21:12:47 clusterSetup.go:346: Waiting for app: TestMetaBucketDelete to get deployed
2021/12/15 21:12:47 clusterSetup.go:354: App: TestMetaBucketDelete got deployed
2021/12/15 21:12:47 handler_test.go:902: Deleting metadata bucket: eventing
2021/12/15 21:12:48 handler_test.go:904: Deleted metadata bucket: eventing
2021/12/15 21:12:53 clusterSetup.go:397: Waiting for app: TestMetaBucketDelete to get un-deployed
2021/12/15 21:12:53 clusterSetup.go:405: App: TestMetaBucketDelete got un-deployed
2021/12/15 21:13:08 clusterSetup.go:397: Waiting for app: TestMetaBucketDelete to get un-deployed
2021/12/15 21:13:08 clusterSetup.go:405: App: TestMetaBucketDelete got un-deployed
2021/12/15 21:13:08 eventingSetup.go:46: No eventing-con process running
2021/12/15 21:13:08 eventingSetup.go:418: Function: TestMetaBucketDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMetaBucketDelete deleting in the background"
}
2021/12/15 21:13:10 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 21:13:15 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 21:13:16 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 21:13:21 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestMetaBucketDelete (78.40s)
=== RUN   TestMetaBucketDeleteWithBootstrap
2021/12/15 21:13:31 clusterSetup.go:397: Waiting for app: TestMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/15 21:13:31 clusterSetup.go:405: App: TestMetaBucketDeleteWithBootstrap got un-deployed
2021/12/15 21:13:31 eventingSetup.go:46: No eventing-con process running
2021/12/15 21:13:31 eventingSetup.go:418: Function: TestMetaBucketDeleteWithBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMetaBucketDeleteWithBootstrap not found"
 }
}
2021/12/15 21:13:33 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 21:13:38 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 21:13:39 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 21:13:44 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 21:13:44 eventingSetup.go:125: Deploying app: TestMetaBucketDeleteWithBootstrap. Item count src bucket: 0 dst bucket: 0
2021/12/15 21:13:46 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMetaBucketDeleteWithBootstrap' in metakv",
  "warnings": null
 }
}
2021/12/15 21:14:04 handler_test.go:923: Deleting metadata bucket: eventing
2021/12/15 21:14:05 handler_test.go:925: Deleted metadata bucket: eventing
2021/12/15 21:14:20 clusterSetup.go:397: Waiting for app: TestMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/15 21:14:20 clusterSetup.go:405: App: TestMetaBucketDeleteWithBootstrap got un-deployed
2021/12/15 21:14:35 clusterSetup.go:397: Waiting for app: TestMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/15 21:14:35 clusterSetup.go:405: App: TestMetaBucketDeleteWithBootstrap got un-deployed
2021/12/15 21:14:35 eventingSetup.go:46: No eventing-con process running
2021/12/15 21:14:35 eventingSetup.go:418: Function: TestMetaBucketDeleteWithBootstrap delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMetaBucketDeleteWithBootstrap deleting in the background"
}
2021/12/15 21:14:36 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 21:14:41 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 21:14:43 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 21:14:48 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestMetaBucketDeleteWithBootstrap (86.79s)
=== RUN   TestSourceBucketDelete
2021/12/15 21:14:58 clusterSetup.go:397: Waiting for app: TestSourceBucketDelete to get un-deployed
2021/12/15 21:14:58 clusterSetup.go:405: App: TestSourceBucketDelete got un-deployed
2021/12/15 21:14:58 eventingSetup.go:46: No eventing-con process running
2021/12/15 21:14:58 eventingSetup.go:418: Function: TestSourceBucketDelete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestSourceBucketDelete not found"
 }
}
2021/12/15 21:14:59 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 21:15:04 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 21:15:06 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 21:15:11 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 21:15:11 eventingSetup.go:125: Deploying app: TestSourceBucketDelete. Item count src bucket: 0 dst bucket: 0
2021/12/15 21:15:12 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestSourceBucketDelete' in metakv",
  "warnings": null
 }
}
2021/12/15 21:15:17 clusterSetup.go:346: Waiting for app: TestSourceBucketDelete to get deployed
2021/12/15 21:15:22 clusterSetup.go:346: Waiting for app: TestSourceBucketDelete to get deployed
2021/12/15 21:15:27 clusterSetup.go:346: Waiting for app: TestSourceBucketDelete to get deployed
2021/12/15 21:15:32 clusterSetup.go:346: Waiting for app: TestSourceBucketDelete to get deployed
2021/12/15 21:15:32 clusterSetup.go:354: App: TestSourceBucketDelete got deployed
2021/12/15 21:15:32 handler_test.go:945: Deleting source bucket: default
2021/12/15 21:15:34 handler_test.go:947: Deleted source bucket: default
2021/12/15 21:15:39 clusterSetup.go:397: Waiting for app: TestSourceBucketDelete to get un-deployed
2021/12/15 21:15:39 clusterSetup.go:405: App: TestSourceBucketDelete got un-deployed
2021/12/15 21:15:54 clusterSetup.go:397: Waiting for app: TestSourceBucketDelete to get un-deployed
2021/12/15 21:15:54 clusterSetup.go:405: App: TestSourceBucketDelete got un-deployed
2021/12/15 21:15:54 eventingSetup.go:46: No eventing-con process running
2021/12/15 21:15:54 eventingSetup.go:418: Function: TestSourceBucketDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestSourceBucketDelete deleting in the background"
}
2021/12/15 21:15:55 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 21:16:00 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 21:16:02 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 21:16:07 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestSourceBucketDelete (79.04s)
=== RUN   TestSourceBucketDeleteWithBootstrap
2021/12/15 21:16:17 clusterSetup.go:397: Waiting for app: TestSourceBucketDeleteWithBootstrap to get un-deployed
2021/12/15 21:16:17 clusterSetup.go:405: App: TestSourceBucketDeleteWithBootstrap got un-deployed
2021/12/15 21:16:17 eventingSetup.go:46: No eventing-con process running
2021/12/15 21:16:17 eventingSetup.go:418: Function: TestSourceBucketDeleteWithBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestSourceBucketDeleteWithBootstrap not found"
 }
}
2021/12/15 21:16:18 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 21:16:23 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 21:16:25 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 21:16:30 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 21:16:30 eventingSetup.go:125: Deploying app: TestSourceBucketDeleteWithBootstrap. Item count src bucket: 0 dst bucket: 0
2021/12/15 21:16:31 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestSourceBucketDeleteWithBootstrap' in metakv",
  "warnings": null
 }
}
2021/12/15 21:16:50 handler_test.go:966: Deleting source bucket: default
2021/12/15 21:16:51 handler_test.go:968: Deleted source bucket: default
2021/12/15 21:17:07 clusterSetup.go:397: Waiting for app: TestSourceBucketDeleteWithBootstrap to get un-deployed
2021/12/15 21:17:07 clusterSetup.go:405: App: TestSourceBucketDeleteWithBootstrap got un-deployed
2021/12/15 21:17:22 clusterSetup.go:397: Waiting for app: TestSourceBucketDeleteWithBootstrap to get un-deployed
2021/12/15 21:17:22 clusterSetup.go:405: App: TestSourceBucketDeleteWithBootstrap got un-deployed
2021/12/15 21:17:22 eventingSetup.go:46: No eventing-con process running
2021/12/15 21:17:22 eventingSetup.go:418: Function: TestSourceBucketDeleteWithBootstrap delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestSourceBucketDeleteWithBootstrap deleting in the background"
}
2021/12/15 21:17:23 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 21:17:28 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 21:17:30 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 21:17:35 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestSourceBucketDeleteWithBootstrap (87.54s)
=== RUN   TestSourceAndMetaBucketDeleteWithBootstrap
2021/12/15 21:17:45 clusterSetup.go:397: Waiting for app: TestSourceAndMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/15 21:17:45 clusterSetup.go:405: App: TestSourceAndMetaBucketDeleteWithBootstrap got un-deployed
2021/12/15 21:17:45 eventingSetup.go:46: No eventing-con process running
2021/12/15 21:17:45 eventingSetup.go:418: Function: TestSourceAndMetaBucketDeleteWithBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestSourceAndMetaBucketDeleteWithBootstrap not found"
 }
}
2021/12/15 21:17:46 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 21:17:51 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 21:17:53 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 21:17:58 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 21:17:58 eventingSetup.go:125: Deploying app: TestSourceAndMetaBucketDeleteWithBootstrap. Item count src bucket: 0 dst bucket: 0
2021/12/15 21:17:59 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestSourceAndMetaBucketDeleteWithBootstrap' in metakv",
  "warnings": null
 }
}
2021/12/15 21:18:18 handler_test.go:988: Deleting source bucket: default
2021/12/15 21:18:19 handler_test.go:990: Deleted source bucket: default
2021/12/15 21:18:19 handler_test.go:992: Deleting metadata bucket: eventing
2021/12/15 21:18:20 handler_test.go:994: Deleted metadata bucket: eventing
2021/12/15 21:18:35 clusterSetup.go:397: Waiting for app: TestSourceAndMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/15 21:18:35 clusterSetup.go:405: App: TestSourceAndMetaBucketDeleteWithBootstrap got un-deployed
2021/12/15 21:18:50 clusterSetup.go:397: Waiting for app: TestSourceAndMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/15 21:18:50 clusterSetup.go:405: App: TestSourceAndMetaBucketDeleteWithBootstrap got un-deployed
2021/12/15 21:18:50 eventingSetup.go:46: No eventing-con process running
2021/12/15 21:18:50 eventingSetup.go:418: Function: TestSourceAndMetaBucketDeleteWithBootstrap delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestSourceAndMetaBucketDeleteWithBootstrap deleting in the background"
}
2021/12/15 21:18:51 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 21:18:56 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 21:18:58 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 21:19:03 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestSourceAndMetaBucketDeleteWithBootstrap (88.29s)
=== RUN   TestUndeployDuringBootstrap
2021/12/15 21:19:13 clusterSetup.go:397: Waiting for app: TestUndeployDuringBootstrap to get un-deployed
2021/12/15 21:19:13 clusterSetup.go:405: App: TestUndeployDuringBootstrap got un-deployed
2021/12/15 21:19:13 eventingSetup.go:46: No eventing-con process running
2021/12/15 21:19:13 eventingSetup.go:418: Function: TestUndeployDuringBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestUndeployDuringBootstrap not found"
 }
}
2021/12/15 21:19:14 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 21:19:19 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 21:19:21 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 21:19:26 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 21:19:26 eventingSetup.go:125: Deploying app: TestUndeployDuringBootstrap. Item count src bucket: 0 dst bucket: 0
2021/12/15 21:19:27 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestUndeployDuringBootstrap' in metakv",
  "warnings": null
 }
}
2021/12/15 21:19:27 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/15 21:19:39 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 21:19:39 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 21:19:39 clusterSetup.go:375: Apps undergoing bootstrap: true
2021/12/15 21:19:39 clusterSetup.go:378: App: TestUndeployDuringBootstrap started bootstrap
2021/12/15 21:19:39 clusterSetup.go:375: Apps undergoing bootstrap: true
2021/12/15 21:19:44 clusterSetup.go:367: No apps undergoing bootstrap
2021/12/15 21:19:49 clusterSetup.go:397: Waiting for app: TestUndeployDuringBootstrap to get un-deployed
2021/12/15 21:19:54 clusterSetup.go:397: Waiting for app: TestUndeployDuringBootstrap to get un-deployed
2021/12/15 21:19:59 clusterSetup.go:397: Waiting for app: TestUndeployDuringBootstrap to get un-deployed
2021/12/15 21:19:59 clusterSetup.go:405: App: TestUndeployDuringBootstrap got un-deployed
2021/12/15 21:19:59 eventingSetup.go:46: No eventing-con process running
2021/12/15 21:20:24 clusterSetup.go:397: Waiting for app: TestUndeployDuringBootstrap to get un-deployed
2021/12/15 21:20:24 clusterSetup.go:405: App: TestUndeployDuringBootstrap got un-deployed
2021/12/15 21:20:24 eventingSetup.go:46: No eventing-con process running
2021/12/15 21:20:24 eventingSetup.go:418: Function: TestUndeployDuringBootstrap delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestUndeployDuringBootstrap deleting in the background"
}
2021/12/15 21:20:26 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 21:20:36 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/15 21:20:36 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 21:20:37 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 21:20:47 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1835
2021/12/15 21:20:47 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestUndeployDuringBootstrap (104.49s)
=== RUN   TestDeleteBeforeUndeploy
2021/12/15 21:20:52 eventingSetup.go:125: Deploying app: TestDeleteBeforeUndeploy. Item count src bucket: 0 dst bucket: 0
2021/12/15 21:20:54 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeleteBeforeUndeploy' in metakv",
  "warnings": null
 }
}
2021/12/15 21:20:59 clusterSetup.go:346: Waiting for app: TestDeleteBeforeUndeploy to get deployed
2021/12/15 21:21:04 clusterSetup.go:346: Waiting for app: TestDeleteBeforeUndeploy to get deployed
2021/12/15 21:21:09 clusterSetup.go:346: Waiting for app: TestDeleteBeforeUndeploy to get deployed
2021/12/15 21:21:14 clusterSetup.go:346: Waiting for app: TestDeleteBeforeUndeploy to get deployed
2021/12/15 21:21:14 clusterSetup.go:354: App: TestDeleteBeforeUndeploy got deployed
2021/12/15 21:21:14 eventingSetup.go:418: Function: TestDeleteBeforeUndeploy delete from main store request response code: 422 dump: {
 "name": "ERR_APP_NOT_UNDEPLOYED",
 "code": 22,
 "description": "Function hasn't undeployed",
 "attributes": null,
 "runtime_info": {
  "code": 22,
  "info": "Function: TestDeleteBeforeUndeploy skipping delete request from primary store, as it hasn't been undeployed"
 }
}
2021/12/15 21:21:14 eventingSetup.go:270: Function: TestDeleteBeforeUndeploy update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/15 21:21:14 eventingSetup.go:418: Function: TestDeleteBeforeUndeploy delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDeleteBeforeUndeploy deleting in the background"
}
2021/12/15 21:21:19 clusterSetup.go:397: Waiting for app: TestDeleteBeforeUndeploy to get un-deployed
2021/12/15 21:21:19 clusterSetup.go:405: App: TestDeleteBeforeUndeploy got un-deployed
2021/12/15 21:21:19 eventingSetup.go:46: No eventing-con process running
2021/12/15 21:21:19 eventingSetup.go:418: Function: TestDeleteBeforeUndeploy delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDeleteBeforeUndeploy not found"
 }
}
2021/12/15 21:21:20 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 21:21:25 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 21:21:27 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 21:21:32 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDeleteBeforeUndeploy (44.59s)
=== RUN   TestUndeployWhenTimersAreFired
2021/12/15 21:21:42 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/15 21:21:42 clusterSetup.go:405: App: TestUndeployWhenTimersAreFired got un-deployed
2021/12/15 21:21:42 eventingSetup.go:46: No eventing-con process running
2021/12/15 21:21:42 eventingSetup.go:418: Function: TestUndeployWhenTimersAreFired delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestUndeployWhenTimersAreFired not found"
 }
}
2021/12/15 21:21:43 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 21:21:48 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 21:21:50 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 21:21:55 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 21:21:55 eventingSetup.go:125: Deploying app: TestUndeployWhenTimersAreFired. Item count src bucket: 0 dst bucket: 0
2021/12/15 21:21:57 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestUndeployWhenTimersAreFired' in metakv",
  "warnings": null
 }
}
2021/12/15 21:22:02 clusterSetup.go:346: Waiting for app: TestUndeployWhenTimersAreFired to get deployed
2021/12/15 21:22:07 clusterSetup.go:346: Waiting for app: TestUndeployWhenTimersAreFired to get deployed
2021/12/15 21:22:12 clusterSetup.go:346: Waiting for app: TestUndeployWhenTimersAreFired to get deployed
2021/12/15 21:22:17 clusterSetup.go:346: Waiting for app: TestUndeployWhenTimersAreFired to get deployed
2021/12/15 21:22:17 clusterSetup.go:354: App: TestUndeployWhenTimersAreFired got deployed
2021/12/15 21:22:17 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/15 21:22:47 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 21:22:52 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/15 21:22:57 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/15 21:23:02 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/15 21:23:07 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/15 21:23:07 clusterSetup.go:405: App: TestUndeployWhenTimersAreFired got un-deployed
2021/12/15 21:23:07 eventingSetup.go:46: No eventing-con process running
2021/12/15 21:24:52 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/15 21:24:52 clusterSetup.go:405: App: TestUndeployWhenTimersAreFired got un-deployed
2021/12/15 21:24:52 eventingSetup.go:46: No eventing-con process running
2021/12/15 21:24:52 eventingSetup.go:418: Function: TestUndeployWhenTimersAreFired delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestUndeployWhenTimersAreFired deleting in the background"
}
2021/12/15 21:24:53 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 21:25:03 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/15 21:25:08 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/15 21:25:08 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 21:25:10 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 21:25:20 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3184
2021/12/15 21:25:25 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3184
2021/12/15 21:25:25 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestUndeployWhenTimersAreFired (232.97s)
=== RUN   TestUndeployWithKVFailover
2021/12/15 21:25:35 clusterSetup.go:397: Waiting for app: TestUndeployWithKVFailover to get un-deployed
2021/12/15 21:25:35 clusterSetup.go:405: App: TestUndeployWithKVFailover got un-deployed
2021/12/15 21:25:35 eventingSetup.go:46: No eventing-con process running
2021/12/15 21:25:35 eventingSetup.go:418: Function: TestUndeployWithKVFailover delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestUndeployWithKVFailover not found"
 }
}
2021/12/15 21:25:36 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 21:25:41 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 21:25:43 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 21:25:48 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 21:25:48 eventingSetup.go:125: Deploying app: TestUndeployWithKVFailover. Item count src bucket: 0 dst bucket: 0
2021/12/15 21:25:49 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestUndeployWithKVFailover' in metakv",
  "warnings": null
 }
}
2021/12/15 21:25:49 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/15 21:25:56 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 21:26:01 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 21:26:06 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 21:26:11 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 21:26:16 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 21:26:21 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 21:26:26 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 1434
2021/12/15 21:26:31 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 3934
2021/12/15 21:26:31 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/15 21:26:31 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: kv to the cluster
2021/12/15 21:26:32 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/15 21:26:37 Rebalance progress: 29.16666666666666
2021/12/15 21:26:42 Rebalance progress: 29.16666666666666
2021/12/15 21:26:47 Rebalance progress: 100
2021/12/15 21:26:47 Failing over: [127.0.0.1:19003]
2021/12/15 21:26:57 Function: TestUndeployWithKVFailover update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/15 21:27:57 Removing node(s): [] from the cluster
2021/12/15 21:28:02 Rebalance progress: 29.16666666666666
2021/12/15 21:28:07 Rebalance progress: 29.16666666666666
2021/12/15 21:28:12 Rebalance progress: 100
2021/12/15 21:28:17 Waiting for app: TestUndeployWithKVFailover to get un-deployed
2021/12/15 21:28:17 App: TestUndeployWithKVFailover got un-deployed
2021/12/15 21:28:17 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 21:28:22 Waiting for app: TestUndeployWithKVFailover to get un-deployed
2021/12/15 21:28:22 App: TestUndeployWithKVFailover got un-deployed
2021/12/15 21:28:22 No eventing-con process running
2021/12/15 21:28:22 Function: TestUndeployWithKVFailover delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestUndeployWithKVFailover deleting in the background"
}
2021/12/15 21:28:23 Context: Bucket flush, response code: 200 dump: 
2021/12/15 21:28:33 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/15 21:28:38 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/15 21:28:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 21:28:40 Context: Bucket flush, response code: 200 dump: 
2021/12/15 21:28:50 Waiting for dst bucket item count to get to: 0 curr count: 3114
2021/12/15 21:28:55 Waiting for dst bucket item count to get to: 0 curr count: 3114
2021/12/15 21:28:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestUndeployWithKVFailover (209.98s)
=== RUN   TestBucketFlushWhileFnDeployed
2021/12/15 21:29:05 Waiting for app: TestBucketFlushWhileFnDeployed to get un-deployed
2021/12/15 21:29:05 App: TestBucketFlushWhileFnDeployed got un-deployed
2021/12/15 21:29:05 No eventing-con process running
2021/12/15 21:29:05 Function: TestBucketFlushWhileFnDeployed delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestBucketFlushWhileFnDeployed not found"
 }
}
2021/12/15 21:29:06 Context: Bucket flush, response code: 200 dump: 
2021/12/15 21:29:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 21:29:13 Context: Bucket flush, response code: 200 dump: 
2021/12/15 21:29:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 21:29:18 Deploying app: TestBucketFlushWhileFnDeployed. Item count src bucket: 0 dst bucket: 0
2021/12/15 21:29:19 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestBucketFlushWhileFnDeployed' in metakv",
  "warnings": null
 }
}
2021/12/15 21:29:24 Waiting for app: TestBucketFlushWhileFnDeployed to get deployed
2021/12/15 21:29:29 Waiting for app: TestBucketFlushWhileFnDeployed to get deployed
2021/12/15 21:29:34 Waiting for app: TestBucketFlushWhileFnDeployed to get deployed
2021/12/15 21:29:39 Waiting for app: TestBucketFlushWhileFnDeployed to get deployed
2021/12/15 21:29:39 App: TestBucketFlushWhileFnDeployed got deployed
2021/12/15 21:29:39 Starting bucket ops to source bucket
2021/12/15 21:30:00 Waiting for dst bucket item count to get to: 20000 curr count: 11327
2021/12/15 21:30:05 Waiting for dst bucket item count to get to: 20000 curr count: 16335
2021/12/15 21:30:05 src & dst bucket item count matched up. src bucket count: 20000 dst bucket count: 20000
2021/12/15 21:30:05 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 21:30:08 Context: Bucket flush, response code: 200 dump: 
2021/12/15 21:30:18 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/15 21:30:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 21:30:19 Context: Bucket flush, response code: 200 dump: 
2021/12/15 21:30:29 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/15 21:30:34 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/15 21:30:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 21:30:34 Starting bucket ops to source bucket
2021/12/15 21:30:47 Waiting for dst bucket item count to get to: 10000 curr count: 4789
2021/12/15 21:30:52 Waiting for dst bucket item count to get to: 10000 curr count: 4789
2021/12/15 21:30:52 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/15 21:30:52 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 21:30:52 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 21:30:57 Waiting for app: TestBucketFlushWhileFnDeployed to get un-deployed
2021/12/15 21:31:02 Waiting for app: TestBucketFlushWhileFnDeployed to get un-deployed
2021/12/15 21:31:07 Waiting for app: TestBucketFlushWhileFnDeployed to get un-deployed
2021/12/15 21:31:12 Waiting for app: TestBucketFlushWhileFnDeployed to get un-deployed
2021/12/15 21:31:12 App: TestBucketFlushWhileFnDeployed got un-deployed
2021/12/15 21:31:12 No eventing-con process running
2021/12/15 21:31:13 Function: TestBucketFlushWhileFnDeployed delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestBucketFlushWhileFnDeployed deleting in the background"
}
2021/12/15 21:31:14 Context: Bucket flush, response code: 200 dump: 
2021/12/15 21:31:24 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/15 21:31:29 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/15 21:31:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 21:31:31 Context: Bucket flush, response code: 200 dump: 
2021/12/15 21:31:41 Waiting for dst bucket item count to get to: 0 curr count: 8317
2021/12/15 21:31:46 Waiting for dst bucket item count to get to: 0 curr count: 8317
2021/12/15 21:31:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestBucketFlushWhileFnDeployed (170.50s)
=== RUN   TestTimerOverwrite
2021/12/15 21:31:56 Waiting for app: TestTimerOverwrite to get un-deployed
2021/12/15 21:31:56 App: TestTimerOverwrite got un-deployed
2021/12/15 21:31:56 No eventing-con process running
2021/12/15 21:31:56 Function: TestTimerOverwrite delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestTimerOverwrite not found"
 }
}
2021/12/15 21:31:57 Context: Bucket flush, response code: 200 dump: 
2021/12/15 21:32:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 21:32:04 Context: Bucket flush, response code: 200 dump: 
2021/12/15 21:32:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 21:32:09 Deploying app: TestTimerOverwrite. Item count src bucket: 0 dst bucket: 0
2021/12/15 21:32:10 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestTimerOverwrite' in metakv",
  "warnings": null
 }
}
2021/12/15 21:32:15 Waiting for app: TestTimerOverwrite to get deployed
2021/12/15 21:32:20 Waiting for app: TestTimerOverwrite to get deployed
2021/12/15 21:32:25 Waiting for app: TestTimerOverwrite to get deployed
2021/12/15 21:32:30 Waiting for app: TestTimerOverwrite to get deployed
2021/12/15 21:32:30 App: TestTimerOverwrite got deployed
2021/12/15 21:32:40 Starting bucket ops to source bucket
2021/12/15 21:32:49 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 21:32:54 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 21:32:59 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 21:33:04 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 21:33:09 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 21:33:14 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 21:33:19 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 21:33:24 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 21:33:29 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 21:33:34 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 21:33:39 Waiting for dst bucket item count to get to: 5000 curr count: 625
2021/12/15 21:33:44 Waiting for dst bucket item count to get to: 5000 curr count: 625
2021/12/15 21:33:44 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/15 21:33:54 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 21:33:54 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 21:33:59 Waiting for app: TestTimerOverwrite to get un-deployed
2021/12/15 21:34:04 Waiting for app: TestTimerOverwrite to get un-deployed
2021/12/15 21:34:09 Waiting for app: TestTimerOverwrite to get un-deployed
2021/12/15 21:34:14 Waiting for app: TestTimerOverwrite to get un-deployed
2021/12/15 21:34:14 App: TestTimerOverwrite got un-deployed
2021/12/15 21:34:14 No eventing-con process running
2021/12/15 21:34:14 Function: TestTimerOverwrite delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestTimerOverwrite deleting in the background"
}
2021/12/15 21:34:16 Context: Bucket flush, response code: 200 dump: 
2021/12/15 21:34:26 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/15 21:34:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 21:34:27 Context: Bucket flush, response code: 200 dump: 
2021/12/15 21:34:37 Waiting for dst bucket item count to get to: 0 curr count: 2431
2021/12/15 21:34:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestTimerOverwrite (171.84s)
=== RUN   TestUndeployBackdoorDuringBootstrap
2021/12/15 21:34:42 Adding node: https://127.0.0.1:19003 with role: kv to the cluster
2021/12/15 21:34:43 Removing node(s): [] from the cluster
2021/12/15 21:34:48 Rebalance progress: 100
2021/12/15 21:34:48 Failing over: [127.0.0.1:19003]
2021/12/15 21:34:53 Waiting for app: TestUndeployBackdoorDuringBootstrap to get un-deployed
2021/12/15 21:34:53 App: TestUndeployBackdoorDuringBootstrap got un-deployed
2021/12/15 21:34:53 No eventing-con process running
2021/12/15 21:34:53 Function: TestUndeployBackdoorDuringBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestUndeployBackdoorDuringBootstrap not found"
 }
}
2021/12/15 21:34:54 Context: Bucket flush, response code: 200 dump: 
2021/12/15 21:34:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 21:35:01 Context: Bucket flush, response code: 200 dump: 
2021/12/15 21:35:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 21:35:06 Deploying app: TestUndeployBackdoorDuringBootstrap. Item count src bucket: 0 dst bucket: 0
2021/12/15 21:35:07 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestUndeployBackdoorDuringBootstrap' in metakv",
  "warnings": null
 }
}
2021/12/15 21:35:07 Starting bucket ops to source bucket
2021/12/15 21:35:17 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 21:35:17 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 21:35:17 Context: Post to set retry counter, response code: 200 dump: 
2021/12/15 21:35:17 set retry counter, response &{[] }
2021/12/15 21:36:22 Waiting for app: TestUndeployBackdoorDuringBootstrap to get un-deployed
2021/12/15 21:36:22 App: TestUndeployBackdoorDuringBootstrap got un-deployed
2021/12/15 21:36:22 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 21:36:22 Function: TestUndeployBackdoorDuringBootstrap delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestUndeployBackdoorDuringBootstrap deleting in the background"
}
2021/12/15 21:36:27 Waiting for app: TestUndeployBackdoorDuringBootstrap to get un-deployed
2021/12/15 21:36:27 App: TestUndeployBackdoorDuringBootstrap got un-deployed
2021/12/15 21:36:27 No eventing-con process running
2021/12/15 21:36:27 Function: TestUndeployBackdoorDuringBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestUndeployBackdoorDuringBootstrap not found"
 }
}
2021/12/15 21:36:29 Context: Bucket flush, response code: 200 dump: 
2021/12/15 21:36:39 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/15 21:36:44 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/15 21:36:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 21:36:45 Context: Bucket flush, response code: 200 dump: 
2021/12/15 21:36:55 Waiting for dst bucket item count to get to: 0 curr count: 2301
2021/12/15 21:36:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 21:36:55 Removing node(s): [] from the cluster
2021/12/15 21:37:00 Rebalance progress: 100
--- PASS: TestUndeployBackdoorDuringBootstrap (143.05s)
=== RUN   TestOnUpdateSrcMutation
2021/12/15 21:37:10 Waiting for app: TestOnUpdateSrcMutation to get un-deployed
2021/12/15 21:37:10 App: TestOnUpdateSrcMutation got un-deployed
2021/12/15 21:37:10 No eventing-con process running
2021/12/15 21:37:10 Function: TestOnUpdateSrcMutation delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateSrcMutation not found"
 }
}
2021/12/15 21:37:12 Context: Bucket flush, response code: 200 dump: 
2021/12/15 21:37:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 21:37:18 Context: Bucket flush, response code: 200 dump: 
2021/12/15 21:37:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 21:37:24 Deploying app: TestOnUpdateSrcMutation. Item count src bucket: 0 dst bucket: 0
2021/12/15 21:37:25 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateSrcMutation' in metakv",
  "warnings": null
 }
}
2021/12/15 21:37:25 Starting bucket ops to source bucket
2021/12/15 21:37:52 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/15 21:37:52 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 21:37:52 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 21:37:57 Waiting for app: TestOnUpdateSrcMutation to get un-deployed
2021/12/15 21:38:02 Waiting for app: TestOnUpdateSrcMutation to get un-deployed
2021/12/15 21:38:07 Waiting for app: TestOnUpdateSrcMutation to get un-deployed
2021/12/15 21:38:12 Waiting for app: TestOnUpdateSrcMutation to get un-deployed
2021/12/15 21:38:12 App: TestOnUpdateSrcMutation got un-deployed
2021/12/15 21:38:12 No eventing-con process running
2021/12/15 21:38:12 Function: TestOnUpdateSrcMutation delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateSrcMutation deleting in the background"
}
2021/12/15 21:38:13 Context: Bucket flush, response code: 200 dump: 
2021/12/15 21:38:23 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/15 21:38:28 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/15 21:38:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 21:38:30 Context: Bucket flush, response code: 200 dump: 
2021/12/15 21:38:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateSrcMutation (94.61s)
=== RUN   TestOnDeleteSrcMutation
2021/12/15 21:38:45 Waiting for app: TestOnDeleteSrcMutation to get un-deployed
2021/12/15 21:38:45 App: TestOnDeleteSrcMutation got un-deployed
2021/12/15 21:38:45 No eventing-con process running
2021/12/15 21:38:45 Function: TestOnDeleteSrcMutation delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnDeleteSrcMutation not found"
 }
}
2021/12/15 21:38:46 Context: Bucket flush, response code: 200 dump: 
2021/12/15 21:38:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 21:38:53 Context: Bucket flush, response code: 200 dump: 
2021/12/15 21:38:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 21:38:58 Deploying app: TestOnDeleteSrcMutation. Item count src bucket: 0 dst bucket: 0
2021/12/15 21:38:59 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnDeleteSrcMutation' in metakv",
  "warnings": null
 }
}
2021/12/15 21:38:59 Starting bucket ops to source bucket
2021/12/15 21:39:28 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/15 21:39:28 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 21:39:28 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 21:39:33 Waiting for app: TestOnDeleteSrcMutation to get un-deployed
2021/12/15 21:39:38 Waiting for app: TestOnDeleteSrcMutation to get un-deployed
2021/12/15 21:39:43 Waiting for app: TestOnDeleteSrcMutation to get un-deployed
2021/12/15 21:39:48 Waiting for app: TestOnDeleteSrcMutation to get un-deployed
2021/12/15 21:39:48 App: TestOnDeleteSrcMutation got un-deployed
2021/12/15 21:39:48 No eventing-con process running
2021/12/15 21:39:48 Function: TestOnDeleteSrcMutation delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnDeleteSrcMutation deleting in the background"
}
2021/12/15 21:39:49 Context: Bucket flush, response code: 200 dump: 
2021/12/15 21:39:59 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/15 21:40:04 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/15 21:40:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 21:40:06 Context: Bucket flush, response code: 200 dump: 
2021/12/15 21:40:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnDeleteSrcMutation (96.26s)
=== RUN   TestOnUpdateSrcMutationWithTimer
2021/12/15 21:40:21 Waiting for app: TestOnUpdateSrcMutationWithTimer to get un-deployed
2021/12/15 21:40:21 App: TestOnUpdateSrcMutationWithTimer got un-deployed
2021/12/15 21:40:21 No eventing-con process running
2021/12/15 21:40:21 Function: TestOnUpdateSrcMutationWithTimer delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateSrcMutationWithTimer not found"
 }
}
2021/12/15 21:40:23 Context: Bucket flush, response code: 200 dump: 
2021/12/15 21:40:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 21:40:29 Context: Bucket flush, response code: 200 dump: 
2021/12/15 21:40:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 21:40:34 Deploying app: TestOnUpdateSrcMutationWithTimer. Item count src bucket: 0 dst bucket: 0
2021/12/15 21:40:36 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateSrcMutationWithTimer' in metakv",
  "warnings": null
 }
}
2021/12/15 21:40:36 Starting bucket ops to source bucket
2021/12/15 21:41:13 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/15 21:41:13 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 21:41:13 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 21:41:18 Waiting for app: TestOnUpdateSrcMutationWithTimer to get un-deployed
2021/12/15 21:41:23 Waiting for app: TestOnUpdateSrcMutationWithTimer to get un-deployed
2021/12/15 21:41:28 Waiting for app: TestOnUpdateSrcMutationWithTimer to get un-deployed
2021/12/15 21:41:33 Waiting for app: TestOnUpdateSrcMutationWithTimer to get un-deployed
2021/12/15 21:41:33 App: TestOnUpdateSrcMutationWithTimer got un-deployed
2021/12/15 21:41:33 No eventing-con process running
2021/12/15 21:41:33 Function: TestOnUpdateSrcMutationWithTimer delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateSrcMutationWithTimer deleting in the background"
}
2021/12/15 21:41:34 Context: Bucket flush, response code: 200 dump: 
2021/12/15 21:41:44 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/15 21:41:49 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/15 21:41:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 21:41:51 Context: Bucket flush, response code: 200 dump: 
2021/12/15 21:41:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateSrcMutationWithTimer (104.51s)
=== RUN   TestOnDeleteSrcMutationsWithTimer
2021/12/15 21:42:06 Waiting for app: TestOnDeleteSrcMutationsWithTimer to get un-deployed
2021/12/15 21:42:06 App: TestOnDeleteSrcMutationsWithTimer got un-deployed
2021/12/15 21:42:06 No eventing-con process running
2021/12/15 21:42:06 Function: TestOnDeleteSrcMutationsWithTimer delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnDeleteSrcMutationsWithTimer not found"
 }
}
2021/12/15 21:42:07 Context: Bucket flush, response code: 200 dump: 
2021/12/15 21:42:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 21:42:14 Context: Bucket flush, response code: 200 dump: 
2021/12/15 21:42:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 21:42:19 Deploying app: TestOnDeleteSrcMutationsWithTimer. Item count src bucket: 0 dst bucket: 0
2021/12/15 21:42:20 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnDeleteSrcMutationsWithTimer' in metakv",
  "warnings": null
 }
}
2021/12/15 21:42:20 Starting bucket ops to source bucket
2021/12/15 21:43:04 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/15 21:43:04 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 21:43:04 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 21:43:09 Waiting for app: TestOnDeleteSrcMutationsWithTimer to get un-deployed
2021/12/15 21:43:14 Waiting for app: TestOnDeleteSrcMutationsWithTimer to get un-deployed
2021/12/15 21:43:19 Waiting for app: TestOnDeleteSrcMutationsWithTimer to get un-deployed
2021/12/15 21:43:24 Waiting for app: TestOnDeleteSrcMutationsWithTimer to get un-deployed
2021/12/15 21:43:24 App: TestOnDeleteSrcMutationsWithTimer got un-deployed
2021/12/15 21:43:24 No eventing-con process running
2021/12/15 21:43:24 Function: TestOnDeleteSrcMutationsWithTimer delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnDeleteSrcMutationsWithTimer deleting in the background"
}
2021/12/15 21:43:25 Context: Bucket flush, response code: 200 dump: 
2021/12/15 21:43:35 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/15 21:43:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 21:43:37 Context: Bucket flush, response code: 200 dump: 
2021/12/15 21:43:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnDeleteSrcMutationsWithTimer (106.47s)
=== RUN   TestInterHandlerRecursion
2021/12/15 21:43:52 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/15 21:43:52 App: TestInterHandlerRecursion_source_bucket_update_op got un-deployed
2021/12/15 21:43:52 No eventing-con process running
2021/12/15 21:43:52 Function: TestInterHandlerRecursion_source_bucket_update_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestInterHandlerRecursion_source_bucket_update_op not found"
 }
}
2021/12/15 21:43:54 Context: Bucket flush, response code: 200 dump: 
2021/12/15 21:43:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 21:44:00 Context: Bucket flush, response code: 200 dump: 
2021/12/15 21:44:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 21:44:10 Waiting for app: TestInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/15 21:44:10 App: TestInterHandlerRecursion_src_bucket_op_on_delete got un-deployed
2021/12/15 21:44:10 No eventing-con process running
2021/12/15 21:44:10 Function: TestInterHandlerRecursion_src_bucket_op_on_delete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestInterHandlerRecursion_src_bucket_op_on_delete not found"
 }
}
2021/12/15 21:44:12 Context: Bucket flush, response code: 200 dump: 
2021/12/15 21:44:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 21:44:18 Context: Bucket flush, response code: 200 dump: 
2021/12/15 21:44:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 21:44:23 Deploying app: TestInterHandlerRecursion_source_bucket_update_op. Item count src bucket: 0 dst bucket: 0
2021/12/15 21:44:25 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterHandlerRecursion_source_bucket_update_op' in metakv",
  "warnings": null
 }
}
2021/12/15 21:44:25 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterHandlerRecursion_source_bucket_update_op' in metakv",
  "warnings": null
 }
} err 
2021/12/15 21:44:30 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/15 21:44:35 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/15 21:44:40 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/15 21:44:45 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/15 21:44:45 App: TestInterHandlerRecursion_source_bucket_update_op got deployed
2021/12/15 21:44:45 Deploying app: TestInterHandlerRecursion_src_bucket_op_on_delete. Item count src bucket: 0 dst bucket: 0
2021/12/15 21:44:45 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_INTER_FUNCTION_RECURSION",
 "code": 50,
 "description": "Inter function recursion error, only one function is allowed to do source bucket mutation/delete on a bucket",
 "attributes": null,
 "runtime_info": {
  "code": 50,
  "info": "Inter handler recursion error"
 }
}
2021/12/15 21:44:45 response body {
 "name": "ERR_INTER_FUNCTION_RECURSION",
 "code": 50,
 "description": "Inter function recursion error, only one function is allowed to do source bucket mutation/delete on a bucket",
 "attributes": null,
 "runtime_info": {
  "code": 50,
  "info": "Inter handler recursion error"
 }
} err 
2021/12/15 21:44:45 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 21:44:50 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/15 21:44:55 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/15 21:45:00 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/15 21:45:05 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/15 21:45:05 App: TestInterHandlerRecursion_source_bucket_update_op got un-deployed
2021/12/15 21:45:05 No eventing-con process running
2021/12/15 21:45:05 Function: TestInterHandlerRecursion_source_bucket_update_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestInterHandlerRecursion_source_bucket_update_op deleting in the background"
}
2021/12/15 21:45:06 Context: Bucket flush, response code: 200 dump: 
2021/12/15 21:45:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 21:45:13 Context: Bucket flush, response code: 200 dump: 
2021/12/15 21:45:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 21:45:23 Waiting for app: TestInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/15 21:45:23 App: TestInterHandlerRecursion_src_bucket_op_on_delete got un-deployed
2021/12/15 21:45:23 No eventing-con process running
2021/12/15 21:45:23 Function: TestInterHandlerRecursion_src_bucket_op_on_delete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestInterHandlerRecursion_src_bucket_op_on_delete not found"
 }
}
2021/12/15 21:45:24 Context: Bucket flush, response code: 200 dump: 
2021/12/15 21:45:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 21:45:31 Context: Bucket flush, response code: 200 dump: 
2021/12/15 21:45:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestInterHandlerRecursion (113.79s)
=== RUN   TestInterBucketRecursion
2021/12/15 21:45:41 Deploying app: TestInterBucketRecursion_function1. Item count src bucket: 0 dst bucket: 0
2021/12/15 21:45:42 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterBucketRecursion_function1' in metakv",
  "warnings": null
 }
}
2021/12/15 21:45:42 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterBucketRecursion_function1' in metakv",
  "warnings": null
 }
} err 
2021/12/15 21:45:42 Deploying app: TestInterBucketRecursion_function2. Item count src bucket: 0 dst bucket: 0
2021/12/15 21:45:42 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_INTER_BUCKET_RECURSION",
 "code": 51,
 "description": "Inter bucket recursion error, deployment of current handler will cause inter bucket recursion",
 "attributes": null,
 "runtime_info": {
  "code": 51,
  "info": "Inter bucket recursion error; function: TestInterBucketRecursion_function2 causes a cycle involving functions: [TestInterBucketRecursion_function1], hence deployment is disallowed"
 }
}
2021/12/15 21:45:47 Waiting for app: TestInterBucketRecursion_function1 to get deployed
2021/12/15 21:45:52 Waiting for app: TestInterBucketRecursion_function1 to get deployed
2021/12/15 21:45:57 Waiting for app: TestInterBucketRecursion_function1 to get deployed
2021/12/15 21:46:02 Waiting for app: TestInterBucketRecursion_function1 to get deployed
2021/12/15 21:46:02 App: TestInterBucketRecursion_function1 got deployed
2021/12/15 21:46:02 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 21:46:07 Waiting for app: TestInterBucketRecursion_function1 to get un-deployed
2021/12/15 21:46:12 Waiting for app: TestInterBucketRecursion_function1 to get un-deployed
2021/12/15 21:46:17 Waiting for app: TestInterBucketRecursion_function1 to get un-deployed
2021/12/15 21:46:22 Waiting for app: TestInterBucketRecursion_function1 to get un-deployed
2021/12/15 21:46:22 App: TestInterBucketRecursion_function1 got un-deployed
2021/12/15 21:46:22 No eventing-con process running
2021/12/15 21:46:22 Function: TestInterBucketRecursion_function1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestInterBucketRecursion_function1 deleting in the background"
}
2021/12/15 21:46:24 Context: Bucket flush, response code: 200 dump: 
2021/12/15 21:46:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 21:46:30 Context: Bucket flush, response code: 200 dump: 
2021/12/15 21:46:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 21:46:40 Waiting for app: TestInterBucketRecursion_function2 to get un-deployed
2021/12/15 21:46:40 App: TestInterBucketRecursion_function2 got un-deployed
2021/12/15 21:46:40 No eventing-con process running
2021/12/15 21:46:40 Function: TestInterBucketRecursion_function2 delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestInterBucketRecursion_function2 not found"
 }
}
2021/12/15 21:46:42 Context: Bucket flush, response code: 200 dump: 
2021/12/15 21:46:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 21:46:49 Context: Bucket flush, response code: 200 dump: 
2021/12/15 21:46:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestInterBucketRecursion (77.48s)
=== RUN   TestLargeHandler
2021/12/15 21:46:54 Changed force_compress value to false
2021/12/15 21:46:54 Deploying app: TestLargeHandler_function. Item count src bucket: 0 dst bucket: 0
2021/12/15 21:46:54 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_APPCODE_SIZE",
 "code": 39,
 "description": "Function Code size is more than the configured limit",
 "attributes": null,
 "runtime_info": {
  "code": 39,
  "info": "Function: TestLargeHandler_function handler Code size is more than 131072. Code Size: 131768"
 }
}
2021/12/15 21:46:54 Changed force_compress value to true
2021/12/15 21:46:54 Deploying app: TestLargeHandler_function. Item count src bucket: 0 dst bucket: 0
2021/12/15 21:46:56 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestLargeHandler_function' in metakv",
  "warnings": null
 }
}
2021/12/15 21:47:01 Waiting for app: TestLargeHandler_function to get deployed
2021/12/15 21:47:06 Waiting for app: TestLargeHandler_function to get deployed
2021/12/15 21:47:11 Waiting for app: TestLargeHandler_function to get deployed
2021/12/15 21:47:16 Waiting for app: TestLargeHandler_function to get deployed
2021/12/15 21:47:16 App: TestLargeHandler_function got deployed
2021/12/15 21:47:16 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 21:47:21 Waiting for app: TestLargeHandler_function to get un-deployed
2021/12/15 21:47:26 Waiting for app: TestLargeHandler_function to get un-deployed
2021/12/15 21:47:31 Waiting for app: TestLargeHandler_function to get un-deployed
2021/12/15 21:47:36 Waiting for app: TestLargeHandler_function to get un-deployed
2021/12/15 21:47:36 App: TestLargeHandler_function got un-deployed
2021/12/15 21:47:36 No eventing-con process running
2021/12/15 21:47:36 Function: TestLargeHandler_function delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestLargeHandler_function deleting in the background"
}
2021/12/15 21:47:37 Context: Bucket flush, response code: 200 dump: 
2021/12/15 21:47:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 21:47:44 Context: Bucket flush, response code: 200 dump: 
2021/12/15 21:47:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestLargeHandler (55.51s)
=== RUN   TestAllowInterHandlerRecursion
2021/12/15 21:47:59 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/15 21:47:59 App: TestAllowInterHandlerRecursion_source_bucket_update_op got un-deployed
2021/12/15 21:47:59 No eventing-con process running
2021/12/15 21:47:59 Function: TestAllowInterHandlerRecursion_source_bucket_update_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestAllowInterHandlerRecursion_source_bucket_update_op not found"
 }
}
2021/12/15 21:48:00 Context: Bucket flush, response code: 200 dump: 
2021/12/15 21:48:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 21:48:07 Context: Bucket flush, response code: 200 dump: 
2021/12/15 21:48:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 21:48:17 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/15 21:48:17 App: TestAllowInterHandlerRecursion_src_bucket_op_on_delete got un-deployed
2021/12/15 21:48:17 No eventing-con process running
2021/12/15 21:48:17 Function: TestAllowInterHandlerRecursion_src_bucket_op_on_delete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestAllowInterHandlerRecursion_src_bucket_op_on_delete not found"
 }
}
2021/12/15 21:48:18 Context: Bucket flush, response code: 200 dump: 
2021/12/15 21:48:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 21:48:25 Context: Bucket flush, response code: 200 dump: 
2021/12/15 21:48:30 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 21:48:30 Deploying app: TestAllowInterHandlerRecursion_source_bucket_update_op. Item count src bucket: 0 dst bucket: 0
2021/12/15 21:48:31 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterHandlerRecursion_source_bucket_update_op' in metakv",
  "warnings": null
 }
}
2021/12/15 21:48:31 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterHandlerRecursion_source_bucket_update_op' in metakv",
  "warnings": null
 }
} err 
2021/12/15 21:48:36 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/15 21:48:41 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/15 21:48:46 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/15 21:48:51 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/15 21:48:51 App: TestAllowInterHandlerRecursion_source_bucket_update_op got deployed
2021/12/15 21:48:51 Deploying app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete. Item count src bucket: 0 dst bucket: 0
2021/12/15 21:48:53 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterHandlerRecursion_src_bucket_op_on_delete' in metakv",
  "warnings": null
 }
}
2021/12/15 21:48:53 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterHandlerRecursion_src_bucket_op_on_delete' in metakv",
  "warnings": null
 }
} err 
2021/12/15 21:48:58 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get deployed
2021/12/15 21:49:03 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get deployed
2021/12/15 21:49:08 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get deployed
2021/12/15 21:49:13 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get deployed
2021/12/15 21:49:13 App: TestAllowInterHandlerRecursion_src_bucket_op_on_delete got deployed
2021/12/15 21:49:13 TestAllowInterHandlerRecursion_src_bucket_op_on_delete is deployed
2021/12/15 21:49:13 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 21:49:18 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/15 21:49:23 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/15 21:49:28 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/15 21:49:33 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/15 21:49:33 App: TestAllowInterHandlerRecursion_source_bucket_update_op got un-deployed
2021/12/15 21:49:33 No eventing-con process running
2021/12/15 21:49:33 Function: TestAllowInterHandlerRecursion_source_bucket_update_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAllowInterHandlerRecursion_source_bucket_update_op deleting in the background"
}
2021/12/15 21:49:35 Context: Bucket flush, response code: 200 dump: 
2021/12/15 21:49:40 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 21:49:42 Context: Bucket flush, response code: 200 dump: 
2021/12/15 21:49:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 21:49:47 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 21:49:52 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/15 21:49:57 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/15 21:50:02 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/15 21:50:07 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/15 21:50:07 App: TestAllowInterHandlerRecursion_src_bucket_op_on_delete got un-deployed
2021/12/15 21:50:07 No eventing-con process running
2021/12/15 21:50:07 Function: TestAllowInterHandlerRecursion_src_bucket_op_on_delete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAllowInterHandlerRecursion_src_bucket_op_on_delete deleting in the background"
}
2021/12/15 21:50:09 Context: Bucket flush, response code: 200 dump: 
2021/12/15 21:50:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 21:50:15 Context: Bucket flush, response code: 200 dump: 
2021/12/15 21:50:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAllowInterHandlerRecursion (151.32s)
=== RUN   TestAllowInterBucketRecursion
2021/12/15 21:50:25 Deploying app: TestAllowInterBucketRecursion_function1. Item count src bucket: 0 dst bucket: 0
2021/12/15 21:50:27 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterBucketRecursion_function1' in metakv",
  "warnings": null
 }
}
2021/12/15 21:50:27 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterBucketRecursion_function1' in metakv",
  "warnings": null
 }
} err 
2021/12/15 21:50:32 Waiting for app: TestAllowInterBucketRecursion_function1 to get deployed
2021/12/15 21:50:37 Waiting for app: TestAllowInterBucketRecursion_function1 to get deployed
2021/12/15 21:50:42 Waiting for app: TestAllowInterBucketRecursion_function1 to get deployed
2021/12/15 21:50:47 Waiting for app: TestAllowInterBucketRecursion_function1 to get deployed
2021/12/15 21:50:47 App: TestAllowInterBucketRecursion_function1 got deployed
2021/12/15 21:50:47 Deploying app: TestAllowInterBucketRecursion_function2. Item count src bucket: 0 dst bucket: 0
2021/12/15 21:50:48 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterBucketRecursion_function2' in metakv",
  "warnings": null
 }
}
2021/12/15 21:50:48 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterBucketRecursion_function2' in metakv",
  "warnings": null
 }
} err 
2021/12/15 21:50:53 Waiting for app: TestAllowInterBucketRecursion_function2 to get deployed
2021/12/15 21:50:58 Waiting for app: TestAllowInterBucketRecursion_function2 to get deployed
2021/12/15 21:51:03 Waiting for app: TestAllowInterBucketRecursion_function2 to get deployed
2021/12/15 21:51:08 Waiting for app: TestAllowInterBucketRecursion_function2 to get deployed
2021/12/15 21:51:08 App: TestAllowInterBucketRecursion_function2 got deployed
2021/12/15 21:51:08 TestAllowInterBucketRecursion_function2 is deployed
2021/12/15 21:51:08 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 21:51:13 Waiting for app: TestAllowInterBucketRecursion_function1 to get un-deployed
2021/12/15 21:51:18 Waiting for app: TestAllowInterBucketRecursion_function1 to get un-deployed
2021/12/15 21:51:23 Waiting for app: TestAllowInterBucketRecursion_function1 to get un-deployed
2021/12/15 21:51:28 Waiting for app: TestAllowInterBucketRecursion_function1 to get un-deployed
2021/12/15 21:51:28 App: TestAllowInterBucketRecursion_function1 got un-deployed
2021/12/15 21:51:28 No eventing-con process running
2021/12/15 21:51:28 Function: TestAllowInterBucketRecursion_function1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAllowInterBucketRecursion_function1 deleting in the background"
}
2021/12/15 21:51:29 Context: Bucket flush, response code: 200 dump: 
2021/12/15 21:51:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 21:51:37 Context: Bucket flush, response code: 200 dump: 
2021/12/15 21:51:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 21:51:43 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 21:51:48 Waiting for app: TestAllowInterBucketRecursion_function2 to get un-deployed
2021/12/15 21:51:53 Waiting for app: TestAllowInterBucketRecursion_function2 to get un-deployed
2021/12/15 21:51:58 Waiting for app: TestAllowInterBucketRecursion_function2 to get un-deployed
2021/12/15 21:52:03 Waiting for app: TestAllowInterBucketRecursion_function2 to get un-deployed
2021/12/15 21:52:03 App: TestAllowInterBucketRecursion_function2 got un-deployed
2021/12/15 21:52:03 No eventing-con process running
2021/12/15 21:52:03 Function: TestAllowInterBucketRecursion_function2 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAllowInterBucketRecursion_function2 deleting in the background"
}
2021/12/15 21:52:04 Context: Bucket flush, response code: 200 dump: 
2021/12/15 21:52:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 21:52:11 Context: Bucket flush, response code: 200 dump: 
2021/12/15 21:52:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAllowInterBucketRecursion (115.49s)
=== RUN   TestInterBucketRecursion2
2021/12/15 21:52:22 Context: Bucket flush, response code: 200 dump: 
2021/12/15 21:52:28 Context: Bucket flush, response code: 200 dump: 
2021/12/15 21:52:34 Context: Bucket flush, response code: 200 dump: 
2021/12/15 21:52:40 Context: Bucket flush, response code: 200 dump: 
2021/12/15 21:52:45 Deploying app: TestInterBucketRecursion2_function1. Item count src bucket: 0 dst bucket: 0
2021/12/15 21:52:46 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterBucketRecursion2_function1' in metakv",
  "warnings": null
 }
}
2021/12/15 21:52:51 Waiting for app: TestInterBucketRecursion2_function1 to get deployed
2021/12/15 21:52:56 Waiting for app: TestInterBucketRecursion2_function1 to get deployed
2021/12/15 21:53:01 Waiting for app: TestInterBucketRecursion2_function1 to get deployed
2021/12/15 21:53:06 Waiting for app: TestInterBucketRecursion2_function1 to get deployed
2021/12/15 21:53:06 App: TestInterBucketRecursion2_function1 got deployed
2021/12/15 21:53:06 Deploying app: TestInterBucketRecursion2_function2. Item count src bucket: 0 dst bucket: 0
2021/12/15 21:53:07 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterBucketRecursion2_function2' in metakv",
  "warnings": null
 }
}
2021/12/15 21:53:12 Waiting for app: TestInterBucketRecursion2_function2 to get deployed
2021/12/15 21:53:17 Waiting for app: TestInterBucketRecursion2_function2 to get deployed
2021/12/15 21:53:22 Waiting for app: TestInterBucketRecursion2_function2 to get deployed
2021/12/15 21:53:28 Waiting for app: TestInterBucketRecursion2_function2 to get deployed
2021/12/15 21:53:28 App: TestInterBucketRecursion2_function2 got deployed
2021/12/15 21:53:28 Deploying app: TestInterBucketRecursion2_function3. Item count src bucket: 0 dst bucket: 0
2021/12/15 21:53:29 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterBucketRecursion2_function3' in metakv",
  "warnings": null
 }
}
2021/12/15 21:53:34 Waiting for app: TestInterBucketRecursion2_function3 to get deployed
2021/12/15 21:53:39 Waiting for app: TestInterBucketRecursion2_function3 to get deployed
2021/12/15 21:53:44 Waiting for app: TestInterBucketRecursion2_function3 to get deployed
2021/12/15 21:53:49 Waiting for app: TestInterBucketRecursion2_function3 to get deployed
2021/12/15 21:53:49 App: TestInterBucketRecursion2_function3 got deployed
2021/12/15 21:53:49 Deploying app: TestInterBucketRecursion2_function4. Item count src bucket: 0 dst bucket: 0
2021/12/15 21:53:49 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_INTER_BUCKET_RECURSION",
 "code": 51,
 "description": "Inter bucket recursion error, deployment of current handler will cause inter bucket recursion",
 "attributes": null,
 "runtime_info": {
  "code": 51,
  "info": "Inter bucket recursion error; function: TestInterBucketRecursion2_function4 causes a cycle involving functions: [TestInterBucketRecursion2_function1 TestInterBucketRecursion2_function2 TestInterBucketRecursion2_function3], hence deployment is disallowed"
 }
}
2021/12/15 21:53:49 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 21:53:54 Waiting for app: TestInterBucketRecursion2_function1 to get un-deployed
2021/12/15 21:53:59 Waiting for app: TestInterBucketRecursion2_function1 to get un-deployed
2021/12/15 21:54:04 Waiting for app: TestInterBucketRecursion2_function1 to get un-deployed
2021/12/15 21:54:09 Waiting for app: TestInterBucketRecursion2_function1 to get un-deployed
2021/12/15 21:54:09 App: TestInterBucketRecursion2_function1 got un-deployed
2021/12/15 21:54:09 No eventing-con process running
2021/12/15 21:54:09 Function: TestInterBucketRecursion2_function1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestInterBucketRecursion2_function1 deleting in the background"
}
2021/12/15 21:54:10 Context: Bucket flush, response code: 200 dump: 
2021/12/15 21:54:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 21:54:17 Context: Bucket flush, response code: 200 dump: 
2021/12/15 21:54:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 21:54:22 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 21:54:27 Waiting for app: TestInterBucketRecursion2_function2 to get un-deployed
2021/12/15 21:54:32 Waiting for app: TestInterBucketRecursion2_function2 to get un-deployed
2021/12/15 21:54:37 Waiting for app: TestInterBucketRecursion2_function2 to get un-deployed
2021/12/15 21:54:42 Waiting for app: TestInterBucketRecursion2_function2 to get un-deployed
2021/12/15 21:54:42 App: TestInterBucketRecursion2_function2 got un-deployed
2021/12/15 21:54:42 No eventing-con process running
2021/12/15 21:54:42 Function: TestInterBucketRecursion2_function2 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestInterBucketRecursion2_function2 deleting in the background"
}
2021/12/15 21:54:44 Context: Bucket flush, response code: 200 dump: 
2021/12/15 21:54:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 21:54:50 Context: Bucket flush, response code: 200 dump: 
2021/12/15 21:54:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 21:54:56 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 21:55:01 Waiting for app: TestInterBucketRecursion2_function3 to get un-deployed
2021/12/15 21:55:06 Waiting for app: TestInterBucketRecursion2_function3 to get un-deployed
2021/12/15 21:55:11 Waiting for app: TestInterBucketRecursion2_function3 to get un-deployed
2021/12/15 21:55:16 Waiting for app: TestInterBucketRecursion2_function3 to get un-deployed
2021/12/15 21:55:16 App: TestInterBucketRecursion2_function3 got un-deployed
2021/12/15 21:55:16 No eventing-con process running
2021/12/15 21:55:16 Function: TestInterBucketRecursion2_function3 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestInterBucketRecursion2_function3 deleting in the background"
}
2021/12/15 21:55:17 Context: Bucket flush, response code: 200 dump: 
2021/12/15 21:55:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 21:55:24 Context: Bucket flush, response code: 200 dump: 
2021/12/15 21:55:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 21:55:34 Waiting for app: TestInterBucketRecursion2_function4 to get un-deployed
2021/12/15 21:55:34 App: TestInterBucketRecursion2_function4 got un-deployed
2021/12/15 21:55:34 No eventing-con process running
2021/12/15 21:55:34 Function: TestInterBucketRecursion2_function4 delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestInterBucketRecursion2_function4 not found"
 }
}
2021/12/15 21:55:35 Context: Bucket flush, response code: 200 dump: 
2021/12/15 21:55:40 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 21:55:42 Context: Bucket flush, response code: 200 dump: 
2021/12/15 21:55:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestInterBucketRecursion2 (213.11s)
=== RUN   TestBucketDeleteWithRebOut
2021/12/15 21:55:54 Removing node(s): [127.0.0.1:19001 127.0.0.1:19002 127.0.0.1:19003] from the cluster
2021/12/15 21:55:59 Rebalance progress: 100

VB distribution across eventing nodes and their workers::

2021/12/15 21:56:02 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/15 21:56:17 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/15 21:56:32 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/15 21:56:47 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/15 21:57:02 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/15 21:57:17 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/15 21:57:32 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/15 21:57:47 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/15 21:58:02 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/15 21:58:17 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/15 21:58:32 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/15 21:58:47 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/15 21:59:02 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/15 21:59:17 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/15 21:59:32 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/15 21:59:47 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/15 22:00:02 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/15 22:00:17 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/15 22:00:32 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/15 22:00:47 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/15 22:01:02 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/15 22:01:17 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/15 22:01:32 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/15 22:01:47 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]

DCP Stream statuses:

vbucket curr owner:


2021/12/15 22:02:02 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 22:02:02 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/15 22:02:03 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/15 22:02:08 Rebalance progress: 100

VB distribution across eventing nodes and their workers::


DCP Stream statuses:

vbucket curr owner:


2021/12/15 22:02:08 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 22:02:08 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/15 22:02:08 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/15 22:02:13 Rebalance progress: 100

VB distribution across eventing nodes and their workers::


DCP Stream statuses:

vbucket curr owner:


2021/12/15 22:02:13 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 22:02:18 clusterSetup.go:397: Waiting for app: TestBucketDeleteWithRebOut to get un-deployed
2021/12/15 22:02:18 clusterSetup.go:405: App: TestBucketDeleteWithRebOut got un-deployed
2021/12/15 22:02:18 eventingSetup.go:46: No eventing-con process running
2021/12/15 22:02:18 eventingSetup.go:418: Function: TestBucketDeleteWithRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestBucketDeleteWithRebOut not found"
 }
}
2021/12/15 22:02:20 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 22:02:25 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 22:02:27 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 22:02:32 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 22:02:37 eventingSetup.go:125: Deploying app: TestBucketDeleteWithRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/15 22:02:38 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestBucketDeleteWithRebOut' in metakv",
  "warnings": null
 }
}
2021/12/15 22:02:43 clusterSetup.go:346: Waiting for app: TestBucketDeleteWithRebOut to get deployed
2021/12/15 22:02:48 clusterSetup.go:346: Waiting for app: TestBucketDeleteWithRebOut to get deployed
2021/12/15 22:02:53 clusterSetup.go:346: Waiting for app: TestBucketDeleteWithRebOut to get deployed
2021/12/15 22:02:58 clusterSetup.go:346: Waiting for app: TestBucketDeleteWithRebOut to get deployed
2021/12/15 22:02:58 clusterSetup.go:354: App: TestBucketDeleteWithRebOut got deployed

VB distribution across eventing nodes and their workers::


DCP Stream statuses:
	stream status: running
	len: 1024
	vb list dump: [0-1023]

vbucket curr owner:
Producer node: 127.0.0.1:9303	Node UUID: 201f4904ab605131d94b62bdd8e3b413
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [342-455]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [456-569]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 113
	v: [570-682]
Producer node: 127.0.0.1:9302	Node UUID: 71dd74e37b11698f0577664c1e8e9b69
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [0-113]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [114-227]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 114
	v: [228-341]
Producer node: 172.31.2.134:9300	Node UUID: 397db3716346d2b10c24acbfeda15c2c
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 113
	v: [911-1023]
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [683-796]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [797-910]


2021/12/15 22:02:58 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 22:02:58 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/15 22:03:03 Rebalance progress: 29.16666666666666
2021/12/15 22:03:08 Rebalance progress: 29.16666666666666
2021/12/15 22:03:13 Rebalance progress: 100

VB distribution across eventing nodes and their workers::


DCP Stream statuses:
	stream status: running
	len: 1024
	vb list dump: [0-1023]

vbucket curr owner:
Producer node: 127.0.0.1:9302	Node UUID: 71dd74e37b11698f0577664c1e8e9b69
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [0-113]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [114-227]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 114
	v: [228-341]
Producer node: 172.31.2.134:9300	Node UUID: 397db3716346d2b10c24acbfeda15c2c
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 113
	v: [911-1023]
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [683-796]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [797-910]
Producer node: 127.0.0.1:9303	Node UUID: 201f4904ab605131d94b62bdd8e3b413
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [342-455]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [456-569]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 113
	v: [570-682]


2021/12/15 22:03:13 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 22:03:13 handler_test.go:1674: Deleting source bucket: default
2021/12/15 22:03:15 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/15 22:03:15 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/15 22:03:20 Rebalance progress: 29.16666666666666
2021/12/15 22:03:25 Rebalance progress: 29.16666666666666
2021/12/15 22:03:30 Rebalance progress: 100

VB distribution across eventing nodes and their workers::


DCP Stream statuses:
	stream status: stopped
	len: 126
	vb list dump: [9, 21-23, 38-39, 48, 51, 58, 74, 79, 87, 90, 105, 132, 138, 149, 160, 163, 165, 168, 170, 172, 179, 190, 193-194, 200, 203, 210, 229, 250, 268, 288, 297, 301, 315-316, 323, 331, 333, 335, 337, 340, 345, 362, 367, 370, 382, 386-388, 391, 393-394, 400, 407, 419, 427, 434, 445, 457, 463, 472, 476, 478, 480, 489, 492, 494, 499, 503, 507-508, 571, 574, 581, 586, 593, 600-601, 604, 639, 650, 659, 662, 681, 697, 702, 713, 717, 727, 739, 749, 761, 764, 767, 770, 788, 790, 794, 802, 819, 834, 839, 846, 853, 855, 861, 870, 877, 883, 889, 904, 913, 926, 932, 959-960, 975, 986, 992, 994, 996, 1008, 1017]
	stream status: running
	len: 898
	vb list dump: [0-8, 10-20, 24-37, 40-47, 49-50, 52-57, 59-73, 75-78, 80-86, 88-89, 91-104, 106-131, 133-137, 139-148, 150-159, 161-162, 164, 166-167, 169, 171, 173-178, 180-189, 191-192, 195-199, 201-202, 204-209, 211-228, 230-249, 251-267, 269-287, 289-296, 298-300, 302-314, 317-322, 324-330, 332, 334, 336, 338-339, 341-344, 346-361, 363-366, 368-369, 371-381, 383-385, 389-390, 392, 395-399, 401-406, 408-418, 420-426, 428-433, 435-444, 446-456, 458-462, 464-471, 473-475, 477, 479, 481-488, 490-491, 493, 495-498, 500-502, 504-506, 509-570, 572-573, 575-580, 582-585, 587-592, 594-599, 602-603, 605-638, 640-649, 651-658, 660-661, 663-680, 682-696, 698-701, 703-712, 714-716, 718-726, 728-738, 740-748, 750-760, 762-763, 765-766, 768-769, 771-787, 789, 791-793, 795-801, 803-818, 820-833, 835-838, 840-845, 847-852, 854, 856-860, 862-869, 871-876, 878-882, 884-888, 890-903, 905-912, 914-925, 927-931, 933-958, 961-974, 976-985, 987-991, 993, 995, 997-1007, 1009-1016, 1018-1023]

vbucket curr owner:
Producer node: 127.0.0.1:9302	Node UUID: 71dd74e37b11698f0577664c1e8e9b69
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 100
	v: [0-8, 10-20, 24-37, 40-47, 49-50, 52-57, 59-73, 75-78, 80-86, 88-89, 91-104, 106-113]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 98
	v: [114-131, 133-137, 139-148, 150-159, 161-162, 164, 166-167, 169, 171, 173-178, 180-189, 191-192, 195-199, 201-202, 204-209, 211-227]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 100
	v: [228, 230-249, 251-267, 269-287, 289-296, 298-300, 302-314, 317-322, 324-330, 332, 334, 336, 338-339, 341]
Producer node: 172.31.2.134:9300	Node UUID: 397db3716346d2b10c24acbfeda15c2c
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 101
	v: [797-801, 803-818, 820-833, 835-838, 840-845, 847-852, 854, 856-860, 862-869, 871-876, 878-882, 884-888, 890-903, 905-910]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 101
	v: [911-912, 914-925, 927-931, 933-958, 961-974, 976-985, 987-991, 993, 995, 997-1007, 1009-1016, 1018-1023]
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 100
	v: [683-696, 698-701, 703-712, 714-716, 718-726, 728-738, 740-748, 750-760, 762-763, 765-766, 768-769, 771-787, 789, 791-793, 795-796]
Producer node: 127.0.0.1:9303	Node UUID: 201f4904ab605131d94b62bdd8e3b413
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 97
	v: [342-344, 346-361, 363-366, 368-369, 371-381, 383-385, 389-390, 392, 395-399, 401-406, 408-418, 420-426, 428-433, 435-444, 446-455]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 101
	v: [456, 458-462, 464-471, 473-475, 477, 479, 481-488, 490-491, 493, 495-498, 500-502, 504-506, 509-569]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 100
	v: [570, 572-573, 575-580, 582-585, 587-592, 594-599, 602-603, 605-638, 640-649, 651-658, 660-661, 663-680, 682]


2021/12/15 22:03:31 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 22:03:36 eventingSetup.go:719: Waiting for function: TestBucketDeleteWithRebOut status to change to undeployed
2021/12/15 22:03:36 eventingSetup.go:727: Function: TestBucketDeleteWithRebOut status changed to undeployed
2021/12/15 22:03:41 clusterSetup.go:397: Waiting for app: TestBucketDeleteWithRebOut to get un-deployed
2021/12/15 22:03:41 clusterSetup.go:405: App: TestBucketDeleteWithRebOut got un-deployed
2021/12/15 22:03:41 eventingSetup.go:46: No eventing-con process running
2021/12/15 22:03:41 eventingSetup.go:418: Function: TestBucketDeleteWithRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestBucketDeleteWithRebOut deleting in the background"
}
2021/12/15 22:03:42 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 22:03:47 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 22:03:49 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 22:03:54 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 22:03:54 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv,index,n1ql to the cluster
2021/12/15 22:03:54 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/15 22:03:59 Rebalance progress: 100
--- PASS: TestBucketDeleteWithRebOut (492.90s)
=== RUN   TestN1QLRecursion
2021/12/15 22:04:02 Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/15 22:04:02 Error in adding nodes : , response: ["Prepare join failed. Node is already part of cluster."]
2021/12/15 22:04:02 Removing node(s): [] from the cluster
2021/12/15 22:04:07 Rebalance progress: 100
2021/12/15 22:04:13 Context: Bucket flush, response code: 200 dump: 
2021/12/15 22:04:19 Context: Bucket flush, response code: 200 dump: 
2021/12/15 22:04:25 Context: Bucket flush, response code: 200 dump: 
2021/12/15 22:04:30 Deploying app: TestN1QLRecursion_function1. Item count src bucket: 0 dst bucket: 0
2021/12/15 22:04:31 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLRecursion_function1' in metakv",
  "warnings": null
 }
}
2021/12/15 22:04:31 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLRecursion_function1' in metakv",
  "warnings": null
 }
} err 
2021/12/15 22:04:36 Waiting for app: TestN1QLRecursion_function1 to get deployed
2021/12/15 22:04:41 Waiting for app: TestN1QLRecursion_function1 to get deployed
2021/12/15 22:04:46 Waiting for app: TestN1QLRecursion_function1 to get deployed
2021/12/15 22:04:51 Waiting for app: TestN1QLRecursion_function1 to get deployed
2021/12/15 22:04:56 Waiting for app: TestN1QLRecursion_function1 to get deployed
2021/12/15 22:05:01 Waiting for app: TestN1QLRecursion_function1 to get deployed
2021/12/15 22:05:01 App: TestN1QLRecursion_function1 got deployed
2021/12/15 22:05:01 TestN1QLRecursion_function1 is deployed
2021/12/15 22:05:01 Deploying app: TestN1QLRecursion_function3. Item count src bucket: 0 dst bucket: 0
2021/12/15 22:05:03 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLRecursion_function3' in metakv",
  "warnings": null
 }
}
2021/12/15 22:05:03 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLRecursion_function3' in metakv",
  "warnings": null
 }
} err 
2021/12/15 22:05:08 Waiting for app: TestN1QLRecursion_function3 to get deployed
2021/12/15 22:05:13 Waiting for app: TestN1QLRecursion_function3 to get deployed
2021/12/15 22:05:18 Waiting for app: TestN1QLRecursion_function3 to get deployed
2021/12/15 22:05:23 Waiting for app: TestN1QLRecursion_function3 to get deployed
2021/12/15 22:05:23 App: TestN1QLRecursion_function3 got deployed
2021/12/15 22:05:23 TestN1QLRecursion_function3 is deployed
2021/12/15 22:05:23 Deploying app: TestN1QLRecursion_function2. Item count src bucket: 0 dst bucket: 0
2021/12/15 22:05:23 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_INTER_BUCKET_RECURSION",
 "code": 51,
 "description": "Inter bucket recursion error, deployment of current handler will cause inter bucket recursion",
 "attributes": null,
 "runtime_info": {
  "code": 51,
  "info": "Inter bucket recursion error; function: TestN1QLRecursion_function2 causes a cycle involving functions: [TestN1QLRecursion_function3 TestN1QLRecursion_function1], hence deployment is disallowed"
 }
}
2021/12/15 22:05:23 response body {
 "name": "ERR_INTER_BUCKET_RECURSION",
 "code": 51,
 "description": "Inter bucket recursion error, deployment of current handler will cause inter bucket recursion",
 "attributes": null,
 "runtime_info": {
  "code": 51,
  "info": "Inter bucket recursion error; function: TestN1QLRecursion_function2 causes a cycle involving functions: [TestN1QLRecursion_function3 TestN1QLRecursion_function1], hence deployment is disallowed"
 }
} err 
2021/12/15 22:05:23 Success: TestN1QLRecursion_function2 is not deployed
2021/12/15 22:05:23 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 22:05:28 Waiting for app: TestN1QLRecursion_function1 to get un-deployed
2021/12/15 22:05:33 Waiting for app: TestN1QLRecursion_function1 to get un-deployed
2021/12/15 22:05:38 Waiting for app: TestN1QLRecursion_function1 to get un-deployed
2021/12/15 22:05:43 Waiting for app: TestN1QLRecursion_function1 to get un-deployed
2021/12/15 22:05:43 App: TestN1QLRecursion_function1 got un-deployed
2021/12/15 22:05:43 No eventing-con process running
2021/12/15 22:05:43 Function: TestN1QLRecursion_function1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLRecursion_function1 deleting in the background"
}
2021/12/15 22:05:45 Context: Bucket flush, response code: 200 dump: 
2021/12/15 22:05:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 22:05:52 Context: Bucket flush, response code: 200 dump: 
2021/12/15 22:05:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 22:05:57 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 22:06:02 Waiting for app: TestN1QLRecursion_function3 to get un-deployed
2021/12/15 22:06:07 Waiting for app: TestN1QLRecursion_function3 to get un-deployed
2021/12/15 22:06:12 Waiting for app: TestN1QLRecursion_function3 to get un-deployed
2021/12/15 22:06:17 Waiting for app: TestN1QLRecursion_function3 to get un-deployed
2021/12/15 22:06:17 App: TestN1QLRecursion_function3 got un-deployed
2021/12/15 22:06:17 No eventing-con process running
2021/12/15 22:06:17 Function: TestN1QLRecursion_function3 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLRecursion_function3 deleting in the background"
}
2021/12/15 22:06:19 Context: Bucket flush, response code: 200 dump: 
2021/12/15 22:06:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 22:06:26 Context: Bucket flush, response code: 200 dump: 
2021/12/15 22:06:31 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 22:06:32 Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/15 22:06:37 Rebalance progress: 100
--- PASS: TestN1QLRecursion (155.60s)
=== RUN   TestN1QLAllowRecursion
2021/12/15 22:06:37 Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/15 22:06:38 Error in adding nodes : , response: ["Prepare join failed. Node is already part of cluster."]
2021/12/15 22:06:38 Removing node(s): [] from the cluster
2021/12/15 22:06:43 Rebalance progress: 100
2021/12/15 22:06:49 Context: Bucket flush, response code: 200 dump: 
2021/12/15 22:06:55 Context: Bucket flush, response code: 200 dump: 
2021/12/15 22:07:01 Context: Bucket flush, response code: 200 dump: 
2021/12/15 22:07:06 Deploying app: TestN1QLAllowRecursion_function1. Item count src bucket: 0 dst bucket: 0
2021/12/15 22:07:07 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function1' in metakv",
  "warnings": null
 }
}
2021/12/15 22:07:07 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function1' in metakv",
  "warnings": null
 }
} err 
2021/12/15 22:07:12 Waiting for app: TestN1QLAllowRecursion_function1 to get deployed
2021/12/15 22:07:17 Waiting for app: TestN1QLAllowRecursion_function1 to get deployed
2021/12/15 22:07:22 Waiting for app: TestN1QLAllowRecursion_function1 to get deployed
2021/12/15 22:07:27 Waiting for app: TestN1QLAllowRecursion_function1 to get deployed
2021/12/15 22:07:27 App: TestN1QLAllowRecursion_function1 got deployed
2021/12/15 22:07:27 TestN1QLAllowRecursion_function1 is deployed
2021/12/15 22:07:27 Deploying app: TestN1QLAllowRecursion_function3. Item count src bucket: 0 dst bucket: 0
2021/12/15 22:07:28 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function3' in metakv",
  "warnings": null
 }
}
2021/12/15 22:07:28 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function3' in metakv",
  "warnings": null
 }
} err 
2021/12/15 22:07:33 Waiting for app: TestN1QLAllowRecursion_function3 to get deployed
2021/12/15 22:07:38 Waiting for app: TestN1QLAllowRecursion_function3 to get deployed
2021/12/15 22:07:43 Waiting for app: TestN1QLAllowRecursion_function3 to get deployed
2021/12/15 22:07:48 Waiting for app: TestN1QLAllowRecursion_function3 to get deployed
2021/12/15 22:07:48 App: TestN1QLAllowRecursion_function3 got deployed
2021/12/15 22:07:48 TestN1QLAllowRecursion_function3 is deployed
2021/12/15 22:07:48 Deploying app: TestN1QLAllowRecursion_function2. Item count src bucket: 0 dst bucket: 0
2021/12/15 22:07:49 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function2' in metakv",
  "warnings": null
 }
}
2021/12/15 22:07:49 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function2' in metakv",
  "warnings": null
 }
} err 
2021/12/15 22:07:54 Waiting for app: TestN1QLAllowRecursion_function2 to get deployed
2021/12/15 22:07:59 Waiting for app: TestN1QLAllowRecursion_function2 to get deployed
2021/12/15 22:08:04 Waiting for app: TestN1QLAllowRecursion_function2 to get deployed
2021/12/15 22:08:09 Waiting for app: TestN1QLAllowRecursion_function2 to get deployed
2021/12/15 22:08:09 App: TestN1QLAllowRecursion_function2 got deployed
2021/12/15 22:08:09 Success: TestN1QLAllowRecursion_function2 is deployed after allow_interbucket_recursion
2021/12/15 22:08:09 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 22:08:14 Waiting for app: TestN1QLAllowRecursion_function1 to get un-deployed
2021/12/15 22:08:19 Waiting for app: TestN1QLAllowRecursion_function1 to get un-deployed
2021/12/15 22:08:24 Waiting for app: TestN1QLAllowRecursion_function1 to get un-deployed
2021/12/15 22:08:29 Waiting for app: TestN1QLAllowRecursion_function1 to get un-deployed
2021/12/15 22:08:29 App: TestN1QLAllowRecursion_function1 got un-deployed
2021/12/15 22:08:29 No eventing-con process running
2021/12/15 22:08:29 Function: TestN1QLAllowRecursion_function1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLAllowRecursion_function1 deleting in the background"
}
2021/12/15 22:08:31 Context: Bucket flush, response code: 200 dump: 
2021/12/15 22:08:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 22:08:38 Context: Bucket flush, response code: 200 dump: 
2021/12/15 22:08:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 22:08:43 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 22:08:48 Waiting for app: TestN1QLAllowRecursion_function2 to get un-deployed
2021/12/15 22:08:53 Waiting for app: TestN1QLAllowRecursion_function2 to get un-deployed
2021/12/15 22:08:58 Waiting for app: TestN1QLAllowRecursion_function2 to get un-deployed
2021/12/15 22:09:03 Waiting for app: TestN1QLAllowRecursion_function2 to get un-deployed
2021/12/15 22:09:03 App: TestN1QLAllowRecursion_function2 got un-deployed
2021/12/15 22:09:03 No eventing-con process running
2021/12/15 22:09:03 Function: TestN1QLAllowRecursion_function2 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLAllowRecursion_function2 deleting in the background"
}
2021/12/15 22:09:05 Context: Bucket flush, response code: 200 dump: 
2021/12/15 22:09:10 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 22:09:12 Context: Bucket flush, response code: 200 dump: 
2021/12/15 22:09:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 22:09:17 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 22:09:22 Waiting for app: TestN1QLAllowRecursion_function3 to get un-deployed
2021/12/15 22:09:27 Waiting for app: TestN1QLAllowRecursion_function3 to get un-deployed
2021/12/15 22:09:32 Waiting for app: TestN1QLAllowRecursion_function3 to get un-deployed
2021/12/15 22:09:37 Waiting for app: TestN1QLAllowRecursion_function3 to get un-deployed
2021/12/15 22:09:37 App: TestN1QLAllowRecursion_function3 got un-deployed
2021/12/15 22:09:37 No eventing-con process running
2021/12/15 22:09:37 Function: TestN1QLAllowRecursion_function3 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLAllowRecursion_function3 deleting in the background"
}
2021/12/15 22:09:39 Context: Bucket flush, response code: 200 dump: 
2021/12/15 22:09:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 22:09:46 Context: Bucket flush, response code: 200 dump: 
2021/12/15 22:09:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 22:09:53 Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/15 22:09:58 Rebalance progress: 100
--- PASS: TestN1QLAllowRecursion (200.33s)
=== RUN   TestOnDeleteExpiryBucketOp
2021/12/15 22:09:58 Sleeping for some time
2021/12/15 22:10:08 Waiting for app: TestOnDeleteExpiryBucketOp to get un-deployed
2021/12/15 22:10:08 App: TestOnDeleteExpiryBucketOp got un-deployed
2021/12/15 22:10:08 No eventing-con process running
2021/12/15 22:10:08 Function: TestOnDeleteExpiryBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnDeleteExpiryBucketOp not found"
 }
}
2021/12/15 22:10:10 Context: Bucket flush, response code: 200 dump: 
2021/12/15 22:10:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 22:10:16 Context: Bucket flush, response code: 200 dump: 
2021/12/15 22:10:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 22:10:21 Deploying app: TestOnDeleteExpiryBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/15 22:10:23 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnDeleteExpiryBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/15 22:10:23 Starting bucket ops to source bucket
2021/12/15 22:11:12 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 22:11:17 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 22:11:22 Waiting for dst bucket item count to get to: 5000 curr count: 4929
2021/12/15 22:11:27 Waiting for dst bucket item count to get to: 5000 curr count: 4929
2021/12/15 22:11:27 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/15 22:11:27 Starting bucket ops to source bucket
2021/12/15 22:11:28 Starting bucket ops to source bucket
2021/12/15 22:12:34 Waiting for dst bucket item count to get to: 7000 curr count: 5000
2021/12/15 22:12:39 Waiting for dst bucket item count to get to: 7000 curr count: 6000
2021/12/15 22:12:39 src & dst bucket item count matched up. src bucket count: 7000 dst bucket count: 7000
2021/12/15 22:12:39 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 22:12:39 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 22:12:44 Waiting for app: TestOnDeleteExpiryBucketOp to get un-deployed
2021/12/15 22:12:49 Waiting for app: TestOnDeleteExpiryBucketOp to get un-deployed
2021/12/15 22:12:54 Waiting for app: TestOnDeleteExpiryBucketOp to get un-deployed
2021/12/15 22:12:59 Waiting for app: TestOnDeleteExpiryBucketOp to get un-deployed
2021/12/15 22:12:59 App: TestOnDeleteExpiryBucketOp got un-deployed
2021/12/15 22:12:59 No eventing-con process running
2021/12/15 22:12:59 Function: TestOnDeleteExpiryBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnDeleteExpiryBucketOp deleting in the background"
}
2021/12/15 22:13:01 Context: Bucket flush, response code: 200 dump: 
2021/12/15 22:13:11 Waiting for dst bucket item count to get to: 0 curr count: 944
2021/12/15 22:13:16 Waiting for dst bucket item count to get to: 0 curr count: 944
2021/12/15 22:13:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 22:13:18 Context: Bucket flush, response code: 200 dump: 
2021/12/15 22:13:28 Waiting for dst bucket item count to get to: 0 curr count: 3500
2021/12/15 22:13:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnDeleteExpiryBucketOp (209.87s)
=== RUN   TestTimerOverWriteSameReference
2021/12/15 22:13:33 Waiting for app: TestTimerOverWriteSameReference to get un-deployed
2021/12/15 22:13:33 App: TestTimerOverWriteSameReference got un-deployed
2021/12/15 22:13:33 No eventing-con process running
2021/12/15 22:13:33 Function: TestTimerOverWriteSameReference delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestTimerOverWriteSameReference not found"
 }
}
2021/12/15 22:13:34 Context: Bucket flush, response code: 200 dump: 
2021/12/15 22:13:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 22:13:41 Context: Bucket flush, response code: 200 dump: 
2021/12/15 22:13:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 22:13:46 Deploying app: TestTimerOverWriteSameReference. Item count src bucket: 0 dst bucket: 0
2021/12/15 22:13:47 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestTimerOverWriteSameReference' in metakv",
  "warnings": null
 }
}
2021/12/15 22:13:52 Waiting for app: TestTimerOverWriteSameReference to get deployed
2021/12/15 22:13:57 Waiting for app: TestTimerOverWriteSameReference to get deployed
2021/12/15 22:14:02 Waiting for app: TestTimerOverWriteSameReference to get deployed
2021/12/15 22:14:07 Waiting for app: TestTimerOverWriteSameReference to get deployed
2021/12/15 22:14:07 App: TestTimerOverWriteSameReference got deployed
2021/12/15 22:14:17 Starting bucket ops to source bucket
2021/12/15 22:15:19 src & dst bucket item count matched up. src bucket count: 1 dst bucket count: 1
2021/12/15 22:15:29 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 22:15:29 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 22:15:34 Waiting for app: TestTimerOverWriteSameReference to get un-deployed
2021/12/15 22:15:39 Waiting for app: TestTimerOverWriteSameReference to get un-deployed
2021/12/15 22:15:44 Waiting for app: TestTimerOverWriteSameReference to get un-deployed
2021/12/15 22:15:49 Waiting for app: TestTimerOverWriteSameReference to get un-deployed
2021/12/15 22:15:49 App: TestTimerOverWriteSameReference got un-deployed
2021/12/15 22:15:49 No eventing-con process running
2021/12/15 22:15:49 Function: TestTimerOverWriteSameReference delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestTimerOverWriteSameReference deleting in the background"
}
2021/12/15 22:15:51 Context: Bucket flush, response code: 200 dump: 
2021/12/15 22:16:01 Waiting for dst bucket item count to get to: 0 curr count: 844
2021/12/15 22:16:06 Waiting for dst bucket item count to get to: 0 curr count: 844
2021/12/15 22:16:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 22:16:07 Context: Bucket flush, response code: 200 dump: 
2021/12/15 22:16:17 Waiting for dst bucket item count to get to: 0 curr count: 1
2021/12/15 22:16:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestTimerOverWriteSameReference (169.78s)
=== RUN   TestCancelTimerBucketop
2021/12/15 22:16:22 Waiting for app: TestCancelTimerBucketop to get un-deployed
2021/12/15 22:16:22 App: TestCancelTimerBucketop got un-deployed
2021/12/15 22:16:22 No eventing-con process running
2021/12/15 22:16:22 Function: TestCancelTimerBucketop delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestCancelTimerBucketop not found"
 }
}
2021/12/15 22:16:24 Context: Bucket flush, response code: 200 dump: 
2021/12/15 22:16:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 22:16:31 Context: Bucket flush, response code: 200 dump: 
2021/12/15 22:16:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 22:16:36 Deploying app: TestCancelTimerBucketop. Item count src bucket: 0 dst bucket: 0
2021/12/15 22:16:37 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCancelTimerBucketop' in metakv",
  "warnings": null
 }
}
2021/12/15 22:16:42 Waiting for app: TestCancelTimerBucketop to get deployed
2021/12/15 22:16:47 Waiting for app: TestCancelTimerBucketop to get deployed
2021/12/15 22:16:52 Waiting for app: TestCancelTimerBucketop to get deployed
2021/12/15 22:16:57 Waiting for app: TestCancelTimerBucketop to get deployed
2021/12/15 22:16:57 App: TestCancelTimerBucketop got deployed
2021/12/15 22:16:57 Starting bucket ops to source bucket
2021/12/15 22:17:05 Starting bucket ops to source bucket
2021/12/15 22:18:07 src & dst bucket item count matched up. src bucket count: 1000 dst bucket count: 1000
2021/12/15 22:18:07 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 22:18:07 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 22:18:12 Waiting for app: TestCancelTimerBucketop to get un-deployed
2021/12/15 22:18:17 Waiting for app: TestCancelTimerBucketop to get un-deployed
2021/12/15 22:18:22 Waiting for app: TestCancelTimerBucketop to get un-deployed
2021/12/15 22:18:27 Waiting for app: TestCancelTimerBucketop to get un-deployed
2021/12/15 22:18:27 App: TestCancelTimerBucketop got un-deployed
2021/12/15 22:18:27 No eventing-con process running
2021/12/15 22:18:27 Function: TestCancelTimerBucketop delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCancelTimerBucketop deleting in the background"
}
2021/12/15 22:18:29 Context: Bucket flush, response code: 200 dump: 
2021/12/15 22:18:39 Waiting for dst bucket item count to get to: 0 curr count: 951
2021/12/15 22:18:44 Waiting for dst bucket item count to get to: 0 curr count: 451
2021/12/15 22:18:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 22:18:46 Context: Bucket flush, response code: 200 dump: 
2021/12/15 22:18:56 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/15 22:18:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCancelTimerBucketop (158.25s)
=== RUN   TestJSExpiryDate
2021/12/15 22:19:01 Starting bucket ops to source bucket
2021/12/15 22:19:14 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/15 22:19:14 Deploying app: TestJSExpiryDate. Item count src bucket: 5000 dst bucket: 0
2021/12/15 22:19:15 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestJSExpiryDate' in metakv",
  "warnings": null
 }
}
2021/12/15 22:19:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 22:19:45 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 22:19:45 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 22:19:50 Waiting for app: TestJSExpiryDate to get un-deployed
2021/12/15 22:19:55 Waiting for app: TestJSExpiryDate to get un-deployed
2021/12/15 22:20:00 Waiting for app: TestJSExpiryDate to get un-deployed
2021/12/15 22:20:05 Waiting for app: TestJSExpiryDate to get un-deployed
2021/12/15 22:20:05 App: TestJSExpiryDate got un-deployed
2021/12/15 22:20:05 No eventing-con process running
2021/12/15 22:20:05 Function: TestJSExpiryDate delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestJSExpiryDate deleting in the background"
}
2021/12/15 22:20:07 Context: Bucket flush, response code: 200 dump: 
2021/12/15 22:20:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 22:20:15 Context: Bucket flush, response code: 200 dump: 
2021/12/15 22:20:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestJSExpiryDate (83.84s)
=== RUN   TestBinaryDoc
2021/12/15 22:20:25 Starting bucket ops to source bucket
2021/12/15 22:20:25 Deploying app: TestBinaryDoc. Item count src bucket: 0 dst bucket: 0
2021/12/15 22:20:26 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestBinaryDoc' in metakv",
  "warnings": null
 }
}
2021/12/15 22:20:26 Starting bucket ops to source bucket
2021/12/15 22:20:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 22:20:26 Deleting source bucket docs now
2021/12/15 22:20:26 Starting bucket ops to source bucket
2021/12/15 22:21:03 Waiting for dst bucket item count to get to: 32 curr count: 21
2021/12/15 22:21:08 Waiting for dst bucket item count to get to: 32 curr count: 31
2021/12/15 22:21:13 Waiting for dst bucket item count to get to: 32 curr count: 31
2021/12/15 22:21:13 src & dst bucket item count matched up. src bucket count: 32 dst bucket count: 32
2021/12/15 22:21:13 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 22:21:13 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 22:21:18 Waiting for app: TestBinaryDoc to get un-deployed
2021/12/15 22:21:23 Waiting for app: TestBinaryDoc to get un-deployed
2021/12/15 22:21:28 Waiting for app: TestBinaryDoc to get un-deployed
2021/12/15 22:21:33 Waiting for app: TestBinaryDoc to get un-deployed
2021/12/15 22:21:33 App: TestBinaryDoc got un-deployed
2021/12/15 22:21:33 No eventing-con process running
2021/12/15 22:21:33 Function: TestBinaryDoc delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestBinaryDoc deleting in the background"
}
2021/12/15 22:21:35 Context: Bucket flush, response code: 200 dump: 
2021/12/15 22:21:45 Waiting for dst bucket item count to get to: 0 curr count: 32
2021/12/15 22:21:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 22:21:47 Context: Bucket flush, response code: 200 dump: 
2021/12/15 22:21:57 Waiting for dst bucket item count to get to: 0 curr count: 32
2021/12/15 22:21:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestBinaryDoc (97.29s)
=== RUN   TestConstantBindings
2021/12/15 22:21:57 Deploying app: TestConstantBindings. Item count src bucket: 0 dst bucket: 0
2021/12/15 22:21:58 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestConstantBindings' in metakv",
  "warnings": null
 }
}
2021/12/15 22:21:58 Starting bucket ops to source bucket
2021/12/15 22:22:04 Waiting for dst bucket item count to get to: 6000 curr count: 0
2021/12/15 22:22:09 Waiting for dst bucket item count to get to: 6000 curr count: 0
2021/12/15 22:22:14 Waiting for dst bucket item count to get to: 6000 curr count: 0
2021/12/15 22:22:19 Waiting for dst bucket item count to get to: 6000 curr count: 22
2021/12/15 22:22:24 Waiting for dst bucket item count to get to: 6000 curr count: 22
2021/12/15 22:22:24 src & dst bucket item count matched up. src bucket count: 6000 dst bucket count: 6000
2021/12/15 22:22:24 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 22:22:24 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 22:22:29 Waiting for app: TestConstantBindings to get un-deployed
2021/12/15 22:22:34 Waiting for app: TestConstantBindings to get un-deployed
2021/12/15 22:22:39 Waiting for app: TestConstantBindings to get un-deployed
2021/12/15 22:22:44 Waiting for app: TestConstantBindings to get un-deployed
2021/12/15 22:22:44 App: TestConstantBindings got un-deployed
2021/12/15 22:22:44 No eventing-con process running
2021/12/15 22:22:44 Function: TestConstantBindings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestConstantBindings deleting in the background"
}
2021/12/15 22:22:46 Context: Bucket flush, response code: 200 dump: 
2021/12/15 22:22:56 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/15 22:22:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 22:22:58 Context: Bucket flush, response code: 200 dump: 
2021/12/15 22:23:08 Waiting for dst bucket item count to get to: 0 curr count: 3000
2021/12/15 22:23:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestConstantBindings (70.75s)
=== RUN   TestCodeInsight
2021/12/15 22:23:13 Waiting for app: TestCodeInsight to get un-deployed
2021/12/15 22:23:13 App: TestCodeInsight got un-deployed
2021/12/15 22:23:13 No eventing-con process running
2021/12/15 22:23:13 Function: TestCodeInsight delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestCodeInsight not found"
 }
}
2021/12/15 22:23:14 Context: Bucket flush, response code: 200 dump: 
2021/12/15 22:23:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 22:23:21 Context: Bucket flush, response code: 200 dump: 
2021/12/15 22:23:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 22:23:26 Starting bucket ops to source bucket
2021/12/15 22:23:27 Deploying app: TestCodeInsight. Item count src bucket: 0 dst bucket: 0
2021/12/15 22:23:28 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCodeInsight' in metakv",
  "warnings": null
 }
}
2021/12/15 22:23:33 Waiting for app: TestCodeInsight to get deployed
2021/12/15 22:23:38 Waiting for app: TestCodeInsight to get deployed
2021/12/15 22:23:43 Waiting for app: TestCodeInsight to get deployed
2021/12/15 22:23:48 Waiting for app: TestCodeInsight to get deployed
2021/12/15 22:23:48 App: TestCodeInsight got deployed
2021/12/15 22:23:53 Waiting for dst bucket item count to get to: 1000 curr count: 0
2021/12/15 22:23:53 src & dst bucket item count matched up. src bucket count: 1000 dst bucket count: 1000
2021/12/15 22:23:53 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 22:23:58 Waiting for app: TestCodeInsight to get un-deployed
2021/12/15 22:24:03 Waiting for app: TestCodeInsight to get un-deployed
2021/12/15 22:24:08 Waiting for app: TestCodeInsight to get un-deployed
2021/12/15 22:24:13 Waiting for app: TestCodeInsight to get un-deployed
2021/12/15 22:24:13 App: TestCodeInsight got un-deployed
2021/12/15 22:24:13 No eventing-con process running
2021/12/15 22:24:13 Function: TestCodeInsight delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCodeInsight deleting in the background"
}
2021/12/15 22:24:15 Context: Bucket flush, response code: 200 dump: 
2021/12/15 22:24:25 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/15 22:24:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 22:24:27 Context: Bucket flush, response code: 200 dump: 
2021/12/15 22:24:37 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/15 22:24:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCodeInsight (89.23s)
=== RUN   TestParserTransform
--- PASS: TestParserTransform (0.05s)
=== RUN   TestParserDetect
--- PASS: TestParserDetect (0.00s)
PASS
ok  	github.com/couchbase/eventing/tests/functional_tests	13033.017s

Functional tests took 3 hours, 37 minutes and 16 seconds.

N1ql tests

2021/12/15 22:24:46 Started n1ql
Post "http://127.0.0.1:9000/pools/default": EOF
Quota setup Post "http://127.0.0.1:9000/pools/default": EOF
2021/12/15 22:24:57 Adding node: https://127.0.0.1:19001 with role: kv,index,n1ql to the cluster
2021/12/15 22:24:57 Error in adding nodes : , response: ["Prepare join failed. Node is already part of cluster."]
2021/12/15 22:24:57 Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/15 22:24:58 Removing node(s): [] from the cluster
2021/12/15 22:25:03 Rebalance progress: 100
=== RUN   TestRecursiveMutationN1QL
2021/12/15 22:25:08 Waiting for app: TestRecursiveMutationN1QL to get un-deployed
2021/12/15 22:25:08 App: TestRecursiveMutationN1QL got un-deployed
2021/12/15 22:25:08 No eventing-con process running
2021/12/15 22:25:08 Function: TestRecursiveMutationN1QL delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestRecursiveMutationN1QL not found"
 }
}
2021/12/15 22:25:09 Context: Bucket flush, response code: 200 dump: 
2021/12/15 22:25:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 22:25:16 Context: Bucket flush, response code: 200 dump: 
2021/12/15 22:25:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 22:25:21 Deploying app: TestRecursiveMutationN1QL. Item count src bucket: 0 dst bucket: 0
2021/12/15 22:25:21 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_INTER_BUCKET_RECURSION",
 "code": 51,
 "description": "Inter bucket recursion error, deployment of current handler will cause inter bucket recursion",
 "attributes": null,
 "runtime_info": {
  "code": 51,
  "info": "Function: TestRecursiveMutationN1QL N1QL DML to source keyspace default:default"
 }
}
--- PASS: TestRecursiveMutationN1QL (18.51s)
=== RUN   TestFlexReset1
2021/12/15 22:25:26 Waiting for app: TestFlexReset1 to get un-deployed
2021/12/15 22:25:26 App: TestFlexReset1 got un-deployed
2021/12/15 22:25:26 No eventing-con process running
2021/12/15 22:25:26 Function: TestFlexReset1 delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestFlexReset1 not found"
 }
}
2021/12/15 22:25:28 Context: Bucket flush, response code: 200 dump: 
2021/12/15 22:25:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 22:25:35 Context: Bucket flush, response code: 200 dump: 
2021/12/15 22:25:40 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 22:25:40 Starting bucket ops to source bucket
2021/12/15 22:25:41 Deploying app: TestFlexReset1. Item count src bucket: 0 dst bucket: 0
2021/12/15 22:25:42 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestFlexReset1' in metakv",
  "warnings": null
 }
}
2021/12/15 22:25:47 Waiting for app: TestFlexReset1 to get deployed
2021/12/15 22:25:52 Waiting for app: TestFlexReset1 to get deployed
2021/12/15 22:25:57 Waiting for app: TestFlexReset1 to get deployed
2021/12/15 22:26:02 Waiting for app: TestFlexReset1 to get deployed
2021/12/15 22:26:02 App: TestFlexReset1 got deployed
2021/12/15 22:26:02 src & dst bucket item count matched up. src bucket count: 1000 dst bucket count: 1000
2021/12/15 22:26:02 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 22:26:02 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 22:26:07 Waiting for app: TestFlexReset1 to get un-deployed
2021/12/15 22:26:12 Waiting for app: TestFlexReset1 to get un-deployed
2021/12/15 22:26:17 Waiting for app: TestFlexReset1 to get un-deployed
2021/12/15 22:26:22 Waiting for app: TestFlexReset1 to get un-deployed
2021/12/15 22:26:22 App: TestFlexReset1 got un-deployed
2021/12/15 22:26:22 No eventing-con process running
2021/12/15 22:26:22 Function: TestFlexReset1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestFlexReset1 deleting in the background"
}
2021/12/15 22:26:24 Context: Bucket flush, response code: 200 dump: 
2021/12/15 22:26:34 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/15 22:26:39 Waiting for dst bucket item count to get to: 0 curr count: 500
2021/12/15 22:26:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 22:26:41 Context: Bucket flush, response code: 200 dump: 
2021/12/15 22:26:51 Waiting for dst bucket item count to get to: 0 curr count: 832
2021/12/15 22:26:56 Waiting for dst bucket item count to get to: 0 curr count: 832
2021/12/15 22:26:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestFlexReset1 (94.28s)
=== RUN   TestFlexReset2
2021/12/15 22:27:01 Waiting for app: TestFlexReset2 to get un-deployed
2021/12/15 22:27:01 App: TestFlexReset2 got un-deployed
2021/12/15 22:27:01 No eventing-con process running
2021/12/15 22:27:01 Function: TestFlexReset2 delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestFlexReset2 not found"
 }
}
2021/12/15 22:27:02 Context: Bucket flush, response code: 200 dump: 
2021/12/15 22:27:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 22:27:09 Context: Bucket flush, response code: 200 dump: 
2021/12/15 22:27:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 22:27:14 Starting bucket ops to source bucket
2021/12/15 22:27:15 Deploying app: TestFlexReset2. Item count src bucket: 0 dst bucket: 0
2021/12/15 22:27:16 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestFlexReset2' in metakv",
  "warnings": null
 }
}
2021/12/15 22:27:21 Waiting for app: TestFlexReset2 to get deployed
2021/12/15 22:27:26 Waiting for app: TestFlexReset2 to get deployed
2021/12/15 22:27:31 Waiting for app: TestFlexReset2 to get deployed
2021/12/15 22:27:36 Waiting for app: TestFlexReset2 to get deployed
2021/12/15 22:27:36 App: TestFlexReset2 got deployed
2021/12/15 22:27:41 Waiting for dst bucket item count to get to: 1000 curr count: 206
2021/12/15 22:27:41 src & dst bucket item count matched up. src bucket count: 1000 dst bucket count: 1000
2021/12/15 22:27:41 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 22:27:41 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 22:27:46 Waiting for app: TestFlexReset2 to get un-deployed
2021/12/15 22:27:51 Waiting for app: TestFlexReset2 to get un-deployed
2021/12/15 22:27:56 Waiting for app: TestFlexReset2 to get un-deployed
2021/12/15 22:28:01 Waiting for app: TestFlexReset2 to get un-deployed
2021/12/15 22:28:01 App: TestFlexReset2 got un-deployed
2021/12/15 22:28:01 No eventing-con process running
2021/12/15 22:28:01 Function: TestFlexReset2 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestFlexReset2 deleting in the background"
}
2021/12/15 22:28:03 Context: Bucket flush, response code: 200 dump: 
2021/12/15 22:28:13 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/15 22:28:18 Waiting for dst bucket item count to get to: 0 curr count: 500
2021/12/15 22:28:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 22:28:20 Context: Bucket flush, response code: 200 dump: 
2021/12/15 22:28:30 Waiting for dst bucket item count to get to: 0 curr count: 586
2021/12/15 22:28:35 Waiting for dst bucket item count to get to: 0 curr count: 586
2021/12/15 22:28:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestFlexReset2 (99.33s)
=== RUN   TestN1QLLabelledBreak
2021/12/15 22:28:40 Waiting for app: TestN1QLLabelledBreak to get un-deployed
2021/12/15 22:28:40 App: TestN1QLLabelledBreak got un-deployed
2021/12/15 22:28:40 No eventing-con process running
2021/12/15 22:28:40 Function: TestN1QLLabelledBreak delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLLabelledBreak not found"
 }
}
2021/12/15 22:28:42 Context: Bucket flush, response code: 200 dump: 
2021/12/15 22:28:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 22:28:48 Context: Bucket flush, response code: 200 dump: 
2021/12/15 22:28:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 22:28:53 Deploying app: TestN1QLLabelledBreak. Item count src bucket: 0 dst bucket: 0
2021/12/15 22:28:55 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLLabelledBreak' in metakv",
  "warnings": null
 }
}
2021/12/15 22:29:00 Waiting for app: TestN1QLLabelledBreak to get deployed
2021/12/15 22:29:05 Waiting for app: TestN1QLLabelledBreak to get deployed
2021/12/15 22:29:10 Waiting for app: TestN1QLLabelledBreak to get deployed
2021/12/15 22:29:15 Waiting for app: TestN1QLLabelledBreak to get deployed
2021/12/15 22:29:15 App: TestN1QLLabelledBreak got deployed
2021/12/15 22:29:15 Starting bucket ops to source bucket
2021/12/15 22:29:24 Waiting for dst bucket item count to get to: 10000 curr count: 0
2021/12/15 22:29:29 Waiting for dst bucket item count to get to: 10000 curr count: 163
2021/12/15 22:29:34 Waiting for dst bucket item count to get to: 10000 curr count: 520
2021/12/15 22:29:39 Waiting for dst bucket item count to get to: 10000 curr count: 2302
2021/12/15 22:29:44 Waiting for dst bucket item count to get to: 10000 curr count: 4323
2021/12/15 22:29:49 Waiting for dst bucket item count to get to: 10000 curr count: 6367
2021/12/15 22:29:54 Waiting for dst bucket item count to get to: 10000 curr count: 8403
2021/12/15 22:29:54 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/15 22:29:54 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 22:29:54 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 22:29:59 Waiting for app: TestN1QLLabelledBreak to get un-deployed
2021/12/15 22:30:04 Waiting for app: TestN1QLLabelledBreak to get un-deployed
2021/12/15 22:30:09 Waiting for app: TestN1QLLabelledBreak to get un-deployed
2021/12/15 22:30:14 Waiting for app: TestN1QLLabelledBreak to get un-deployed
2021/12/15 22:30:14 App: TestN1QLLabelledBreak got un-deployed
2021/12/15 22:30:14 No eventing-con process running
2021/12/15 22:30:14 Function: TestN1QLLabelledBreak delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLLabelledBreak deleting in the background"
}
2021/12/15 22:30:16 Context: Bucket flush, response code: 200 dump: 
2021/12/15 22:30:26 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/15 22:30:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 22:30:28 Context: Bucket flush, response code: 200 dump: 
2021/12/15 22:30:38 Waiting for dst bucket item count to get to: 0 curr count: 6261
2021/12/15 22:30:43 Waiting for dst bucket item count to get to: 0 curr count: 1261
2021/12/15 22:30:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLLabelledBreak (127.96s)
=== RUN   TestN1QLUnlabelledBreak
2021/12/15 22:30:48 Waiting for app: TestN1QLUnlabelledBreak to get un-deployed
2021/12/15 22:30:48 App: TestN1QLUnlabelledBreak got un-deployed
2021/12/15 22:30:48 No eventing-con process running
2021/12/15 22:30:48 Function: TestN1QLUnlabelledBreak delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLUnlabelledBreak not found"
 }
}
2021/12/15 22:30:50 Context: Bucket flush, response code: 200 dump: 
2021/12/15 22:30:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 22:30:56 Context: Bucket flush, response code: 200 dump: 
2021/12/15 22:31:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 22:31:01 Deploying app: TestN1QLUnlabelledBreak. Item count src bucket: 0 dst bucket: 0
2021/12/15 22:31:03 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLUnlabelledBreak' in metakv",
  "warnings": null
 }
}
2021/12/15 22:31:08 Waiting for app: TestN1QLUnlabelledBreak to get deployed
2021/12/15 22:31:13 Waiting for app: TestN1QLUnlabelledBreak to get deployed
2021/12/15 22:31:18 Waiting for app: TestN1QLUnlabelledBreak to get deployed
2021/12/15 22:31:23 Waiting for app: TestN1QLUnlabelledBreak to get deployed
2021/12/15 22:31:23 App: TestN1QLUnlabelledBreak got deployed
2021/12/15 22:31:23 Starting bucket ops to source bucket
2021/12/15 22:31:32 Waiting for dst bucket item count to get to: 10000 curr count: 0
2021/12/15 22:31:37 Waiting for dst bucket item count to get to: 10000 curr count: 1630
2021/12/15 22:31:42 Waiting for dst bucket item count to get to: 10000 curr count: 1630
2021/12/15 22:31:47 Waiting for dst bucket item count to get to: 10000 curr count: 1630
2021/12/15 22:31:52 Waiting for dst bucket item count to get to: 10000 curr count: 7249
2021/12/15 22:31:57 Waiting for dst bucket item count to get to: 10000 curr count: 7249
2021/12/15 22:31:57 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/15 22:31:57 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 22:31:57 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 22:32:02 Waiting for app: TestN1QLUnlabelledBreak to get un-deployed
2021/12/15 22:32:07 Waiting for app: TestN1QLUnlabelledBreak to get un-deployed
2021/12/15 22:32:12 Waiting for app: TestN1QLUnlabelledBreak to get un-deployed
2021/12/15 22:32:17 Waiting for app: TestN1QLUnlabelledBreak to get un-deployed
2021/12/15 22:32:17 App: TestN1QLUnlabelledBreak got un-deployed
2021/12/15 22:32:18 No eventing-con process running
2021/12/15 22:32:18 Function: TestN1QLUnlabelledBreak delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLUnlabelledBreak deleting in the background"
}
2021/12/15 22:32:19 Context: Bucket flush, response code: 200 dump: 
2021/12/15 22:32:29 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/15 22:32:34 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/15 22:32:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 22:32:36 Context: Bucket flush, response code: 200 dump: 
2021/12/15 22:32:46 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/15 22:32:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLUnlabelledBreak (123.33s)
=== RUN   TestN1QLThrowStatement
2021/12/15 22:32:51 Waiting for app: TestN1QLThrowStatement to get un-deployed
2021/12/15 22:32:51 App: TestN1QLThrowStatement got un-deployed
2021/12/15 22:32:51 No eventing-con process running
2021/12/15 22:32:51 Function: TestN1QLThrowStatement delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLThrowStatement not found"
 }
}
2021/12/15 22:32:53 Context: Bucket flush, response code: 200 dump: 
2021/12/15 22:32:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 22:33:00 Context: Bucket flush, response code: 200 dump: 
2021/12/15 22:33:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 22:33:05 Deploying app: TestN1QLThrowStatement. Item count src bucket: 0 dst bucket: 0
2021/12/15 22:33:06 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLThrowStatement' in metakv",
  "warnings": null
 }
}
2021/12/15 22:33:11 Waiting for app: TestN1QLThrowStatement to get deployed
2021/12/15 22:33:16 Waiting for app: TestN1QLThrowStatement to get deployed
2021/12/15 22:33:21 Waiting for app: TestN1QLThrowStatement to get deployed
2021/12/15 22:33:26 Waiting for app: TestN1QLThrowStatement to get deployed
2021/12/15 22:33:26 App: TestN1QLThrowStatement got deployed
2021/12/15 22:33:26 Starting bucket ops to source bucket
2021/12/15 22:33:35 Waiting for dst bucket item count to get to: 10000 curr count: 0
2021/12/15 22:33:40 Waiting for dst bucket item count to get to: 10000 curr count: 312
2021/12/15 22:33:45 Waiting for dst bucket item count to get to: 10000 curr count: 312
2021/12/15 22:33:50 Waiting for dst bucket item count to get to: 10000 curr count: 3721
2021/12/15 22:33:55 Waiting for dst bucket item count to get to: 10000 curr count: 3721
2021/12/15 22:34:00 Waiting for dst bucket item count to get to: 10000 curr count: 7686
2021/12/15 22:34:05 Waiting for dst bucket item count to get to: 10000 curr count: 7686
2021/12/15 22:34:05 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/15 22:34:05 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 22:34:06 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 22:34:11 Waiting for app: TestN1QLThrowStatement to get un-deployed
2021/12/15 22:34:16 Waiting for app: TestN1QLThrowStatement to get un-deployed
2021/12/15 22:34:21 Waiting for app: TestN1QLThrowStatement to get un-deployed
2021/12/15 22:34:26 Waiting for app: TestN1QLThrowStatement to get un-deployed
2021/12/15 22:34:26 App: TestN1QLThrowStatement got un-deployed
2021/12/15 22:34:26 No eventing-con process running
2021/12/15 22:34:26 Function: TestN1QLThrowStatement delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLThrowStatement deleting in the background"
}
2021/12/15 22:34:28 Context: Bucket flush, response code: 200 dump: 
2021/12/15 22:34:38 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/15 22:34:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 22:34:39 Context: Bucket flush, response code: 200 dump: 
2021/12/15 22:34:49 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/15 22:34:54 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/15 22:34:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLThrowStatement (128.22s)
=== RUN   TestN1QLNestedForLoop
2021/12/15 22:34:59 Waiting for app: TestN1QLNestedForLoop to get un-deployed
2021/12/15 22:34:59 App: TestN1QLNestedForLoop got un-deployed
2021/12/15 22:34:59 No eventing-con process running
2021/12/15 22:34:59 Function: TestN1QLNestedForLoop delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLNestedForLoop not found"
 }
}
2021/12/15 22:35:01 Context: Bucket flush, response code: 200 dump: 
2021/12/15 22:35:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 22:35:08 Context: Bucket flush, response code: 200 dump: 
2021/12/15 22:35:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 22:35:13 Deploying app: TestN1QLNestedForLoop. Item count src bucket: 0 dst bucket: 0
2021/12/15 22:35:14 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLNestedForLoop' in metakv",
  "warnings": null
 }
}
2021/12/15 22:35:19 Waiting for app: TestN1QLNestedForLoop to get deployed
2021/12/15 22:35:24 Waiting for app: TestN1QLNestedForLoop to get deployed
2021/12/15 22:35:29 Waiting for app: TestN1QLNestedForLoop to get deployed
2021/12/15 22:35:34 Waiting for app: TestN1QLNestedForLoop to get deployed
2021/12/15 22:35:34 App: TestN1QLNestedForLoop got deployed
2021/12/15 22:35:34 Starting bucket ops to source bucket
2021/12/15 22:35:44 Waiting for dst bucket item count to get to: 5000 curr count: 103
2021/12/15 22:35:49 Waiting for dst bucket item count to get to: 5000 curr count: 103
2021/12/15 22:35:54 Waiting for dst bucket item count to get to: 5000 curr count: 338
2021/12/15 22:35:59 Waiting for dst bucket item count to get to: 5000 curr count: 1396
2021/12/15 22:36:04 Waiting for dst bucket item count to get to: 5000 curr count: 2570
2021/12/15 22:36:09 Waiting for dst bucket item count to get to: 5000 curr count: 3763
2021/12/15 22:36:14 Waiting for dst bucket item count to get to: 5000 curr count: 4854
2021/12/15 22:36:14 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/15 22:36:14 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 22:36:14 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 22:36:19 Waiting for app: TestN1QLNestedForLoop to get un-deployed
2021/12/15 22:36:24 Waiting for app: TestN1QLNestedForLoop to get un-deployed
2021/12/15 22:36:29 Waiting for app: TestN1QLNestedForLoop to get un-deployed
2021/12/15 22:36:34 Waiting for app: TestN1QLNestedForLoop to get un-deployed
2021/12/15 22:36:34 App: TestN1QLNestedForLoop got un-deployed
2021/12/15 22:36:34 No eventing-con process running
2021/12/15 22:36:34 Function: TestN1QLNestedForLoop delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLNestedForLoop deleting in the background"
}
2021/12/15 22:36:36 Context: Bucket flush, response code: 200 dump: 
2021/12/15 22:36:46 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/15 22:36:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 22:36:48 Context: Bucket flush, response code: 200 dump: 
2021/12/15 22:36:58 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/15 22:36:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLNestedForLoop (123.17s)
=== RUN   TestN1QLAutoClose
2021/12/15 22:37:03 Waiting for app: TestN1QLAutoClose to get un-deployed
2021/12/15 22:37:03 App: TestN1QLAutoClose got un-deployed
2021/12/15 22:37:03 No eventing-con process running
2021/12/15 22:37:03 Function: TestN1QLAutoClose delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLAutoClose not found"
 }
}
2021/12/15 22:37:04 Context: Bucket flush, response code: 200 dump: 
2021/12/15 22:37:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 22:37:11 Context: Bucket flush, response code: 200 dump: 
2021/12/15 22:37:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 22:37:16 Deploying app: TestN1QLAutoClose. Item count src bucket: 0 dst bucket: 0
2021/12/15 22:37:17 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAutoClose' in metakv",
  "warnings": null
 }
}
2021/12/15 22:37:22 Waiting for app: TestN1QLAutoClose to get deployed
2021/12/15 22:37:27 Waiting for app: TestN1QLAutoClose to get deployed
2021/12/15 22:37:32 Waiting for app: TestN1QLAutoClose to get deployed
2021/12/15 22:37:37 Waiting for app: TestN1QLAutoClose to get deployed
2021/12/15 22:37:37 App: TestN1QLAutoClose got deployed
2021/12/15 22:37:37 Starting bucket ops to source bucket
2021/12/15 22:37:43 Waiting for dst bucket item count to get to: 500 curr count: 0
2021/12/15 22:37:48 Waiting for dst bucket item count to get to: 500 curr count: 1
2021/12/15 22:37:53 Waiting for dst bucket item count to get to: 500 curr count: 1
2021/12/15 22:37:58 Waiting for dst bucket item count to get to: 500 curr count: 21
2021/12/15 22:38:03 Waiting for dst bucket item count to get to: 500 curr count: 21
2021/12/15 22:38:08 Waiting for dst bucket item count to get to: 500 curr count: 45
2021/12/15 22:38:13 Waiting for dst bucket item count to get to: 500 curr count: 45
2021/12/15 22:38:18 Waiting for dst bucket item count to get to: 500 curr count: 66
2021/12/15 22:38:23 Waiting for dst bucket item count to get to: 500 curr count: 66
2021/12/15 22:38:28 Waiting for dst bucket item count to get to: 500 curr count: 88
2021/12/15 22:38:33 Waiting for dst bucket item count to get to: 500 curr count: 97
2021/12/15 22:38:38 Waiting for dst bucket item count to get to: 500 curr count: 109
2021/12/15 22:38:43 Waiting for dst bucket item count to get to: 500 curr count: 118
2021/12/15 22:38:48 Waiting for dst bucket item count to get to: 500 curr count: 118
2021/12/15 22:38:53 Waiting for dst bucket item count to get to: 500 curr count: 130
2021/12/15 22:38:58 Waiting for dst bucket item count to get to: 500 curr count: 140
2021/12/15 22:39:03 Waiting for dst bucket item count to get to: 500 curr count: 152
2021/12/15 22:39:08 Waiting for dst bucket item count to get to: 500 curr count: 161
2021/12/15 22:39:13 Waiting for dst bucket item count to get to: 500 curr count: 172
2021/12/15 22:39:18 Waiting for dst bucket item count to get to: 500 curr count: 185
2021/12/15 22:39:23 Waiting for dst bucket item count to get to: 500 curr count: 195
2021/12/15 22:39:28 Waiting for dst bucket item count to get to: 500 curr count: 205
2021/12/15 22:39:33 Waiting for dst bucket item count to get to: 500 curr count: 217
2021/12/15 22:39:38 Waiting for dst bucket item count to get to: 500 curr count: 226
2021/12/15 22:39:43 Waiting for dst bucket item count to get to: 500 curr count: 237
2021/12/15 22:39:48 Waiting for dst bucket item count to get to: 500 curr count: 246
2021/12/15 22:39:53 Waiting for dst bucket item count to get to: 500 curr count: 256
2021/12/15 22:39:58 Waiting for dst bucket item count to get to: 500 curr count: 278
2021/12/15 22:40:03 Waiting for dst bucket item count to get to: 500 curr count: 278
2021/12/15 22:40:08 Waiting for dst bucket item count to get to: 500 curr count: 298
2021/12/15 22:40:13 Waiting for dst bucket item count to get to: 500 curr count: 298
2021/12/15 22:40:18 Waiting for dst bucket item count to get to: 500 curr count: 320
2021/12/15 22:40:23 Waiting for dst bucket item count to get to: 500 curr count: 320
2021/12/15 22:40:28 Waiting for dst bucket item count to get to: 500 curr count: 342
2021/12/15 22:40:33 Waiting for dst bucket item count to get to: 500 curr count: 342
2021/12/15 22:40:38 Waiting for dst bucket item count to get to: 500 curr count: 362
2021/12/15 22:40:43 Waiting for dst bucket item count to get to: 500 curr count: 362
2021/12/15 22:40:48 Waiting for dst bucket item count to get to: 500 curr count: 381
2021/12/15 22:40:53 Waiting for dst bucket item count to get to: 500 curr count: 381
2021/12/15 22:40:58 Waiting for dst bucket item count to get to: 500 curr count: 402
2021/12/15 22:41:03 Waiting for dst bucket item count to get to: 500 curr count: 402
2021/12/15 22:41:08 Waiting for dst bucket item count to get to: 500 curr count: 425
2021/12/15 22:41:13 Waiting for dst bucket item count to get to: 500 curr count: 425
2021/12/15 22:41:18 Waiting for dst bucket item count to get to: 500 curr count: 447
2021/12/15 22:41:23 Waiting for dst bucket item count to get to: 500 curr count: 447
2021/12/15 22:41:28 Waiting for dst bucket item count to get to: 500 curr count: 470
2021/12/15 22:41:33 Waiting for dst bucket item count to get to: 500 curr count: 470
2021/12/15 22:41:38 Waiting for dst bucket item count to get to: 500 curr count: 489
2021/12/15 22:41:43 Waiting for dst bucket item count to get to: 500 curr count: 489
2021/12/15 22:41:43 src & dst bucket item count matched up. src bucket count: 500 dst bucket count: 500
2021/12/15 22:41:43 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 22:41:43 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 22:41:48 Waiting for app: TestN1QLAutoClose to get un-deployed
2021/12/15 22:41:53 Waiting for app: TestN1QLAutoClose to get un-deployed
2021/12/15 22:41:58 Waiting for app: TestN1QLAutoClose to get un-deployed
2021/12/15 22:42:04 Waiting for app: TestN1QLAutoClose to get un-deployed
2021/12/15 22:42:04 App: TestN1QLAutoClose got un-deployed
2021/12/15 22:42:04 No eventing-con process running
2021/12/15 22:42:04 Function: TestN1QLAutoClose delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLAutoClose deleting in the background"
}
2021/12/15 22:42:05 Context: Bucket flush, response code: 200 dump: 
2021/12/15 22:42:15 Waiting for dst bucket item count to get to: 0 curr count: 500
2021/12/15 22:42:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 22:42:17 Context: Bucket flush, response code: 200 dump: 
2021/12/15 22:42:27 Waiting for dst bucket item count to get to: 0 curr count: 500
2021/12/15 22:42:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLAutoClose (329.55s)
=== RUN   TestN1QLPosParams
2021/12/15 22:42:32 Waiting for app: TestN1QLPosParams to get un-deployed
2021/12/15 22:42:32 App: TestN1QLPosParams got un-deployed
2021/12/15 22:42:32 No eventing-con process running
2021/12/15 22:42:32 Function: TestN1QLPosParams delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLPosParams not found"
 }
}
2021/12/15 22:42:34 Context: Bucket flush, response code: 200 dump: 
2021/12/15 22:42:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 22:42:41 Context: Bucket flush, response code: 200 dump: 
2021/12/15 22:42:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 22:42:46 Deploying app: TestN1QLPosParams. Item count src bucket: 0 dst bucket: 0
2021/12/15 22:42:47 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLPosParams' in metakv",
  "warnings": null
 }
}
2021/12/15 22:42:52 Waiting for app: TestN1QLPosParams to get deployed
2021/12/15 22:42:57 Waiting for app: TestN1QLPosParams to get deployed
2021/12/15 22:43:02 Waiting for app: TestN1QLPosParams to get deployed
2021/12/15 22:43:07 Waiting for app: TestN1QLPosParams to get deployed
2021/12/15 22:43:07 App: TestN1QLPosParams got deployed
2021/12/15 22:43:07 Starting bucket ops to source bucket
2021/12/15 22:43:16 Waiting for dst bucket item count to get to: 5000 curr count: 393
2021/12/15 22:43:21 Waiting for dst bucket item count to get to: 5000 curr count: 393
2021/12/15 22:43:26 Waiting for dst bucket item count to get to: 5000 curr count: 393
2021/12/15 22:43:26 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/15 22:43:26 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 22:43:27 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 22:43:32 Waiting for app: TestN1QLPosParams to get un-deployed
2021/12/15 22:43:37 Waiting for app: TestN1QLPosParams to get un-deployed
2021/12/15 22:43:42 Waiting for app: TestN1QLPosParams to get un-deployed
2021/12/15 22:43:47 Waiting for app: TestN1QLPosParams to get un-deployed
2021/12/15 22:43:47 App: TestN1QLPosParams got un-deployed
2021/12/15 22:43:47 No eventing-con process running
2021/12/15 22:43:47 Function: TestN1QLPosParams delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLPosParams deleting in the background"
}
2021/12/15 22:43:49 Context: Bucket flush, response code: 200 dump: 
2021/12/15 22:43:59 Waiting for dst bucket item count to get to: 0 curr count: 4428
2021/12/15 22:44:04 Waiting for dst bucket item count to get to: 0 curr count: 1928
2021/12/15 22:44:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 22:44:05 Context: Bucket flush, response code: 200 dump: 
2021/12/15 22:44:15 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/15 22:44:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLPosParams (108.27s)
=== RUN   TestN1QLExhaustConnPool
2021/12/15 22:44:20 Waiting for app: TestN1QLExhaustConnPool to get un-deployed
2021/12/15 22:44:20 App: TestN1QLExhaustConnPool got un-deployed
2021/12/15 22:44:20 No eventing-con process running
2021/12/15 22:44:20 Function: TestN1QLExhaustConnPool delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLExhaustConnPool not found"
 }
}
2021/12/15 22:44:22 Context: Bucket flush, response code: 200 dump: 
2021/12/15 22:44:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 22:44:29 Context: Bucket flush, response code: 200 dump: 
2021/12/15 22:44:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 22:44:34 Deploying app: TestN1QLExhaustConnPool. Item count src bucket: 0 dst bucket: 0
2021/12/15 22:44:35 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLExhaustConnPool' in metakv",
  "warnings": null
 }
}
2021/12/15 22:44:40 Waiting for app: TestN1QLExhaustConnPool to get deployed
2021/12/15 22:44:45 Waiting for app: TestN1QLExhaustConnPool to get deployed
2021/12/15 22:44:50 Waiting for app: TestN1QLExhaustConnPool to get deployed
2021/12/15 22:44:55 Waiting for app: TestN1QLExhaustConnPool to get deployed
2021/12/15 22:44:55 App: TestN1QLExhaustConnPool got deployed
2021/12/15 22:44:55 Starting bucket ops to source bucket
2021/12/15 22:45:00 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/15 22:45:05 Waiting for dst bucket item count to get to: 100 curr count: 79
2021/12/15 22:45:10 Waiting for dst bucket item count to get to: 100 curr count: 79
2021/12/15 22:45:10 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/15 22:45:10 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 22:45:11 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 22:45:16 Waiting for app: TestN1QLExhaustConnPool to get un-deployed
2021/12/15 22:45:21 Waiting for app: TestN1QLExhaustConnPool to get un-deployed
2021/12/15 22:45:26 Waiting for app: TestN1QLExhaustConnPool to get un-deployed
2021/12/15 22:45:31 Waiting for app: TestN1QLExhaustConnPool to get un-deployed
2021/12/15 22:45:31 App: TestN1QLExhaustConnPool got un-deployed
2021/12/15 22:45:31 No eventing-con process running
2021/12/15 22:45:31 Function: TestN1QLExhaustConnPool delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLExhaustConnPool deleting in the background"
}
2021/12/15 22:45:32 Context: Bucket flush, response code: 200 dump: 
2021/12/15 22:45:42 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/15 22:45:47 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/15 22:45:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 22:45:49 Context: Bucket flush, response code: 200 dump: 
2021/12/15 22:45:59 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/15 22:46:04 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/15 22:46:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLExhaustConnPool (108.82s)
=== RUN   TestN1QLGraceSufficientTimeOut
2021/12/15 22:46:09 Waiting for app: TestN1QLGraceSufficientTimeOut to get un-deployed
2021/12/15 22:46:09 App: TestN1QLGraceSufficientTimeOut got un-deployed
2021/12/15 22:46:09 No eventing-con process running
2021/12/15 22:46:09 Function: TestN1QLGraceSufficientTimeOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLGraceSufficientTimeOut not found"
 }
}
2021/12/15 22:46:11 Context: Bucket flush, response code: 200 dump: 
2021/12/15 22:46:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 22:46:18 Context: Bucket flush, response code: 200 dump: 
2021/12/15 22:46:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 22:46:23 Starting bucket ops to source bucket
2021/12/15 22:46:40 Starting bucket ops to source bucket
2021/12/15 22:46:58 Deploying app: TestN1QLGraceSufficientTimeOut. Item count src bucket: 30000 dst bucket: 13413
2021/12/15 22:46:59 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLGraceSufficientTimeOut' in metakv",
  "warnings": null
 }
}
2021/12/15 22:47:04 Waiting for app: TestN1QLGraceSufficientTimeOut to get deployed
2021/12/15 22:47:09 Waiting for app: TestN1QLGraceSufficientTimeOut to get deployed
2021/12/15 22:47:14 Waiting for app: TestN1QLGraceSufficientTimeOut to get deployed
2021/12/15 22:47:19 Waiting for app: TestN1QLGraceSufficientTimeOut to get deployed
2021/12/15 22:47:19 App: TestN1QLGraceSufficientTimeOut got deployed
2021/12/15 22:47:19 Starting bucket ops to source bucket
2021/12/15 22:47:24 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/15 22:47:29 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/15 22:47:34 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/15 22:47:39 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/15 22:47:45 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/15 22:47:50 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/15 22:47:55 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/15 22:48:00 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/15 22:48:05 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/15 22:48:10 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/15 22:48:15 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/15 22:48:15 src & dst bucket item count matched up. src bucket count: 30001 dst bucket count: 30001
2021/12/15 22:48:15 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 22:48:15 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 22:48:20 Waiting for app: TestN1QLGraceSufficientTimeOut to get un-deployed
2021/12/15 22:48:25 Waiting for app: TestN1QLGraceSufficientTimeOut to get un-deployed
2021/12/15 22:48:30 Waiting for app: TestN1QLGraceSufficientTimeOut to get un-deployed
2021/12/15 22:48:35 Waiting for app: TestN1QLGraceSufficientTimeOut to get un-deployed
2021/12/15 22:48:35 App: TestN1QLGraceSufficientTimeOut got un-deployed
2021/12/15 22:48:35 No eventing-con process running
2021/12/15 22:48:35 Function: TestN1QLGraceSufficientTimeOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLGraceSufficientTimeOut deleting in the background"
}
2021/12/15 22:48:37 Context: Bucket flush, response code: 200 dump: 
2021/12/15 22:48:47 Waiting for dst bucket item count to get to: 0 curr count: 30000
2021/12/15 22:48:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 22:48:49 Context: Bucket flush, response code: 200 dump: 
2021/12/15 22:48:59 Waiting for dst bucket item count to get to: 0 curr count: 26518
2021/12/15 22:49:04 Waiting for dst bucket item count to get to: 0 curr count: 11518
2021/12/15 22:49:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLGraceSufficientTimeOut (179.40s)
=== RUN   TestN1QLQueryConsistency
2021/12/15 22:49:09 Waiting for app: n1qlquery_consistency to get un-deployed
2021/12/15 22:49:09 App: n1qlquery_consistency got un-deployed
2021/12/15 22:49:09 No eventing-con process running
2021/12/15 22:49:09 Function: n1qlquery_consistency delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: n1qlquery_consistency not found"
 }
}
2021/12/15 22:49:10 Context: Bucket flush, response code: 200 dump: 
2021/12/15 22:49:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 22:49:17 Context: Bucket flush, response code: 200 dump: 
2021/12/15 22:49:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 22:49:22 Starting bucket ops to source bucket
2021/12/15 22:49:25 Deploying app: n1qlquery_consistency. Item count src bucket: 0 dst bucket: 0
2021/12/15 22:49:26 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'n1qlquery_consistency' in metakv",
  "warnings": null
 }
}
2021/12/15 22:49:31 Waiting for app: n1qlquery_consistency to get deployed
2021/12/15 22:49:36 Waiting for app: n1qlquery_consistency to get deployed
2021/12/15 22:49:41 Waiting for app: n1qlquery_consistency to get deployed
2021/12/15 22:49:46 Waiting for app: n1qlquery_consistency to get deployed
2021/12/15 22:49:46 App: n1qlquery_consistency got deployed
2021/12/15 22:49:51 Waiting for dst bucket item count to get to: 5000 curr count: 146
2021/12/15 22:49:56 Waiting for dst bucket item count to get to: 5000 curr count: 2646
2021/12/15 22:49:56 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/15 22:49:56 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 22:49:57 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 22:50:02 Waiting for app: n1qlquery_consistency to get un-deployed
2021/12/15 22:50:07 Waiting for app: n1qlquery_consistency to get un-deployed
2021/12/15 22:50:12 Waiting for app: n1qlquery_consistency to get un-deployed
2021/12/15 22:50:17 Waiting for app: n1qlquery_consistency to get un-deployed
2021/12/15 22:50:17 App: n1qlquery_consistency got un-deployed
2021/12/15 22:50:17 No eventing-con process running
2021/12/15 22:50:17 Function: n1qlquery_consistency delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: n1qlquery_consistency deleting in the background"
}
2021/12/15 22:50:19 Context: Bucket flush, response code: 200 dump: 
2021/12/15 22:50:29 Waiting for dst bucket item count to get to: 0 curr count: 4302
2021/12/15 22:50:34 Waiting for dst bucket item count to get to: 0 curr count: 1802
2021/12/15 22:50:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 22:50:35 Context: Bucket flush, response code: 200 dump: 
2021/12/15 22:50:45 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/15 22:50:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLQueryConsistency (101.74s)
PASS
ok  	github.com/couchbase/eventing/tests/functional_tests	1554.560s

N1ql tests took 0 hours, 25 minutes and 59 seconds.

Starting server
Starting server: attempt 1

Rebalance tests

2021/12/15 22:52:20 Started rebalance
Post "http://127.0.0.1:9000/pools/default": read tcp 127.0.0.1:58496->127.0.0.1:9000: read: connection reset by peer
Quota setup Post "http://127.0.0.1:9000/pools/default": read tcp 127.0.0.1:58496->127.0.0.1:9000: read: connection reset by peer
=== RUN   TestDuplicateWithEventingRebIn
2021/12/15 22:52:39 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
running apps fetch error json: cannot unmarshal number into Go value of type string
2021/12/15 22:52:44 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
2021/12/15 22:52:44 App: TestDuplicateWithEventingRebIn got un-deployed
2021/12/15 22:52:44 No eventing-con process running
2021/12/15 22:52:44 Function: TestDuplicateWithEventingRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithEventingRebIn not found"
 }
}
2021/12/15 22:52:46 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 22:52:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 22:52:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 22:52:52 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 22:52:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 22:52:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 22:53:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 22:53:02 Deploying app: TestDuplicateWithEventingRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/15 22:53:03 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithEventingRebIn' in metakv",
  "warnings": null
 }
}
2021/12/15 22:53:08 Waiting for app: TestDuplicateWithEventingRebIn to get deployed
2021/12/15 22:53:13 Waiting for app: TestDuplicateWithEventingRebIn to get deployed
2021/12/15 22:53:18 Waiting for app: TestDuplicateWithEventingRebIn to get deployed
2021/12/15 22:53:23 Waiting for app: TestDuplicateWithEventingRebIn to get deployed
2021/12/15 22:53:24 App: TestDuplicateWithEventingRebIn got deployed
2021/12/15 22:53:24 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/15 22:53:24 Starting bucket ops to source bucket
2021/12/15 22:53:38 Removing node(s): [] from the cluster
2021/12/15 22:53:43 Rebalance progress: 0
2021/12/15 22:53:48 Rebalance progress: 0
2021/12/15 22:53:53 Rebalance progress: 16.66666666666666
2021/12/15 22:53:58 Rebalance progress: 25
2021/12/15 22:54:03 Rebalance progress: 25
2021/12/15 22:54:08 Rebalance progress: 31.12244897959184
2021/12/15 22:54:13 Rebalance progress: 98.46938775510203
2021/12/15 22:54:18 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 22:54:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 22:54:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 22:54:18 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9302/api/v1/stats": dial tcp 127.0.0.1:9302: connect: connection refused
2021/12/15 22:54:18 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/15 22:54:18 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/15 22:54:23 Rebalance progress: 25
2021/12/15 22:54:28 Rebalance progress: 25
2021/12/15 22:54:33 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 22:54:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 22:54:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 22:54:33 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9302/api/v1/stats": dial tcp 127.0.0.1:9302: connect: connection refused
2021/12/15 22:54:33 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/15 22:54:33 Waiting for src bucket item count to get to: 200000 curr count: 90195
2021/12/15 22:54:38 Waiting for src bucket item count to get to: 200000 curr count: 106689
2021/12/15 22:54:43 Waiting for src bucket item count to get to: 200000 curr count: 106689
2021/12/15 22:54:48 Waiting for src bucket item count to get to: 200000 curr count: 123005
2021/12/15 22:54:53 Waiting for src bucket item count to get to: 200000 curr count: 123005
2021/12/15 22:54:58 Waiting for src bucket item count to get to: 200000 curr count: 139665
2021/12/15 22:55:03 Waiting for src bucket item count to get to: 200000 curr count: 139665
2021/12/15 22:55:08 Waiting for src bucket item count to get to: 200000 curr count: 155991
2021/12/15 22:55:13 Waiting for src bucket item count to get to: 200000 curr count: 172386
2021/12/15 22:55:18 Waiting for src bucket item count to get to: 200000 curr count: 172386
2021/12/15 22:55:23 Waiting for src bucket item count to get to: 200000 curr count: 188570
2021/12/15 22:55:28 Waiting for src bucket item count to get to: 200000 curr count: 188570
2021/12/15 22:55:33 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/15 22:55:33 Post SET ops. Expected item count: 200000 got 200000
2021/12/15 22:55:33 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 22:55:38 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
2021/12/15 22:55:43 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
2021/12/15 22:55:48 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
2021/12/15 22:55:53 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
2021/12/15 22:55:53 App: TestDuplicateWithEventingRebIn got un-deployed
2021/12/15 22:55:53 No eventing-con process running
2021/12/15 22:55:53 Function: TestDuplicateWithEventingRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithEventingRebIn deleting in the background"
}
2021/12/15 22:55:55 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 22:56:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 22:56:05 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/15 22:56:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 22:56:07 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 22:56:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 22:56:17 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/15 22:56:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithEventingRebIn (227.65s)
=== RUN   TestDuplicateWithEventingRebOut
2021/12/15 22:56:22 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/15 22:56:22 Removing node(s): [] from the cluster
2021/12/15 22:56:27 Rebalance progress: 100
2021/12/15 22:56:32 Waiting for app: TestDuplicateWithEventingRebOut to get un-deployed
2021/12/15 22:56:32 App: TestDuplicateWithEventingRebOut got un-deployed
2021/12/15 22:56:32 No eventing-con process running
2021/12/15 22:56:32 Function: TestDuplicateWithEventingRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithEventingRebOut not found"
 }
}
2021/12/15 22:56:34 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 22:56:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 22:56:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 22:56:40 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 22:56:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 22:56:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 22:56:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 22:56:50 Deploying app: TestDuplicateWithEventingRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/15 22:56:52 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithEventingRebOut' in metakv",
  "warnings": null
 }
}
2021/12/15 22:56:57 Waiting for app: TestDuplicateWithEventingRebOut to get deployed
2021/12/15 22:57:02 Waiting for app: TestDuplicateWithEventingRebOut to get deployed
2021/12/15 22:57:07 Waiting for app: TestDuplicateWithEventingRebOut to get deployed
2021/12/15 22:57:12 Waiting for app: TestDuplicateWithEventingRebOut to get deployed
2021/12/15 22:57:12 App: TestDuplicateWithEventingRebOut got deployed
2021/12/15 22:57:12 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/15 22:57:12 Starting bucket ops to source bucket
2021/12/15 22:57:17 Rebalance progress: 25
2021/12/15 22:57:22 Rebalance progress: 25
2021/12/15 22:57:27 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 22:57:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 22:57:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 22:57:27 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9302/api/v1/stats": dial tcp 127.0.0.1:9302: connect: connection refused
2021/12/15 22:57:27 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/15 22:57:27 Waiting for src bucket item count to get to: 200000 curr count: 12197
2021/12/15 22:57:32 Waiting for src bucket item count to get to: 200000 curr count: 28821
2021/12/15 22:57:37 Waiting for src bucket item count to get to: 200000 curr count: 28821
2021/12/15 22:57:42 Waiting for src bucket item count to get to: 200000 curr count: 45514
2021/12/15 22:57:47 Waiting for src bucket item count to get to: 200000 curr count: 45514
2021/12/15 22:57:52 Waiting for src bucket item count to get to: 200000 curr count: 62241
2021/12/15 22:57:57 Waiting for src bucket item count to get to: 200000 curr count: 62241
2021/12/15 22:58:02 Waiting for src bucket item count to get to: 200000 curr count: 62241
2021/12/15 22:58:07 Waiting for src bucket item count to get to: 200000 curr count: 78364
2021/12/15 22:58:12 Waiting for src bucket item count to get to: 200000 curr count: 78364
2021/12/15 22:58:17 Waiting for src bucket item count to get to: 200000 curr count: 94739
2021/12/15 22:58:22 Waiting for src bucket item count to get to: 200000 curr count: 94739
2021/12/15 22:58:27 Waiting for src bucket item count to get to: 200000 curr count: 111046
2021/12/15 22:58:32 Waiting for src bucket item count to get to: 200000 curr count: 111046
2021/12/15 22:58:37 Waiting for src bucket item count to get to: 200000 curr count: 127758
2021/12/15 22:58:42 Waiting for src bucket item count to get to: 200000 curr count: 127758
2021/12/15 22:58:47 Waiting for src bucket item count to get to: 200000 curr count: 144443
2021/12/15 22:58:52 Waiting for src bucket item count to get to: 200000 curr count: 144443
2021/12/15 22:58:57 Waiting for src bucket item count to get to: 200000 curr count: 161127
2021/12/15 22:59:02 Waiting for src bucket item count to get to: 200000 curr count: 161127
2021/12/15 22:59:07 Waiting for src bucket item count to get to: 200000 curr count: 177034
2021/12/15 22:59:12 Waiting for src bucket item count to get to: 200000 curr count: 177034
2021/12/15 22:59:17 Waiting for src bucket item count to get to: 200000 curr count: 193339
2021/12/15 22:59:22 Waiting for src bucket item count to get to: 200000 curr count: 193339
2021/12/15 22:59:27 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/15 22:59:27 Post SET ops. Expected item count: 200000 got 200000
2021/12/15 22:59:27 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 22:59:32 Waiting for app: TestDuplicateWithEventingRebOut to get un-deployed
2021/12/15 22:59:37 Waiting for app: TestDuplicateWithEventingRebOut to get un-deployed
2021/12/15 22:59:42 Waiting for app: TestDuplicateWithEventingRebOut to get un-deployed
2021/12/15 22:59:47 Waiting for app: TestDuplicateWithEventingRebOut to get un-deployed
2021/12/15 22:59:47 App: TestDuplicateWithEventingRebOut got un-deployed
2021/12/15 22:59:47 No eventing-con process running
2021/12/15 22:59:47 Function: TestDuplicateWithEventingRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithEventingRebOut deleting in the background"
}
2021/12/15 22:59:49 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 22:59:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 22:59:59 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/15 22:59:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 23:00:00 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:00:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 23:00:10 Waiting for dst bucket item count to get to: 0 curr count: 104324
2021/12/15 23:00:15 Waiting for dst bucket item count to get to: 0 curr count: 104324
2021/12/15 23:00:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithEventingRebOut (238.87s)
=== RUN   TestDuplicateWithEventingSwapReb
2021/12/15 23:00:26 Waiting for app: TestDuplicateWithEventingSwapReb to get un-deployed
2021/12/15 23:00:26 App: TestDuplicateWithEventingSwapReb got un-deployed
2021/12/15 23:00:26 No eventing-con process running
2021/12/15 23:00:26 Function: TestDuplicateWithEventingSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithEventingSwapReb not found"
 }
}
2021/12/15 23:00:27 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:00:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 23:00:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 23:00:34 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:00:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 23:00:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:00:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 23:00:44 Deploying app: TestDuplicateWithEventingSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/15 23:00:45 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithEventingSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/15 23:00:50 Waiting for app: TestDuplicateWithEventingSwapReb to get deployed
2021/12/15 23:00:55 Waiting for app: TestDuplicateWithEventingSwapReb to get deployed
2021/12/15 23:01:00 Waiting for app: TestDuplicateWithEventingSwapReb to get deployed
2021/12/15 23:01:05 Waiting for app: TestDuplicateWithEventingSwapReb to get deployed
2021/12/15 23:01:05 App: TestDuplicateWithEventingSwapReb got deployed
2021/12/15 23:01:05 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/15 23:01:05 Starting bucket ops to source bucket
2021/12/15 23:01:06 Removing node(s): [] from the cluster
2021/12/15 23:01:11 Rebalance progress: 25
2021/12/15 23:01:16 Rebalance progress: 25
2021/12/15 23:01:21 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:01:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:01:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 23:01:21 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9302/api/v1/stats": dial tcp 127.0.0.1:9302: connect: connection refused
2021/12/15 23:01:21 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/15 23:01:21 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/15 23:01:27 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/15 23:01:33 Rebalance progress: 16.66666666666667
2021/12/15 23:01:38 Rebalance progress: 16.66666666666667
2021/12/15 23:01:43 Rebalance progress: 16.66666666666667
2021/12/15 23:01:48 Rebalance progress: 66.66666666666667
2021/12/15 23:01:53 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:01:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:01:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 23:01:53 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/15 23:01:53 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/15 23:01:58 Rebalance progress: 16.66666666666667
2021/12/15 23:02:03 Rebalance progress: 16.66666666666667
2021/12/15 23:02:08 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:02:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:02:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 23:02:08 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/15 23:02:08 Waiting for src bucket item count to get to: 200000 curr count: 81089
2021/12/15 23:02:13 Waiting for src bucket item count to get to: 200000 curr count: 97476
2021/12/15 23:02:18 Waiting for src bucket item count to get to: 200000 curr count: 97476
2021/12/15 23:02:23 Waiting for src bucket item count to get to: 200000 curr count: 113520
2021/12/15 23:02:28 Waiting for src bucket item count to get to: 200000 curr count: 113520
2021/12/15 23:02:33 Waiting for src bucket item count to get to: 200000 curr count: 129495
2021/12/15 23:02:38 Waiting for src bucket item count to get to: 200000 curr count: 129495
2021/12/15 23:02:43 Waiting for src bucket item count to get to: 200000 curr count: 145259
2021/12/15 23:02:48 Waiting for src bucket item count to get to: 200000 curr count: 145259
2021/12/15 23:02:53 Waiting for src bucket item count to get to: 200000 curr count: 161927
2021/12/15 23:02:58 Waiting for src bucket item count to get to: 200000 curr count: 161927
2021/12/15 23:03:03 Waiting for src bucket item count to get to: 200000 curr count: 177846
2021/12/15 23:03:08 Waiting for src bucket item count to get to: 200000 curr count: 177846
2021/12/15 23:03:13 Waiting for src bucket item count to get to: 200000 curr count: 194380
2021/12/15 23:03:18 Waiting for src bucket item count to get to: 200000 curr count: 194380
2021/12/15 23:03:23 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/15 23:03:23 Post SET ops. Expected item count: 200000 got 200000
2021/12/15 23:03:23 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 23:03:28 Waiting for app: TestDuplicateWithEventingSwapReb to get un-deployed
2021/12/15 23:03:33 Waiting for app: TestDuplicateWithEventingSwapReb to get un-deployed
2021/12/15 23:03:38 Waiting for app: TestDuplicateWithEventingSwapReb to get un-deployed
2021/12/15 23:03:43 Waiting for app: TestDuplicateWithEventingSwapReb to get un-deployed
2021/12/15 23:03:43 App: TestDuplicateWithEventingSwapReb got un-deployed
2021/12/15 23:03:43 No eventing-con process running
2021/12/15 23:03:43 Function: TestDuplicateWithEventingSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithEventingSwapReb deleting in the background"
}
2021/12/15 23:03:45 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:03:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 23:03:55 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/15 23:03:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 23:03:56 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:04:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 23:04:06 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/15 23:04:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithEventingSwapReb (230.88s)
=== RUN   TestDuplicateWithKVRebIn
2021/12/15 23:04:16 Waiting for app: TestDuplicateWithKVRebIn to get un-deployed
2021/12/15 23:04:16 App: TestDuplicateWithKVRebIn got un-deployed
2021/12/15 23:04:16 No eventing-con process running
2021/12/15 23:04:16 Function: TestDuplicateWithKVRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithKVRebIn not found"
 }
}
2021/12/15 23:04:18 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:04:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 23:04:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 23:04:25 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:04:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 23:04:30 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:04:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 23:04:35 Deploying app: TestDuplicateWithKVRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/15 23:04:36 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithKVRebIn' in metakv",
  "warnings": null
 }
}
2021/12/15 23:04:41 Waiting for app: TestDuplicateWithKVRebIn to get deployed
2021/12/15 23:04:46 Waiting for app: TestDuplicateWithKVRebIn to get deployed
2021/12/15 23:04:51 Waiting for app: TestDuplicateWithKVRebIn to get deployed
2021/12/15 23:04:56 Waiting for app: TestDuplicateWithKVRebIn to get deployed
2021/12/15 23:04:56 App: TestDuplicateWithKVRebIn got deployed
2021/12/15 23:04:56 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/15 23:04:56 Starting bucket ops to source bucket
2021/12/15 23:04:57 Removing node(s): [] from the cluster
2021/12/15 23:05:02 Rebalance progress: 16.66666666666667
2021/12/15 23:05:07 Rebalance progress: 16.66666666666667
2021/12/15 23:05:12 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:05:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:05:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 23:05:12 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/15 23:05:12 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/15 23:05:17 Rebalance progress: 16.66666666666667
2021/12/15 23:05:22 Rebalance progress: 16.66666666666667
2021/12/15 23:05:27 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:05:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:05:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 23:05:27 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/15 23:05:27 Waiting for src bucket item count to get to: 200000 curr count: 36075
2021/12/15 23:05:32 Waiting for src bucket item count to get to: 200000 curr count: 52213
2021/12/15 23:05:37 Waiting for src bucket item count to get to: 200000 curr count: 52213
2021/12/15 23:05:42 Waiting for src bucket item count to get to: 200000 curr count: 68537
2021/12/15 23:05:47 Waiting for src bucket item count to get to: 200000 curr count: 68537
2021/12/15 23:05:52 Waiting for src bucket item count to get to: 200000 curr count: 84007
2021/12/15 23:05:57 Waiting for src bucket item count to get to: 200000 curr count: 84007
2021/12/15 23:06:02 Waiting for src bucket item count to get to: 200000 curr count: 99991
2021/12/15 23:06:07 Waiting for src bucket item count to get to: 200000 curr count: 99991
2021/12/15 23:06:12 Waiting for src bucket item count to get to: 200000 curr count: 116165
2021/12/15 23:06:17 Waiting for src bucket item count to get to: 200000 curr count: 116165
2021/12/15 23:06:22 Waiting for src bucket item count to get to: 200000 curr count: 132597
2021/12/15 23:06:27 Waiting for src bucket item count to get to: 200000 curr count: 132597
2021/12/15 23:06:32 Waiting for src bucket item count to get to: 200000 curr count: 132597
2021/12/15 23:06:37 Waiting for src bucket item count to get to: 200000 curr count: 148672
2021/12/15 23:06:42 Waiting for src bucket item count to get to: 200000 curr count: 148672
2021/12/15 23:06:47 Waiting for src bucket item count to get to: 200000 curr count: 164721
2021/12/15 23:06:52 Waiting for src bucket item count to get to: 200000 curr count: 164721
2021/12/15 23:06:57 Waiting for src bucket item count to get to: 200000 curr count: 180105
2021/12/15 23:07:02 Waiting for src bucket item count to get to: 200000 curr count: 180105
2021/12/15 23:07:07 Waiting for src bucket item count to get to: 200000 curr count: 196501
2021/12/15 23:07:12 Waiting for src bucket item count to get to: 200000 curr count: 196501
2021/12/15 23:07:17 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/15 23:07:17 Post SET ops. Expected item count: 200000 got 200000
2021/12/15 23:07:17 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 23:07:22 Waiting for app: TestDuplicateWithKVRebIn to get un-deployed
2021/12/15 23:07:27 Waiting for app: TestDuplicateWithKVRebIn to get un-deployed
2021/12/15 23:07:32 Waiting for app: TestDuplicateWithKVRebIn to get un-deployed
2021/12/15 23:07:37 Waiting for app: TestDuplicateWithKVRebIn to get un-deployed
2021/12/15 23:07:37 App: TestDuplicateWithKVRebIn got un-deployed
2021/12/15 23:07:37 No eventing-con process running
2021/12/15 23:07:37 Function: TestDuplicateWithKVRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithKVRebIn deleting in the background"
}
2021/12/15 23:07:39 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:07:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 23:07:49 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/15 23:07:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 23:07:51 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:07:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 23:08:01 Waiting for dst bucket item count to get to: 0 curr count: 130237
2021/12/15 23:08:06 Waiting for dst bucket item count to get to: 0 curr count: 130237
2021/12/15 23:08:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithKVRebIn (239.25s)
=== RUN   TestDuplicateWithKVRebOut
2021/12/15 23:08:11 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/15 23:08:11 Removing node(s): [] from the cluster
2021/12/15 23:08:16 Rebalance progress: 100
2021/12/15 23:08:21 Waiting for app: TestDuplicateWithKVRebOut to get un-deployed
2021/12/15 23:08:21 App: TestDuplicateWithKVRebOut got un-deployed
2021/12/15 23:08:21 No eventing-con process running
2021/12/15 23:08:21 Function: TestDuplicateWithKVRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithKVRebOut not found"
 }
}
2021/12/15 23:08:23 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:08:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 23:08:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 23:08:29 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:08:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 23:08:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:08:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 23:08:39 Deploying app: TestDuplicateWithKVRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/15 23:08:40 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithKVRebOut' in metakv",
  "warnings": null
 }
}
2021/12/15 23:08:45 Waiting for app: TestDuplicateWithKVRebOut to get deployed
2021/12/15 23:08:50 Waiting for app: TestDuplicateWithKVRebOut to get deployed
2021/12/15 23:08:56 Waiting for app: TestDuplicateWithKVRebOut to get deployed
2021/12/15 23:09:01 Waiting for app: TestDuplicateWithKVRebOut to get deployed
2021/12/15 23:09:01 App: TestDuplicateWithKVRebOut got deployed
2021/12/15 23:09:01 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/15 23:09:01 Starting bucket ops to source bucket
2021/12/15 23:09:06 Rebalance progress: 16.66666666666667
2021/12/15 23:09:11 Rebalance progress: 16.66666666666667
2021/12/15 23:09:16 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:09:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:09:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 23:09:16 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/15 23:09:16 Waiting for src bucket item count to get to: 200000 curr count: 13431
2021/12/15 23:09:21 Waiting for src bucket item count to get to: 200000 curr count: 29349
2021/12/15 23:09:26 Waiting for src bucket item count to get to: 200000 curr count: 29349
2021/12/15 23:09:31 Waiting for src bucket item count to get to: 200000 curr count: 45395
2021/12/15 23:09:36 Waiting for src bucket item count to get to: 200000 curr count: 45395
2021/12/15 23:09:41 Waiting for src bucket item count to get to: 200000 curr count: 61776
2021/12/15 23:09:46 Waiting for src bucket item count to get to: 200000 curr count: 61776
2021/12/15 23:09:51 Waiting for src bucket item count to get to: 200000 curr count: 61776
2021/12/15 23:09:56 Waiting for src bucket item count to get to: 200000 curr count: 77494
2021/12/15 23:10:01 Waiting for src bucket item count to get to: 200000 curr count: 77494
2021/12/15 23:10:06 Waiting for src bucket item count to get to: 200000 curr count: 93588
2021/12/15 23:10:11 Waiting for src bucket item count to get to: 200000 curr count: 93588
2021/12/15 23:10:16 Waiting for src bucket item count to get to: 200000 curr count: 109878
2021/12/15 23:10:21 Waiting for src bucket item count to get to: 200000 curr count: 109878
2021/12/15 23:10:26 Waiting for src bucket item count to get to: 200000 curr count: 125875
2021/12/15 23:10:31 Waiting for src bucket item count to get to: 200000 curr count: 125875
2021/12/15 23:10:36 Waiting for src bucket item count to get to: 200000 curr count: 142047
2021/12/15 23:10:41 Waiting for src bucket item count to get to: 200000 curr count: 142047
2021/12/15 23:10:46 Waiting for src bucket item count to get to: 200000 curr count: 158131
2021/12/15 23:10:51 Waiting for src bucket item count to get to: 200000 curr count: 158131
2021/12/15 23:10:56 Waiting for src bucket item count to get to: 200000 curr count: 173508
2021/12/15 23:11:01 Waiting for src bucket item count to get to: 200000 curr count: 173508
2021/12/15 23:11:06 Waiting for src bucket item count to get to: 200000 curr count: 189202
2021/12/15 23:11:11 Waiting for src bucket item count to get to: 200000 curr count: 189202
2021/12/15 23:11:16 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/15 23:11:16 Post SET ops. Expected item count: 200000 got 200000
2021/12/15 23:11:16 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 23:11:21 Waiting for app: TestDuplicateWithKVRebOut to get un-deployed
2021/12/15 23:11:26 Waiting for app: TestDuplicateWithKVRebOut to get un-deployed
2021/12/15 23:11:31 Waiting for app: TestDuplicateWithKVRebOut to get un-deployed
2021/12/15 23:11:36 Waiting for app: TestDuplicateWithKVRebOut to get un-deployed
2021/12/15 23:11:36 App: TestDuplicateWithKVRebOut got un-deployed
2021/12/15 23:11:36 No eventing-con process running
2021/12/15 23:11:36 Function: TestDuplicateWithKVRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithKVRebOut deleting in the background"
}
2021/12/15 23:11:38 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:11:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 23:11:48 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/15 23:11:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 23:11:50 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:11:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 23:11:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithKVRebOut (228.90s)
=== RUN   TestDuplicateWithKVSwapReb
2021/12/15 23:12:05 Waiting for app: TestDuplicateWithKVSwapReb to get un-deployed
2021/12/15 23:12:05 App: TestDuplicateWithKVSwapReb got un-deployed
2021/12/15 23:12:05 No eventing-con process running
2021/12/15 23:12:05 Function: TestDuplicateWithKVSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithKVSwapReb not found"
 }
}
2021/12/15 23:12:06 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:12:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 23:12:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 23:12:13 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:12:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 23:12:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:12:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 23:12:23 Deploying app: TestDuplicateWithKVSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/15 23:12:24 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithKVSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/15 23:12:29 Waiting for app: TestDuplicateWithKVSwapReb to get deployed
2021/12/15 23:12:34 Waiting for app: TestDuplicateWithKVSwapReb to get deployed
2021/12/15 23:12:39 Waiting for app: TestDuplicateWithKVSwapReb to get deployed
2021/12/15 23:12:44 Waiting for app: TestDuplicateWithKVSwapReb to get deployed
2021/12/15 23:12:44 App: TestDuplicateWithKVSwapReb got deployed
2021/12/15 23:12:44 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/15 23:12:44 Starting bucket ops to source bucket
2021/12/15 23:12:45 Removing node(s): [] from the cluster
2021/12/15 23:12:50 Rebalance progress: 16.66666666666667
2021/12/15 23:12:55 Rebalance progress: 16.66666666666667
2021/12/15 23:13:00 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:13:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:13:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 23:13:00 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/15 23:13:00 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/15 23:13:00 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/15 23:13:05 Rebalance progress: 16.66666666666667
2021/12/15 23:13:10 Rebalance progress: 16.66666666666667
2021/12/15 23:13:15 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:13:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:13:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 23:13:15 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/15 23:13:15 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/15 23:13:20 Rebalance progress: 16.66666666666667
2021/12/15 23:13:25 Rebalance progress: 16.66666666666667
2021/12/15 23:13:30 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:13:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:13:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 23:13:30 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/15 23:13:30 Waiting for src bucket item count to get to: 200000 curr count: 54310
2021/12/15 23:13:35 Waiting for src bucket item count to get to: 200000 curr count: 70595
2021/12/15 23:13:40 Waiting for src bucket item count to get to: 200000 curr count: 70595
2021/12/15 23:13:45 Waiting for src bucket item count to get to: 200000 curr count: 85881
2021/12/15 23:13:50 Waiting for src bucket item count to get to: 200000 curr count: 102215
2021/12/15 23:13:55 Waiting for src bucket item count to get to: 200000 curr count: 102215
2021/12/15 23:14:00 Waiting for src bucket item count to get to: 200000 curr count: 118563
2021/12/15 23:14:05 Waiting for src bucket item count to get to: 200000 curr count: 118563
2021/12/15 23:14:10 Waiting for src bucket item count to get to: 200000 curr count: 134804
2021/12/15 23:14:15 Waiting for src bucket item count to get to: 200000 curr count: 134804
2021/12/15 23:14:20 Waiting for src bucket item count to get to: 200000 curr count: 151405
2021/12/15 23:14:25 Waiting for src bucket item count to get to: 200000 curr count: 151405
2021/12/15 23:14:30 Waiting for src bucket item count to get to: 200000 curr count: 167819
2021/12/15 23:14:35 Waiting for src bucket item count to get to: 200000 curr count: 167819
2021/12/15 23:14:40 Waiting for src bucket item count to get to: 200000 curr count: 167819
2021/12/15 23:14:45 Waiting for src bucket item count to get to: 200000 curr count: 183170
2021/12/15 23:14:50 Waiting for src bucket item count to get to: 200000 curr count: 183170
2021/12/15 23:14:55 Waiting for src bucket item count to get to: 200000 curr count: 199644
2021/12/15 23:15:00 Waiting for src bucket item count to get to: 200000 curr count: 199644
2021/12/15 23:15:05 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/15 23:15:05 Post SET ops. Expected item count: 200000 got 200000
2021/12/15 23:15:05 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 23:15:10 Waiting for app: TestDuplicateWithKVSwapReb to get un-deployed
2021/12/15 23:15:15 Waiting for app: TestDuplicateWithKVSwapReb to get un-deployed
2021/12/15 23:15:20 Waiting for app: TestDuplicateWithKVSwapReb to get un-deployed
2021/12/15 23:15:25 Waiting for app: TestDuplicateWithKVSwapReb to get un-deployed
2021/12/15 23:15:26 App: TestDuplicateWithKVSwapReb got un-deployed
2021/12/15 23:15:26 No eventing-con process running
2021/12/15 23:15:26 Function: TestDuplicateWithKVSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithKVSwapReb deleting in the background"
}
2021/12/15 23:15:27 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:15:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 23:15:37 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/15 23:15:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 23:15:39 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:15:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 23:15:49 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/15 23:15:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithKVSwapReb (234.59s)
=== RUN   TestEventingRebKVOpsOnUpdateBucketOpOneByOne
2021/12/15 23:15:59 Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/15 23:15:59 App: TestEventingRebKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/15 23:15:59 No eventing-con process running
2021/12/15 23:15:59 Function: TestEventingRebKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpOneByOne not found"
 }
}
2021/12/15 23:16:01 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:16:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 23:16:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 23:16:08 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:16:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 23:16:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:16:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 23:16:18 Deploying app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne. Item count src bucket: 0 dst bucket: 0
2021/12/15 23:16:19 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebKVOpsOnUpdateBucketOpOneByOne' in metakv",
  "warnings": null
 }
}
2021/12/15 23:16:24 Starting bucket ops to source bucket
2021/12/15 23:16:29 Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/15 23:16:34 Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/15 23:16:39 Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/15 23:16:39 App: TestEventingRebKVOpsOnUpdateBucketOpOneByOne got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:16:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:16:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 23:16:39 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/15 23:16:39 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/15 23:16:40 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/15 23:16:45 Rebalance progress: 16.66666666666667
2021/12/15 23:16:50 Rebalance progress: 16.66666666666667
2021/12/15 23:16:55 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:16:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:16:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 23:16:55 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/15 23:16:55 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/15 23:16:55 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/15 23:17:00 Rebalance progress: 16.66666666666667
2021/12/15 23:17:05 Rebalance progress: 16.66666666666667
2021/12/15 23:17:10 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:17:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:17:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 23:17:10 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/15 23:17:10 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/15 23:17:15 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/15 23:17:20 Rebalance progress: 12.5
2021/12/15 23:17:25 Rebalance progress: 12.5
2021/12/15 23:17:30 Rebalance progress: 12.5
2021/12/15 23:17:35 Rebalance progress: 63.60000000000001
2021/12/15 23:17:40 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:17:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:17:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 23:17:40 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/15 23:17:46 Rebalance progress: 12.5
2021/12/15 23:17:51 Rebalance progress: 12.5
2021/12/15 23:17:56 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:17:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:17:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 23:17:56 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/15 23:18:01 Rebalance progress: 12.5
2021/12/15 23:18:06 Rebalance progress: 12.5
2021/12/15 23:18:11 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:18:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:18:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 23:18:11 clusterSetup.go:253: Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/15 23:18:16 Rebalance progress: 12.5
2021/12/15 23:18:21 Rebalance progress: 12.5
2021/12/15 23:18:26 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:18:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:18:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 23:18:26 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 23:18:31 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/15 23:18:36 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/15 23:18:41 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/15 23:18:46 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/15 23:18:46 clusterSetup.go:405: App: TestEventingRebKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/15 23:18:46 eventingSetup.go:46: No eventing-con process running
2021/12/15 23:18:46 eventingSetup.go:418: Function: TestEventingRebKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpOneByOne deleting in the background"
}
2021/12/15 23:18:48 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:18:53 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 23:18:58 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12187
2021/12/15 23:18:58 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 23:19:00 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:19:05 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 23:19:05 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebKVOpsOnUpdateBucketOpOneByOne (195.44s)
=== RUN   TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce
2021/12/15 23:19:15 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get un-deployed
2021/12/15 23:19:15 clusterSetup.go:405: App: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce got un-deployed
2021/12/15 23:19:15 eventingSetup.go:46: No eventing-con process running
2021/12/15 23:19:15 eventingSetup.go:418: Function: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce not found"
 }
}
2021/12/15 23:19:16 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:19:21 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 23:19:21 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 23:19:23 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:19:28 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 23:19:28 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:19:33 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 23:19:33 eventingSetup.go:125: Deploying app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce. Item count src bucket: 0 dst bucket: 0
2021/12/15 23:19:34 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce' in metakv",
  "warnings": null
 }
}
2021/12/15 23:19:39 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/15 23:19:44 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get deployed
2021/12/15 23:19:49 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get deployed
2021/12/15 23:19:54 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get deployed
2021/12/15 23:19:54 clusterSetup.go:354: App: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:19:54 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:19:54 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 23:19:54 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/15 23:19:55 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/15 23:19:55 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/15 23:19:55 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/15 23:20:00 Rebalance progress: 12.5
2021/12/15 23:20:05 Rebalance progress: 12.5
2021/12/15 23:20:10 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:20:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:20:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 23:20:10 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002 127.0.0.1:19003] from the cluster
2021/12/15 23:20:15 Rebalance progress: 12.5
2021/12/15 23:20:20 Rebalance progress: 12.5
2021/12/15 23:20:25 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:20:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:20:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 23:20:25 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 23:20:30 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get un-deployed
2021/12/15 23:20:35 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get un-deployed
2021/12/15 23:20:40 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get un-deployed
2021/12/15 23:20:46 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get un-deployed
2021/12/15 23:20:46 clusterSetup.go:405: App: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce got un-deployed
2021/12/15 23:20:46 eventingSetup.go:46: No eventing-con process running
2021/12/15 23:20:46 eventingSetup.go:418: Function: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce deleting in the background"
}
2021/12/15 23:20:47 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:20:52 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 23:20:57 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4593
2021/12/15 23:20:57 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 23:20:59 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:21:04 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 23:21:09 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4593
2021/12/15 23:21:09 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce (124.60s)
=== RUN   TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings
2021/12/15 23:21:19 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/15 23:21:19 clusterSetup.go:405: App: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings got un-deployed
2021/12/15 23:21:19 eventingSetup.go:46: No eventing-con process running
2021/12/15 23:21:19 eventingSetup.go:418: Function: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings not found"
 }
}
2021/12/15 23:21:21 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:21:26 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 23:21:26 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 23:21:28 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:21:33 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 23:21:33 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:21:38 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 23:21:38 eventingSetup.go:125: Deploying app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/15 23:21:39 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/15 23:21:44 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/15 23:21:49 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get deployed
2021/12/15 23:21:54 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get deployed
2021/12/15 23:21:59 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get deployed
2021/12/15 23:21:59 clusterSetup.go:354: App: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:21:59 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:21:59 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 23:21:59 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/15 23:21:59 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/15 23:22:04 Rebalance progress: 12.5
2021/12/15 23:22:09 Rebalance progress: 12.5
2021/12/15 23:22:14 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:22:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:22:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 23:22:14 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/15 23:22:15 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/15 23:22:20 Rebalance progress: 12.5
2021/12/15 23:22:25 Rebalance progress: 12.5
2021/12/15 23:22:30 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:22:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:22:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 23:22:30 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/15 23:22:30 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/15 23:22:35 Rebalance progress: 12.5
2021/12/15 23:22:40 Rebalance progress: 12.5
2021/12/15 23:22:45 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:22:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:22:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 23:22:45 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/15 23:22:50 Rebalance progress: 12.5
2021/12/15 23:22:55 Rebalance progress: 12.5
2021/12/15 23:23:00 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:23:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:23:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 23:23:00 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/15 23:23:05 Rebalance progress: 12.5
2021/12/15 23:23:10 Rebalance progress: 12.5
2021/12/15 23:23:15 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:23:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:23:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 23:23:15 clusterSetup.go:253: Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/15 23:23:20 Rebalance progress: 12.5
2021/12/15 23:23:25 Rebalance progress: 12.5
2021/12/15 23:23:30 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:23:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:23:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 23:23:31 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 23:23:36 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/15 23:23:41 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/15 23:23:46 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/15 23:23:51 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/15 23:23:51 clusterSetup.go:405: App: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings got un-deployed
2021/12/15 23:23:51 eventingSetup.go:46: No eventing-con process running
2021/12/15 23:23:51 eventingSetup.go:418: Function: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings deleting in the background"
}
2021/12/15 23:23:52 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:23:57 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 23:24:02 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 10648
2021/12/15 23:24:07 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 10648
2021/12/15 23:24:07 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 23:24:09 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:24:14 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 23:24:19 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 10642
2021/12/15 23:24:19 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings (190.06s)
=== RUN   TestEventingSwapRebOnUpdateBucketOp
2021/12/15 23:24:29 clusterSetup.go:397: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get un-deployed
2021/12/15 23:24:29 clusterSetup.go:405: App: TestEventingSwapRebOnUpdateBucketOp got un-deployed
2021/12/15 23:24:29 eventingSetup.go:46: No eventing-con process running
2021/12/15 23:24:29 eventingSetup.go:418: Function: TestEventingSwapRebOnUpdateBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingSwapRebOnUpdateBucketOp not found"
 }
}
2021/12/15 23:24:31 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:24:36 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 23:24:36 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 23:24:38 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:24:43 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 23:24:43 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:24:43 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 23:24:43 eventingSetup.go:125: Deploying app: TestEventingSwapRebOnUpdateBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/15 23:24:44 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingSwapRebOnUpdateBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/15 23:24:49 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/15 23:24:54 clusterSetup.go:346: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get deployed
2021/12/15 23:24:59 clusterSetup.go:346: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get deployed
2021/12/15 23:25:04 clusterSetup.go:346: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get deployed
2021/12/15 23:25:04 clusterSetup.go:354: App: TestEventingSwapRebOnUpdateBucketOp got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:25:04 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:25:04 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 23:25:04 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/15 23:25:04 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/15 23:25:09 Rebalance progress: 12.5
2021/12/15 23:25:14 Rebalance progress: 12.5
2021/12/15 23:25:19 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:25:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:25:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 23:25:19 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/15 23:25:20 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/15 23:25:25 Rebalance progress: 12.5
2021/12/15 23:25:30 Rebalance progress: 12.5
2021/12/15 23:25:35 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:25:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:25:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 23:25:35 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/15 23:25:40 Rebalance progress: 12.5
2021/12/15 23:25:45 Rebalance progress: 12.5
2021/12/15 23:25:50 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:25:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:25:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 23:25:50 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 23:25:55 clusterSetup.go:397: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get un-deployed
2021/12/15 23:26:00 clusterSetup.go:397: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get un-deployed
2021/12/15 23:26:05 clusterSetup.go:397: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get un-deployed
2021/12/15 23:26:10 clusterSetup.go:397: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get un-deployed
2021/12/15 23:26:10 clusterSetup.go:405: App: TestEventingSwapRebOnUpdateBucketOp got un-deployed
2021/12/15 23:26:10 eventingSetup.go:46: No eventing-con process running
2021/12/15 23:26:10 eventingSetup.go:418: Function: TestEventingSwapRebOnUpdateBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingSwapRebOnUpdateBucketOp deleting in the background"
}
2021/12/15 23:26:12 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:26:17 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 23:26:22 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6088
2021/12/15 23:26:27 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6088
2021/12/15 23:26:27 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 23:26:29 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:26:34 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 23:26:39 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6088
2021/12/15 23:26:39 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingSwapRebOnUpdateBucketOp (139.41s)
=== RUN   TestMetaRollbackWithEventingReb
2021/12/15 23:26:44 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/15 23:26:44 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/15 23:26:49 Rebalance progress: 100
2021/12/15 23:26:54 Waiting for app: TestMetaRollbackWithEventingReb to get un-deployed
2021/12/15 23:26:54 App: TestMetaRollbackWithEventingReb got un-deployed
2021/12/15 23:26:54 No eventing-con process running
2021/12/15 23:26:54 Function: TestMetaRollbackWithEventingReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMetaRollbackWithEventingReb not found"
 }
}
2021/12/15 23:26:56 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:27:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 23:27:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 23:27:03 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:27:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 23:27:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:27:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 23:27:08 Deploying app: TestMetaRollbackWithEventingReb. Item count src bucket: 0 dst bucket: 0
2021/12/15 23:27:09 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMetaRollbackWithEventingReb' in metakv",
  "warnings": null
 }
}
2021/12/15 23:27:14 Starting bucket ops to source bucket
2021/12/15 23:27:19 Waiting for app: TestMetaRollbackWithEventingReb to get deployed
2021/12/15 23:27:24 Waiting for app: TestMetaRollbackWithEventingReb to get deployed
2021/12/15 23:27:29 Waiting for app: TestMetaRollbackWithEventingReb to get deployed
2021/12/15 23:27:29 App: TestMetaRollbackWithEventingReb got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:27:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:27:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 23:27:29 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/15 23:27:29 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/15 23:27:34 Rebalance progress: 12.5
2021/12/15 23:27:39 Rebalance progress: 12.5
2021/12/15 23:27:44 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:27:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:27:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 23:27:44 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002] from the cluster
2021/12/15 23:27:44 bucketOps.go:134: Metakv lookup failed, err: Get "/_metakv/eventing/tempApps/TestMetaRollbackWithEventingReb/0": Unable to initialize cbauth's revrpc: cbauth environment variable CBAUTH_REVRPC_URL is not set
2021/12/15 23:27:49 Rebalance progress: 12.5
2021/12/15 23:27:54 Rebalance progress: 12.5
2021/12/15 23:27:59 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:27:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:27:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 23:27:59 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 23:28:04 clusterSetup.go:397: Waiting for app: TestMetaRollbackWithEventingReb to get un-deployed
2021/12/15 23:28:09 clusterSetup.go:397: Waiting for app: TestMetaRollbackWithEventingReb to get un-deployed
2021/12/15 23:28:14 clusterSetup.go:397: Waiting for app: TestMetaRollbackWithEventingReb to get un-deployed
2021/12/15 23:28:19 clusterSetup.go:397: Waiting for app: TestMetaRollbackWithEventingReb to get un-deployed
2021/12/15 23:28:19 clusterSetup.go:405: App: TestMetaRollbackWithEventingReb got un-deployed
2021/12/15 23:28:19 eventingSetup.go:46: No eventing-con process running
2021/12/15 23:28:19 eventingSetup.go:418: Function: TestMetaRollbackWithEventingReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMetaRollbackWithEventingReb deleting in the background"
}
2021/12/15 23:28:21 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:28:26 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 23:28:31 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4539
2021/12/15 23:28:36 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4539
2021/12/15 23:28:36 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 23:28:38 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:28:43 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 23:28:48 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4539
2021/12/15 23:28:48 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestMetaRollbackWithEventingReb (129.34s)
=== RUN   TestMetaPartialRollbackWithEventingReb
2021/12/15 23:28:53 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/15 23:28:53 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/15 23:28:58 Rebalance progress: 100
2021/12/15 23:29:03 Waiting for app: TestMetaPartialRollbackWithEventingReb to get un-deployed
2021/12/15 23:29:03 App: TestMetaPartialRollbackWithEventingReb got un-deployed
2021/12/15 23:29:03 No eventing-con process running
2021/12/15 23:29:03 Function: TestMetaPartialRollbackWithEventingReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMetaPartialRollbackWithEventingReb not found"
 }
}
2021/12/15 23:29:05 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:29:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 23:29:10 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 23:29:12 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:29:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 23:29:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:29:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 23:29:17 Deploying app: TestMetaPartialRollbackWithEventingReb. Item count src bucket: 0 dst bucket: 0
2021/12/15 23:29:18 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMetaPartialRollbackWithEventingReb' in metakv",
  "warnings": null
 }
}
2021/12/15 23:29:23 Starting bucket ops to source bucket
2021/12/15 23:29:28 Waiting for app: TestMetaPartialRollbackWithEventingReb to get deployed
2021/12/15 23:29:33 Waiting for app: TestMetaPartialRollbackWithEventingReb to get deployed
2021/12/15 23:29:38 Waiting for app: TestMetaPartialRollbackWithEventingReb to get deployed
2021/12/15 23:29:38 App: TestMetaPartialRollbackWithEventingReb got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:29:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:29:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 23:29:38 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/15 23:29:38 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/15 23:29:43 Rebalance progress: 12.5
2021/12/15 23:29:48 Rebalance progress: 12.5
2021/12/15 23:29:53 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:29:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:29:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 23:29:54 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002] from the cluster
2021/12/15 23:29:54 bucketOps.go:65: Metakv lookup failed, err: Get "/_metakv/eventing/tempApps/TestMetaPartialRollbackWithEventingReb/0": Unable to initialize cbauth's revrpc: cbauth environment variable CBAUTH_REVRPC_URL is not set
2021/12/15 23:29:59 Rebalance progress: 12.5
2021/12/15 23:30:04 Rebalance progress: 12.5
2021/12/15 23:30:09 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:30:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:30:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 23:30:09 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 23:30:09 bucketOps.go:65: Metakv lookup failed, err: Get "/_metakv/eventing/tempApps/TestMetaPartialRollbackWithEventingReb/0": Unable to initialize cbauth's revrpc: cbauth environment variable CBAUTH_REVRPC_URL is not set
2021/12/15 23:30:14 clusterSetup.go:397: Waiting for app: TestMetaPartialRollbackWithEventingReb to get un-deployed
2021/12/15 23:30:19 clusterSetup.go:397: Waiting for app: TestMetaPartialRollbackWithEventingReb to get un-deployed
2021/12/15 23:30:24 clusterSetup.go:397: Waiting for app: TestMetaPartialRollbackWithEventingReb to get un-deployed
2021/12/15 23:30:29 clusterSetup.go:397: Waiting for app: TestMetaPartialRollbackWithEventingReb to get un-deployed
2021/12/15 23:30:29 clusterSetup.go:405: App: TestMetaPartialRollbackWithEventingReb got un-deployed
2021/12/15 23:30:29 eventingSetup.go:46: No eventing-con process running
2021/12/15 23:30:29 eventingSetup.go:418: Function: TestMetaPartialRollbackWithEventingReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMetaPartialRollbackWithEventingReb deleting in the background"
}
2021/12/15 23:30:31 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:30:36 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 23:30:41 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 24129
2021/12/15 23:30:46 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 24129
2021/12/15 23:30:46 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 23:30:47 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:30:52 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 23:30:57 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 37709
2021/12/15 23:30:57 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestMetaPartialRollbackWithEventingReb (129.41s)
=== RUN   TestEventingRebMultipleHandlersOneByOne
2021/12/15 23:31:07 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get un-deployed
2021/12/15 23:31:07 clusterSetup.go:405: App: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op got un-deployed
2021/12/15 23:31:07 eventingSetup.go:46: No eventing-con process running
2021/12/15 23:31:07 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op not found"
 }
}
2021/12/15 23:31:09 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:31:14 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 23:31:14 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 23:31:16 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:31:21 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 23:31:21 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:31:26 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 23:31:26 eventingSetup.go:125: Deploying app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op. Item count src bucket: 0 dst bucket: 0
2021/12/15 23:31:27 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op' in metakv",
  "warnings": null
 }
}
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:31:27 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 23:31:27 eventingSetup.go:125: Deploying app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/15 23:31:28 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebMultipleHandlersOneByOnebucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/15 23:31:33 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/15 23:31:38 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get deployed
2021/12/15 23:31:43 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get deployed
2021/12/15 23:31:48 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get deployed
2021/12/15 23:31:48 clusterSetup.go:354: App: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op got deployed
2021/12/15 23:31:53 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get deployed
2021/12/15 23:31:58 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get deployed
2021/12/15 23:32:03 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get deployed
2021/12/15 23:32:03 clusterSetup.go:354: App: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:32:03 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:32:03 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 23:32:03 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/15 23:32:04 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/15 23:32:09 Rebalance progress: 12.5
2021/12/15 23:32:14 Rebalance progress: 12.5
2021/12/15 23:32:19 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:32:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:32:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 23:32:19 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/15 23:32:19 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/15 23:32:24 Rebalance progress: 12.5
2021/12/15 23:32:29 Rebalance progress: 12.5
2021/12/15 23:32:34 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:32:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:32:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 23:32:34 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/15 23:32:35 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/15 23:32:40 Rebalance progress: 12.5
2021/12/15 23:32:45 Rebalance progress: 12.5
2021/12/15 23:32:50 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:32:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:32:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 23:32:50 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/15 23:32:55 Rebalance progress: 12.5
2021/12/15 23:33:00 Rebalance progress: 12.5
2021/12/15 23:33:05 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:33:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:33:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 23:33:05 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/15 23:33:10 Rebalance progress: 12.5
2021/12/15 23:33:15 Rebalance progress: 12.5
2021/12/15 23:33:20 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:33:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:33:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 23:33:20 clusterSetup.go:253: Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/15 23:33:25 Rebalance progress: 12.5
2021/12/15 23:33:30 Rebalance progress: 12.5
2021/12/15 23:33:35 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:33:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:33:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 23:33:35 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 23:33:40 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get un-deployed
2021/12/15 23:33:45 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get un-deployed
2021/12/15 23:33:50 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get un-deployed
2021/12/15 23:33:55 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get un-deployed
2021/12/15 23:33:55 clusterSetup.go:405: App: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op got un-deployed
2021/12/15 23:33:55 eventingSetup.go:46: No eventing-con process running
2021/12/15 23:33:56 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op deleting in the background"
}
2021/12/15 23:33:59 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:34:04 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 23:34:09 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12188
2021/12/15 23:34:09 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 23:34:11 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:34:16 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 23:34:21 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 9022
2021/12/15 23:34:26 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 9022
2021/12/15 23:34:26 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 23:34:26 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 23:34:31 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get un-deployed
2021/12/15 23:34:36 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get un-deployed
2021/12/15 23:34:41 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get un-deployed
2021/12/15 23:34:46 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get un-deployed
2021/12/15 23:34:46 clusterSetup.go:405: App: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update got un-deployed
2021/12/15 23:34:46 eventingSetup.go:46: No eventing-con process running
2021/12/15 23:34:46 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update deleting in the background"
}
2021/12/15 23:34:48 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:34:53 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 23:34:53 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 23:34:54 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:34:59 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 23:34:59 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebMultipleHandlersOneByOne (242.11s)
=== RUN   TestEventingRebMultipleHandlersAllAtOnce
2021/12/15 23:35:09 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get un-deployed
2021/12/15 23:35:10 clusterSetup.go:405: App: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op got un-deployed
2021/12/15 23:35:10 eventingSetup.go:46: No eventing-con process running
2021/12/15 23:35:10 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op not found"
 }
}
2021/12/15 23:35:11 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:35:16 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 23:35:16 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 23:35:18 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:35:23 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 23:35:23 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:35:28 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 23:35:28 eventingSetup.go:125: Deploying app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op. Item count src bucket: 0 dst bucket: 0
2021/12/15 23:35:29 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op' in metakv",
  "warnings": null
 }
}
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:35:29 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 23:35:29 eventingSetup.go:125: Deploying app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/15 23:35:30 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/15 23:35:35 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/15 23:35:40 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get deployed
2021/12/15 23:35:45 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get deployed
2021/12/15 23:35:50 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get deployed
2021/12/15 23:35:50 clusterSetup.go:354: App: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op got deployed
2021/12/15 23:35:55 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get deployed
2021/12/15 23:36:00 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get deployed
2021/12/15 23:36:05 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get deployed
2021/12/15 23:36:05 clusterSetup.go:354: App: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:36:05 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:36:05 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 23:36:05 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/15 23:36:06 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/15 23:36:06 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/15 23:36:07 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/15 23:36:12 Rebalance progress: 12.5
2021/12/15 23:36:17 Rebalance progress: 12.5
2021/12/15 23:36:22 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:36:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:36:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 23:36:22 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002 127.0.0.1:19003] from the cluster
2021/12/15 23:36:27 Rebalance progress: 12.5
2021/12/15 23:36:32 Rebalance progress: 12.5
2021/12/15 23:36:37 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:36:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:36:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 23:36:37 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 23:36:42 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get un-deployed
2021/12/15 23:36:47 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get un-deployed
2021/12/15 23:36:52 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get un-deployed
2021/12/15 23:36:57 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get un-deployed
2021/12/15 23:36:57 clusterSetup.go:405: App: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op got un-deployed
2021/12/15 23:36:57 eventingSetup.go:46: No eventing-con process running
2021/12/15 23:36:57 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op deleting in the background"
}
2021/12/15 23:37:01 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:37:06 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 23:37:11 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 733
2021/12/15 23:37:16 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 733
2021/12/15 23:37:16 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 23:37:17 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:37:22 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 23:37:27 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6145
2021/12/15 23:37:27 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 23:37:27 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 23:37:32 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get un-deployed
2021/12/15 23:37:38 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get un-deployed
2021/12/15 23:37:43 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get un-deployed
2021/12/15 23:37:48 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get un-deployed
2021/12/15 23:37:48 clusterSetup.go:405: App: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update got un-deployed
2021/12/15 23:37:48 eventingSetup.go:46: No eventing-con process running
2021/12/15 23:37:48 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update deleting in the background"
}
2021/12/15 23:37:49 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:37:54 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 23:37:54 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 23:37:56 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:38:01 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 23:38:01 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebMultipleHandlersAllAtOnce (181.70s)
=== RUN   TestEventingSwapRebMultipleHandlers
2021/12/15 23:38:11 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get un-deployed
2021/12/15 23:38:11 clusterSetup.go:405: App: TestEventingSwapRebMultipleHandlersbucket_op_on_update got un-deployed
2021/12/15 23:38:11 eventingSetup.go:46: No eventing-con process running
2021/12/15 23:38:11 eventingSetup.go:418: Function: TestEventingSwapRebMultipleHandlersbucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingSwapRebMultipleHandlersbucket_op_on_update not found"
 }
}
2021/12/15 23:38:13 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:38:18 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 23:38:18 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 23:38:20 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:38:25 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 23:38:25 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 23:38:30 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get un-deployed
2021/12/15 23:38:30 clusterSetup.go:405: App: TestEventingSwapRebMultipleHandlerssys_test_bucket_op got un-deployed
2021/12/15 23:38:30 eventingSetup.go:46: No eventing-con process running
2021/12/15 23:38:30 eventingSetup.go:418: Function: TestEventingSwapRebMultipleHandlerssys_test_bucket_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingSwapRebMultipleHandlerssys_test_bucket_op not found"
 }
}
2021/12/15 23:38:32 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:38:37 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 23:38:37 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 23:38:38 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:38:43 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 23:38:43 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:38:43 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 23:38:43 eventingSetup.go:125: Deploying app: TestEventingSwapRebMultipleHandlersbucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/15 23:38:44 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingSwapRebMultipleHandlersbucket_op_on_update' in metakv",
  "warnings": null
 }
}
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:38:44 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 23:38:44 eventingSetup.go:125: Deploying app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op. Item count src bucket: 0 dst bucket: 0
2021/12/15 23:38:46 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingSwapRebMultipleHandlerssys_test_bucket_op' in metakv",
  "warnings": null
 }
}
2021/12/15 23:38:51 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/15 23:38:56 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get deployed
2021/12/15 23:39:01 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get deployed
2021/12/15 23:39:06 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get deployed
2021/12/15 23:39:06 clusterSetup.go:354: App: TestEventingSwapRebMultipleHandlersbucket_op_on_update got deployed
2021/12/15 23:39:11 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get deployed
2021/12/15 23:39:16 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get deployed
2021/12/15 23:39:21 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get deployed
2021/12/15 23:39:21 clusterSetup.go:354: App: TestEventingSwapRebMultipleHandlerssys_test_bucket_op got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:39:21 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:39:21 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 23:39:21 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/15 23:39:21 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/15 23:39:26 Rebalance progress: 12.5
2021/12/15 23:39:31 Rebalance progress: 12.5
2021/12/15 23:39:36 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:39:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:39:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 23:39:36 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/15 23:39:36 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/15 23:39:41 Rebalance progress: 12.5
2021/12/15 23:39:46 Rebalance progress: 12.5
2021/12/15 23:39:51 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:39:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:39:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 23:39:52 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/15 23:39:57 Rebalance progress: 12.5
2021/12/15 23:40:02 Rebalance progress: 12.5
2021/12/15 23:40:07 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:40:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:40:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 23:40:07 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 23:40:12 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get un-deployed
2021/12/15 23:40:17 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get un-deployed
2021/12/15 23:40:22 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get un-deployed
2021/12/15 23:40:27 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get un-deployed
2021/12/15 23:40:27 clusterSetup.go:405: App: TestEventingSwapRebMultipleHandlersbucket_op_on_update got un-deployed
2021/12/15 23:40:27 eventingSetup.go:46: No eventing-con process running
2021/12/15 23:40:27 eventingSetup.go:418: Function: TestEventingSwapRebMultipleHandlersbucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingSwapRebMultipleHandlersbucket_op_on_update deleting in the background"
}
2021/12/15 23:40:31 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:40:36 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 23:40:41 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 526
2021/12/15 23:40:46 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 526
2021/12/15 23:40:46 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 23:40:47 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:40:52 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 23:40:57 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7594
2021/12/15 23:40:57 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 23:40:58 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 23:41:03 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get un-deployed
2021/12/15 23:41:08 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get un-deployed
2021/12/15 23:41:13 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get un-deployed
2021/12/15 23:41:18 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get un-deployed
2021/12/15 23:41:18 clusterSetup.go:405: App: TestEventingSwapRebMultipleHandlerssys_test_bucket_op got un-deployed
2021/12/15 23:41:18 eventingSetup.go:46: No eventing-con process running
2021/12/15 23:41:18 eventingSetup.go:418: Function: TestEventingSwapRebMultipleHandlerssys_test_bucket_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingSwapRebMultipleHandlerssys_test_bucket_op deleting in the background"
}
2021/12/15 23:41:19 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:41:24 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 23:41:24 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 23:41:26 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:41:31 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 23:41:31 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingSwapRebMultipleHandlers (210.02s)
=== RUN   TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne
2021/12/15 23:41:41 clusterSetup.go:397: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/15 23:41:41 clusterSetup.go:405: App: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/15 23:41:41 eventingSetup.go:46: No eventing-con process running
2021/12/15 23:41:41 eventingSetup.go:418: Function: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne not found"
 }
}
2021/12/15 23:41:43 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:41:48 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 23:41:48 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 23:41:50 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:41:55 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 23:41:55 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:42:00 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 23:42:00 eventingSetup.go:125: Deploying app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne. Item count src bucket: 0 dst bucket: 0
2021/12/15 23:42:01 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne' in metakv",
  "warnings": null
 }
}
2021/12/15 23:42:06 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/15 23:42:11 clusterSetup.go:346: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/15 23:42:16 clusterSetup.go:346: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/15 23:42:21 clusterSetup.go:346: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/15 23:42:21 clusterSetup.go:354: App: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:42:21 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:42:21 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 23:42:21 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/15 23:42:21 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/15 23:42:41 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:42:42 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:42:42 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:43:12 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:43:12 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 23:43:12 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/15 23:43:17 Rebalance progress: 12.5
2021/12/15 23:43:22 Rebalance progress: 12.5
2021/12/15 23:43:27 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:43:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:43:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 23:43:27 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/15 23:43:28 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/15 23:43:33 Rebalance progress: 12.5
2021/12/15 23:43:38 Rebalance progress: 12.5
2021/12/15 23:43:43 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:43:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:43:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 23:43:43 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/15 23:43:48 Rebalance progress: 12.5
2021/12/15 23:43:53 Rebalance progress: 12.5
2021/12/15 23:43:58 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:43:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:43:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 23:43:58 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 23:44:03 clusterSetup.go:397: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/15 23:44:08 clusterSetup.go:397: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/15 23:44:13 clusterSetup.go:397: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/15 23:44:18 clusterSetup.go:397: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/15 23:44:18 clusterSetup.go:405: App: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/15 23:44:18 eventingSetup.go:46: No eventing-con process running
2021/12/15 23:44:18 eventingSetup.go:418: Function: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne deleting in the background"
}
2021/12/15 23:44:20 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:44:25 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 23:44:25 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 23:44:27 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:44:32 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 23:44:37 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 11185
2021/12/15 23:44:37 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne (185.35s)
=== RUN   TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne
2021/12/15 23:44:47 clusterSetup.go:397: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/15 23:44:47 clusterSetup.go:405: App: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/15 23:44:47 eventingSetup.go:46: No eventing-con process running
2021/12/15 23:44:47 eventingSetup.go:418: Function: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne not found"
 }
}
2021/12/15 23:44:48 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:44:53 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 23:44:53 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 23:44:55 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:45:00 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 23:45:00 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:45:05 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 23:45:05 eventingSetup.go:125: Deploying app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne. Item count src bucket: 0 dst bucket: 0
2021/12/15 23:45:06 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne' in metakv",
  "warnings": null
 }
}
2021/12/15 23:45:11 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/15 23:45:16 clusterSetup.go:346: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/15 23:45:21 clusterSetup.go:346: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/15 23:45:26 clusterSetup.go:346: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/15 23:45:26 clusterSetup.go:354: App: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:45:26 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:45:26 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 23:45:26 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/15 23:45:27 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/15 23:45:47 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:45:47 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:45:47 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:45:57 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:45:57 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 23:45:57 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/15 23:45:58 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/15 23:46:18 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:46:18 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:46:18 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:46:28 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:46:28 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 23:46:28 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/15 23:46:28 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/15 23:46:49 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:46:49 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:46:49 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:46:59 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:46:59 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 23:46:59 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/15 23:47:00 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/15 23:47:20 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:47:20 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:47:20 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:47:30 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:47:30 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 23:47:30 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/15 23:47:30 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/15 23:47:50 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:47:51 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:47:51 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:48:01 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:48:01 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 23:48:01 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/15 23:48:06 Rebalance progress: 12.5
2021/12/15 23:48:11 Rebalance progress: 12.5
2021/12/15 23:48:16 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:48:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:48:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 23:48:16 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/15 23:48:16 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/15 23:48:21 Rebalance progress: 12.5
2021/12/15 23:48:26 Rebalance progress: 12.5
2021/12/15 23:48:31 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:48:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:48:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 23:48:32 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/15 23:48:37 Rebalance progress: 12.5
2021/12/15 23:48:42 Rebalance progress: 12.5
2021/12/15 23:48:47 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:48:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:48:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 23:48:47 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 23:48:52 clusterSetup.go:397: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/15 23:48:57 clusterSetup.go:397: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/15 23:49:02 clusterSetup.go:397: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/15 23:49:07 clusterSetup.go:397: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/15 23:49:07 clusterSetup.go:405: App: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/15 23:49:07 eventingSetup.go:46: No eventing-con process running
2021/12/15 23:49:07 eventingSetup.go:418: Function: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne deleting in the background"
}
2021/12/15 23:49:09 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:49:14 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 23:49:19 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 21518
2021/12/15 23:49:19 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 23:49:20 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:49:25 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 23:49:30 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 8968
2021/12/15 23:49:35 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 8968
2021/12/15 23:49:35 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne (298.79s)
=== RUN   TestEventingFailoverOnUpdateBucketOp
2021/12/15 23:49:45 clusterSetup.go:397: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get un-deployed
2021/12/15 23:49:45 clusterSetup.go:405: App: TestEventingFailoverOnUpdateBucketOp got un-deployed
2021/12/15 23:49:45 eventingSetup.go:46: No eventing-con process running
2021/12/15 23:49:45 eventingSetup.go:418: Function: TestEventingFailoverOnUpdateBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingFailoverOnUpdateBucketOp not found"
 }
}
2021/12/15 23:49:47 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:49:52 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 23:49:52 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 23:49:54 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:49:59 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 23:49:59 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:50:04 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 23:50:04 eventingSetup.go:125: Deploying app: TestEventingFailoverOnUpdateBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/15 23:50:05 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingFailoverOnUpdateBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/15 23:50:10 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/15 23:50:15 clusterSetup.go:346: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get deployed
2021/12/15 23:50:20 clusterSetup.go:346: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get deployed
2021/12/15 23:50:25 clusterSetup.go:346: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get deployed
2021/12/15 23:50:25 clusterSetup.go:354: App: TestEventingFailoverOnUpdateBucketOp got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:50:25 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:50:25 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 23:50:25 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/15 23:50:25 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/15 23:50:31 Rebalance progress: 12.5
2021/12/15 23:50:36 Rebalance progress: 12.5
2021/12/15 23:50:41 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:50:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:50:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 23:50:41 clusterSetup.go:229: Failing over: [127.0.0.1:19001]
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:50:51 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:50:51 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 23:50:51 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/15 23:50:56 Rebalance progress: 12.5
2021/12/15 23:51:01 Rebalance progress: 12.5
2021/12/15 23:51:06 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:51:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:51:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 23:51:06 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/15 23:51:06 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/15 23:51:11 Rebalance progress: 12.5
2021/12/15 23:51:16 Rebalance progress: 12.5
2021/12/15 23:51:21 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:51:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:51:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 23:51:21 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/15 23:51:26 Rebalance progress: 12.5
2021/12/15 23:51:31 Rebalance progress: 12.5
2021/12/15 23:51:36 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:51:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:51:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 23:51:36 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 23:51:41 clusterSetup.go:397: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get un-deployed
2021/12/15 23:51:46 clusterSetup.go:397: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get un-deployed
2021/12/15 23:51:51 clusterSetup.go:397: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get un-deployed
2021/12/15 23:51:56 clusterSetup.go:397: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get un-deployed
2021/12/15 23:51:56 clusterSetup.go:405: App: TestEventingFailoverOnUpdateBucketOp got un-deployed
2021/12/15 23:51:56 eventingSetup.go:46: No eventing-con process running
2021/12/15 23:51:56 eventingSetup.go:418: Function: TestEventingFailoverOnUpdateBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingFailoverOnUpdateBucketOp deleting in the background"
}
2021/12/15 23:51:58 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:52:03 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 23:52:08 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 8583
2021/12/15 23:52:08 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 23:52:10 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:52:15 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 23:52:15 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingFailoverOnUpdateBucketOp (159.43s)
=== RUN   TestEventingKVRebalanceOnUpdateBucketOp
2021/12/15 23:52:25 clusterSetup.go:397: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get un-deployed
2021/12/15 23:52:25 clusterSetup.go:405: App: TestEventingKVRebalanceOnUpdateBucketOp got un-deployed
2021/12/15 23:52:25 eventingSetup.go:46: No eventing-con process running
2021/12/15 23:52:25 eventingSetup.go:418: Function: TestEventingKVRebalanceOnUpdateBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingKVRebalanceOnUpdateBucketOp not found"
 }
}
2021/12/15 23:52:27 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:52:32 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 23:52:32 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 23:52:33 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:52:38 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 23:52:38 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:52:43 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 23:52:43 eventingSetup.go:125: Deploying app: TestEventingKVRebalanceOnUpdateBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/15 23:52:44 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingKVRebalanceOnUpdateBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/15 23:52:49 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/15 23:52:54 clusterSetup.go:346: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get deployed
2021/12/15 23:52:59 clusterSetup.go:346: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get deployed
2021/12/15 23:53:04 clusterSetup.go:346: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get deployed
2021/12/15 23:53:04 clusterSetup.go:354: App: TestEventingKVRebalanceOnUpdateBucketOp got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:53:04 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:53:04 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 23:53:05 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing,kv to the cluster
2021/12/15 23:53:05 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/15 23:53:10 Rebalance progress: 12.5
2021/12/15 23:53:15 Rebalance progress: 12.5
2021/12/15 23:53:20 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:53:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:53:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 23:53:20 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing,kv to the cluster
2021/12/15 23:53:20 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/15 23:53:26 Rebalance progress: 12.5
2021/12/15 23:53:31 Rebalance progress: 12.5
2021/12/15 23:53:36 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:53:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:53:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 23:53:36 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/15 23:53:41 Rebalance progress: 12.5
2021/12/15 23:53:46 Rebalance progress: 12.5
2021/12/15 23:53:51 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:53:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:53:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 23:53:51 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 23:53:56 clusterSetup.go:397: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get un-deployed
2021/12/15 23:54:01 clusterSetup.go:397: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get un-deployed
2021/12/15 23:54:06 clusterSetup.go:397: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get un-deployed
2021/12/15 23:54:11 clusterSetup.go:397: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get un-deployed
2021/12/15 23:54:11 clusterSetup.go:405: App: TestEventingKVRebalanceOnUpdateBucketOp got un-deployed
2021/12/15 23:54:11 eventingSetup.go:46: No eventing-con process running
2021/12/15 23:54:11 eventingSetup.go:418: Function: TestEventingKVRebalanceOnUpdateBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingKVRebalanceOnUpdateBucketOp deleting in the background"
}
2021/12/15 23:54:13 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:54:18 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 23:54:23 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6101
2021/12/15 23:54:28 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6101
2021/12/15 23:54:28 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 23:54:29 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:54:34 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 23:54:34 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingKVRebalanceOnUpdateBucketOp (139.59s)
=== RUN   TestEventingRebInWithOps
2021/12/15 23:54:44 clusterSetup.go:397: Waiting for app: TestEventingRebInWithOps to get un-deployed
2021/12/15 23:54:44 clusterSetup.go:405: App: TestEventingRebInWithOps got un-deployed
2021/12/15 23:54:44 eventingSetup.go:46: No eventing-con process running
2021/12/15 23:54:44 eventingSetup.go:418: Function: TestEventingRebInWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebInWithOps not found"
 }
}
2021/12/15 23:54:46 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:54:51 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 23:54:51 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 23:54:53 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:54:58 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 23:54:58 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:55:03 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 23:55:03 eventingSetup.go:125: Deploying app: TestEventingRebInWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/15 23:55:04 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebInWithOps' in metakv",
  "warnings": null
 }
}
2021/12/15 23:55:09 clusterSetup.go:346: Waiting for app: TestEventingRebInWithOps to get deployed
2021/12/15 23:55:14 clusterSetup.go:346: Waiting for app: TestEventingRebInWithOps to get deployed
2021/12/15 23:55:19 clusterSetup.go:346: Waiting for app: TestEventingRebInWithOps to get deployed
2021/12/15 23:55:24 clusterSetup.go:346: Waiting for app: TestEventingRebInWithOps to get deployed
2021/12/15 23:55:24 clusterSetup.go:354: App: TestEventingRebInWithOps got deployed
2021/12/15 23:55:24 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/15 23:55:24 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/15 23:55:24 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/15 23:55:29 Rebalance progress: 12.5
2021/12/15 23:55:34 Rebalance progress: 12.5
2021/12/15 23:55:39 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:55:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:55:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 23:55:39 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/15 23:55:45 Rebalance progress: 12.5
2021/12/15 23:55:50 Rebalance progress: 12.5
2021/12/15 23:55:55 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:55:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 23:55:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 23:55:55 Waiting for src bucket item count to get to: 200000 curr count: 37797
2021/12/15 23:56:00 Waiting for src bucket item count to get to: 200000 curr count: 37797
2021/12/15 23:56:05 Waiting for src bucket item count to get to: 200000 curr count: 53098
2021/12/15 23:56:10 Waiting for src bucket item count to get to: 200000 curr count: 53098
2021/12/15 23:56:15 Waiting for src bucket item count to get to: 200000 curr count: 69254
2021/12/15 23:56:20 Waiting for src bucket item count to get to: 200000 curr count: 69254
2021/12/15 23:56:25 Waiting for src bucket item count to get to: 200000 curr count: 84421
2021/12/15 23:56:30 Waiting for src bucket item count to get to: 200000 curr count: 84421
2021/12/15 23:56:35 Waiting for src bucket item count to get to: 200000 curr count: 100369
2021/12/15 23:56:40 Waiting for src bucket item count to get to: 200000 curr count: 100369
2021/12/15 23:56:45 Waiting for src bucket item count to get to: 200000 curr count: 116201
2021/12/15 23:56:50 Waiting for src bucket item count to get to: 200000 curr count: 116201
2021/12/15 23:56:55 Waiting for src bucket item count to get to: 200000 curr count: 132198
2021/12/15 23:57:00 Waiting for src bucket item count to get to: 200000 curr count: 132198
2021/12/15 23:57:05 Waiting for src bucket item count to get to: 200000 curr count: 148221
2021/12/15 23:57:10 Waiting for src bucket item count to get to: 200000 curr count: 148221
2021/12/15 23:57:15 Waiting for src bucket item count to get to: 200000 curr count: 164090
2021/12/15 23:57:20 Waiting for src bucket item count to get to: 200000 curr count: 164090
2021/12/15 23:57:25 Waiting for src bucket item count to get to: 200000 curr count: 179002
2021/12/15 23:57:30 Waiting for src bucket item count to get to: 200000 curr count: 179002
2021/12/15 23:57:35 Waiting for src bucket item count to get to: 200000 curr count: 194767
2021/12/15 23:57:40 Waiting for src bucket item count to get to: 200000 curr count: 194767
2021/12/15 23:57:45 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/15 23:57:45 Post SET ops. Expected item count: 200000 got 200000
2021/12/15 23:57:45 Starting bucket ops to source bucket
2021/12/16 00:01:35 Waiting for dst bucket item count to get to: 0 curr count: 18900
2021/12/16 00:01:40 Waiting for dst bucket item count to get to: 0 curr count: 417
2021/12/16 00:01:45 Waiting for dst bucket item count to get to: 0 curr count: 417
2021/12/16 00:01:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 00:01:45 Post DELETE ops. Expected item count: 0 got 0
2021/12/16 00:01:45 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 00:01:50 Waiting for app: TestEventingRebInWithOps to get un-deployed
2021/12/16 00:01:55 Waiting for app: TestEventingRebInWithOps to get un-deployed
2021/12/16 00:02:00 Waiting for app: TestEventingRebInWithOps to get un-deployed
2021/12/16 00:02:05 Waiting for app: TestEventingRebInWithOps to get un-deployed
2021/12/16 00:02:05 App: TestEventingRebInWithOps got un-deployed
2021/12/16 00:02:05 No eventing-con process running
2021/12/16 00:02:05 Function: TestEventingRebInWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebInWithOps deleting in the background"
}
2021/12/16 00:02:07 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 00:02:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 00:02:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 00:02:13 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 00:02:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 00:02:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebInWithOps (463.94s)
=== RUN   TestEventingRebOutWithOps
2021/12/16 00:02:23 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/16 00:02:24 Removing node(s): [] from the cluster
2021/12/16 00:02:29 Rebalance progress: 100
2021/12/16 00:02:34 Waiting for app: TestEventingRebOutWithOps to get un-deployed
2021/12/16 00:02:34 App: TestEventingRebOutWithOps got un-deployed
2021/12/16 00:02:34 No eventing-con process running
2021/12/16 00:02:34 Function: TestEventingRebOutWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebOutWithOps not found"
 }
}
2021/12/16 00:02:35 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 00:02:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 00:02:40 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 00:02:42 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 00:02:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 00:02:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 00:02:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 00:02:52 Deploying app: TestEventingRebOutWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/16 00:02:53 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebOutWithOps' in metakv",
  "warnings": null
 }
}
2021/12/16 00:02:58 Waiting for app: TestEventingRebOutWithOps to get deployed
2021/12/16 00:03:03 Waiting for app: TestEventingRebOutWithOps to get deployed
2021/12/16 00:03:08 Waiting for app: TestEventingRebOutWithOps to get deployed
2021/12/16 00:03:13 Waiting for app: TestEventingRebOutWithOps to get deployed
2021/12/16 00:03:13 App: TestEventingRebOutWithOps got deployed
2021/12/16 00:03:13 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/16 00:03:13 Starting bucket ops to source bucket
2021/12/16 00:03:18 Rebalance progress: 12.5
2021/12/16 00:03:23 Rebalance progress: 12.5
2021/12/16 00:03:28 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 00:03:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 00:03:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 00:03:28 Waiting for src bucket item count to get to: 200000 curr count: 8544
2021/12/16 00:03:33 Waiting for src bucket item count to get to: 200000 curr count: 24424
2021/12/16 00:03:38 Waiting for src bucket item count to get to: 200000 curr count: 24424
2021/12/16 00:03:43 Waiting for src bucket item count to get to: 200000 curr count: 40591
2021/12/16 00:03:48 Waiting for src bucket item count to get to: 200000 curr count: 40591
2021/12/16 00:03:53 Waiting for src bucket item count to get to: 200000 curr count: 56667
2021/12/16 00:03:58 Waiting for src bucket item count to get to: 200000 curr count: 56667
2021/12/16 00:04:04 Waiting for src bucket item count to get to: 200000 curr count: 56667
2021/12/16 00:04:09 Waiting for src bucket item count to get to: 200000 curr count: 72340
2021/12/16 00:04:14 Waiting for src bucket item count to get to: 200000 curr count: 72340
2021/12/16 00:04:19 Waiting for src bucket item count to get to: 200000 curr count: 87381
2021/12/16 00:04:24 Waiting for src bucket item count to get to: 200000 curr count: 87381
2021/12/16 00:04:29 Waiting for src bucket item count to get to: 200000 curr count: 103038
2021/12/16 00:04:34 Waiting for src bucket item count to get to: 200000 curr count: 103038
2021/12/16 00:04:39 Waiting for src bucket item count to get to: 200000 curr count: 119077
2021/12/16 00:04:44 Waiting for src bucket item count to get to: 200000 curr count: 119077
2021/12/16 00:04:49 Waiting for src bucket item count to get to: 200000 curr count: 135469
2021/12/16 00:04:54 Waiting for src bucket item count to get to: 200000 curr count: 135469
2021/12/16 00:04:59 Waiting for src bucket item count to get to: 200000 curr count: 151804
2021/12/16 00:05:04 Waiting for src bucket item count to get to: 200000 curr count: 151804
2021/12/16 00:05:09 Waiting for src bucket item count to get to: 200000 curr count: 167953
2021/12/16 00:05:14 Waiting for src bucket item count to get to: 200000 curr count: 183159
2021/12/16 00:05:19 Waiting for src bucket item count to get to: 200000 curr count: 183159
2021/12/16 00:05:24 Waiting for src bucket item count to get to: 200000 curr count: 199355
2021/12/16 00:05:29 Waiting for src bucket item count to get to: 200000 curr count: 199355
2021/12/16 00:05:34 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/16 00:05:34 Post SET ops. Expected item count: 200000 got 200000
2021/12/16 00:05:34 Starting bucket ops to source bucket
2021/12/16 00:09:23 Waiting for dst bucket item count to get to: 0 curr count: 15507
2021/12/16 00:09:28 Waiting for dst bucket item count to get to: 0 curr count: 15507
2021/12/16 00:09:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 00:09:28 Post DELETE ops. Expected item count: 0 got 0
2021/12/16 00:09:28 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 00:09:33 Waiting for app: TestEventingRebOutWithOps to get un-deployed
2021/12/16 00:09:38 Waiting for app: TestEventingRebOutWithOps to get un-deployed
2021/12/16 00:09:43 Waiting for app: TestEventingRebOutWithOps to get un-deployed
2021/12/16 00:09:48 Waiting for app: TestEventingRebOutWithOps to get un-deployed
2021/12/16 00:09:48 App: TestEventingRebOutWithOps got un-deployed
2021/12/16 00:09:48 No eventing-con process running
2021/12/16 00:09:48 Function: TestEventingRebOutWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebOutWithOps deleting in the background"
}
2021/12/16 00:09:50 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 00:09:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 00:09:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 00:09:57 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 00:10:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 00:10:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebOutWithOps (463.25s)
=== RUN   TestEventingSwapRebWithOps
2021/12/16 00:10:12 Waiting for app: TestEventingSwapRebWithOps to get un-deployed
2021/12/16 00:10:12 App: TestEventingSwapRebWithOps got un-deployed
2021/12/16 00:10:12 No eventing-con process running
2021/12/16 00:10:12 Function: TestEventingSwapRebWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingSwapRebWithOps not found"
 }
}
2021/12/16 00:10:13 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 00:10:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 00:10:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 00:10:20 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 00:10:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 00:10:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 00:10:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 00:10:30 Deploying app: TestEventingSwapRebWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/16 00:10:31 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingSwapRebWithOps' in metakv",
  "warnings": null
 }
}
2021/12/16 00:10:36 Waiting for app: TestEventingSwapRebWithOps to get deployed
2021/12/16 00:10:41 Waiting for app: TestEventingSwapRebWithOps to get deployed
2021/12/16 00:10:46 Waiting for app: TestEventingSwapRebWithOps to get deployed
2021/12/16 00:10:51 Waiting for app: TestEventingSwapRebWithOps to get deployed
2021/12/16 00:10:51 App: TestEventingSwapRebWithOps got deployed
2021/12/16 00:10:51 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/16 00:10:51 Starting bucket ops to source bucket
2021/12/16 00:10:52 Removing node(s): [] from the cluster
2021/12/16 00:10:57 Rebalance progress: 12.5
2021/12/16 00:11:02 Rebalance progress: 12.5
2021/12/16 00:11:07 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 00:11:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 00:11:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 00:11:07 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/16 00:11:07 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/16 00:11:12 Rebalance progress: 12.5
2021/12/16 00:11:17 Rebalance progress: 12.5
2021/12/16 00:11:22 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 00:11:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 00:11:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 00:11:23 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/16 00:11:28 Rebalance progress: 12.5
2021/12/16 00:11:33 Rebalance progress: 12.5
2021/12/16 00:11:38 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 00:11:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 00:11:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 00:11:38 Waiting for src bucket item count to get to: 200000 curr count: 58414
2021/12/16 00:11:43 Waiting for src bucket item count to get to: 200000 curr count: 73118
2021/12/16 00:11:48 Waiting for src bucket item count to get to: 200000 curr count: 73118
2021/12/16 00:11:53 Waiting for src bucket item count to get to: 200000 curr count: 88979
2021/12/16 00:11:58 Waiting for src bucket item count to get to: 200000 curr count: 88979
2021/12/16 00:12:03 Waiting for src bucket item count to get to: 200000 curr count: 105419
2021/12/16 00:12:08 Waiting for src bucket item count to get to: 200000 curr count: 105419
2021/12/16 00:12:13 Waiting for src bucket item count to get to: 200000 curr count: 121609
2021/12/16 00:12:18 Waiting for src bucket item count to get to: 200000 curr count: 121609
2021/12/16 00:12:23 Waiting for src bucket item count to get to: 200000 curr count: 137849
2021/12/16 00:12:28 Waiting for src bucket item count to get to: 200000 curr count: 137849
2021/12/16 00:12:33 Waiting for src bucket item count to get to: 200000 curr count: 154151
2021/12/16 00:12:38 Waiting for src bucket item count to get to: 200000 curr count: 154151
2021/12/16 00:12:43 Waiting for src bucket item count to get to: 200000 curr count: 169181
2021/12/16 00:12:48 Waiting for src bucket item count to get to: 200000 curr count: 169181
2021/12/16 00:12:53 Waiting for src bucket item count to get to: 200000 curr count: 185173
2021/12/16 00:12:58 Waiting for src bucket item count to get to: 200000 curr count: 185173
2021/12/16 00:13:03 Waiting for src bucket item count to get to: 200000 curr count: 185173
2021/12/16 00:13:08 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/16 00:13:08 Post SET ops. Expected item count: 200000 got 200000
2021/12/16 00:13:08 Starting bucket ops to source bucket
2021/12/16 00:16:58 Waiting for dst bucket item count to get to: 0 curr count: 6184
2021/12/16 00:17:03 Waiting for dst bucket item count to get to: 0 curr count: 6184
2021/12/16 00:17:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 00:17:03 Post DELETE ops. Expected item count: 0 got 0
2021/12/16 00:17:03 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 00:17:08 Waiting for app: TestEventingSwapRebWithOps to get un-deployed
2021/12/16 00:17:13 Waiting for app: TestEventingSwapRebWithOps to get un-deployed
2021/12/16 00:17:18 Waiting for app: TestEventingSwapRebWithOps to get un-deployed
2021/12/16 00:17:23 Waiting for app: TestEventingSwapRebWithOps to get un-deployed
2021/12/16 00:17:23 App: TestEventingSwapRebWithOps got un-deployed
2021/12/16 00:17:23 No eventing-con process running
2021/12/16 00:17:23 Function: TestEventingSwapRebWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingSwapRebWithOps deleting in the background"
}
2021/12/16 00:17:25 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 00:17:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 00:17:30 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 00:17:31 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 00:17:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 00:17:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingSwapRebWithOps (454.86s)
=== RUN   TestKVRebInWithOps
2021/12/16 00:17:46 Waiting for app: TestKVRebInWithOps to get un-deployed
2021/12/16 00:17:46 App: TestKVRebInWithOps got un-deployed
2021/12/16 00:17:46 No eventing-con process running
2021/12/16 00:17:46 Function: TestKVRebInWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVRebInWithOps not found"
 }
}
2021/12/16 00:17:48 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 00:17:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 00:17:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 00:17:55 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 00:18:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 00:18:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 00:18:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 00:18:05 Deploying app: TestKVRebInWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/16 00:18:06 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVRebInWithOps' in metakv",
  "warnings": null
 }
}
2021/12/16 00:18:11 Waiting for app: TestKVRebInWithOps to get deployed
2021/12/16 00:18:16 Waiting for app: TestKVRebInWithOps to get deployed
2021/12/16 00:18:21 Waiting for app: TestKVRebInWithOps to get deployed
2021/12/16 00:18:26 Waiting for app: TestKVRebInWithOps to get deployed
2021/12/16 00:18:26 App: TestKVRebInWithOps got deployed
2021/12/16 00:18:26 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/16 00:18:26 Starting bucket ops to source bucket
2021/12/16 00:18:27 Removing node(s): [] from the cluster
2021/12/16 00:18:32 Rebalance progress: 12.5
2021/12/16 00:18:37 Rebalance progress: 12.5
2021/12/16 00:18:42 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 00:18:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 00:18:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 00:18:42 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/16 00:18:47 Rebalance progress: 12.5
2021/12/16 00:18:52 Rebalance progress: 12.5
2021/12/16 00:18:57 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 00:18:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 00:18:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 00:18:57 Waiting for src bucket item count to get to: 200000 curr count: 34973
2021/12/16 00:19:02 Waiting for src bucket item count to get to: 200000 curr count: 51086
2021/12/16 00:19:07 Waiting for src bucket item count to get to: 200000 curr count: 51086
2021/12/16 00:19:12 Waiting for src bucket item count to get to: 200000 curr count: 67259
2021/12/16 00:19:17 Waiting for src bucket item count to get to: 200000 curr count: 67259
2021/12/16 00:19:22 Waiting for src bucket item count to get to: 200000 curr count: 81911
2021/12/16 00:19:27 Waiting for src bucket item count to get to: 200000 curr count: 81911
2021/12/16 00:19:32 Waiting for src bucket item count to get to: 200000 curr count: 98150
2021/12/16 00:19:37 Waiting for src bucket item count to get to: 200000 curr count: 98150
2021/12/16 00:19:42 Waiting for src bucket item count to get to: 200000 curr count: 114593
2021/12/16 00:19:47 Waiting for src bucket item count to get to: 200000 curr count: 114593
2021/12/16 00:19:52 Waiting for src bucket item count to get to: 200000 curr count: 114593
2021/12/16 00:19:57 Waiting for src bucket item count to get to: 200000 curr count: 130642
2021/12/16 00:20:02 Waiting for src bucket item count to get to: 200000 curr count: 130642
2021/12/16 00:20:07 Waiting for src bucket item count to get to: 200000 curr count: 146915
2021/12/16 00:20:12 Waiting for src bucket item count to get to: 200000 curr count: 146915
2021/12/16 00:20:17 Waiting for src bucket item count to get to: 200000 curr count: 162997
2021/12/16 00:20:22 Waiting for src bucket item count to get to: 200000 curr count: 162997
2021/12/16 00:20:27 Waiting for src bucket item count to get to: 200000 curr count: 177945
2021/12/16 00:20:32 Waiting for src bucket item count to get to: 200000 curr count: 177945
2021/12/16 00:20:37 Waiting for src bucket item count to get to: 200000 curr count: 194197
2021/12/16 00:20:42 Waiting for src bucket item count to get to: 200000 curr count: 194197
2021/12/16 00:20:47 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/16 00:20:47 Post SET ops. Expected item count: 200000 got 200000
2021/12/16 00:20:47 Starting bucket ops to source bucket
2021/12/16 00:24:37 Waiting for dst bucket item count to get to: 0 curr count: 5690
2021/12/16 00:24:42 Waiting for dst bucket item count to get to: 0 curr count: 5690
2021/12/16 00:24:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 00:24:42 Post DELETE ops. Expected item count: 0 got 0
2021/12/16 00:24:43 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 00:24:48 Waiting for app: TestKVRebInWithOps to get un-deployed
2021/12/16 00:24:53 Waiting for app: TestKVRebInWithOps to get un-deployed
2021/12/16 00:24:58 Waiting for app: TestKVRebInWithOps to get un-deployed
2021/12/16 00:25:03 Waiting for app: TestKVRebInWithOps to get un-deployed
2021/12/16 00:25:03 App: TestKVRebInWithOps got un-deployed
2021/12/16 00:25:03 No eventing-con process running
2021/12/16 00:25:03 Function: TestKVRebInWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVRebInWithOps deleting in the background"
}
2021/12/16 00:25:04 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 00:25:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 00:25:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 00:25:11 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 00:25:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 00:25:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebInWithOps (459.81s)
=== RUN   TestKVRebOutWithOps
2021/12/16 00:25:21 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/16 00:25:21 Removing node(s): [] from the cluster
2021/12/16 00:25:27 Rebalance progress: 100
2021/12/16 00:25:32 Waiting for app: TestKVRebOutWithOps to get un-deployed
2021/12/16 00:25:32 App: TestKVRebOutWithOps got un-deployed
2021/12/16 00:25:32 No eventing-con process running
2021/12/16 00:25:32 Function: TestKVRebOutWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVRebOutWithOps not found"
 }
}
2021/12/16 00:25:33 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 00:25:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 00:25:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 00:25:40 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 00:25:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 00:25:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 00:25:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 00:25:50 Deploying app: TestKVRebOutWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/16 00:25:51 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVRebOutWithOps' in metakv",
  "warnings": null
 }
}
2021/12/16 00:25:56 Waiting for app: TestKVRebOutWithOps to get deployed
2021/12/16 00:26:01 Waiting for app: TestKVRebOutWithOps to get deployed
2021/12/16 00:26:06 Waiting for app: TestKVRebOutWithOps to get deployed
2021/12/16 00:26:11 Waiting for app: TestKVRebOutWithOps to get deployed
2021/12/16 00:26:11 App: TestKVRebOutWithOps got deployed
2021/12/16 00:26:11 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/16 00:26:11 Starting bucket ops to source bucket
2021/12/16 00:26:16 Rebalance progress: 12.5
2021/12/16 00:26:21 Rebalance progress: 12.5
2021/12/16 00:26:26 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 00:26:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 00:26:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 00:26:26 Waiting for src bucket item count to get to: 200000 curr count: 11743
2021/12/16 00:26:31 Waiting for src bucket item count to get to: 200000 curr count: 27687
2021/12/16 00:26:36 Waiting for src bucket item count to get to: 200000 curr count: 27687
2021/12/16 00:26:41 Waiting for src bucket item count to get to: 200000 curr count: 43805
2021/12/16 00:26:46 Waiting for src bucket item count to get to: 200000 curr count: 43805
2021/12/16 00:26:51 Waiting for src bucket item count to get to: 200000 curr count: 59906
2021/12/16 00:26:56 Waiting for src bucket item count to get to: 200000 curr count: 59906
2021/12/16 00:27:01 Waiting for src bucket item count to get to: 200000 curr count: 59906
2021/12/16 00:27:06 Waiting for src bucket item count to get to: 200000 curr count: 75109
2021/12/16 00:27:11 Waiting for src bucket item count to get to: 200000 curr count: 75109
2021/12/16 00:27:16 Waiting for src bucket item count to get to: 200000 curr count: 90804
2021/12/16 00:27:21 Waiting for src bucket item count to get to: 200000 curr count: 90804
2021/12/16 00:27:26 Waiting for src bucket item count to get to: 200000 curr count: 106925
2021/12/16 00:27:31 Waiting for src bucket item count to get to: 200000 curr count: 106925
2021/12/16 00:27:36 Waiting for src bucket item count to get to: 200000 curr count: 122671
2021/12/16 00:27:41 Waiting for src bucket item count to get to: 200000 curr count: 122671
2021/12/16 00:27:47 Waiting for src bucket item count to get to: 200000 curr count: 138773
2021/12/16 00:27:52 Waiting for src bucket item count to get to: 200000 curr count: 138773
2021/12/16 00:27:57 Waiting for src bucket item count to get to: 200000 curr count: 155079
2021/12/16 00:28:02 Waiting for src bucket item count to get to: 200000 curr count: 155079
2021/12/16 00:28:07 Waiting for src bucket item count to get to: 200000 curr count: 170214
2021/12/16 00:28:12 Waiting for src bucket item count to get to: 200000 curr count: 170214
2021/12/16 00:28:17 Waiting for src bucket item count to get to: 200000 curr count: 186088
2021/12/16 00:28:22 Waiting for src bucket item count to get to: 200000 curr count: 186088
2021/12/16 00:28:27 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/16 00:28:27 Post SET ops. Expected item count: 200000 got 200000
2021/12/16 00:28:27 Starting bucket ops to source bucket
2021/12/16 00:32:16 Waiting for dst bucket item count to get to: 0 curr count: 20660
2021/12/16 00:32:21 Waiting for dst bucket item count to get to: 0 curr count: 2591
2021/12/16 00:32:26 Waiting for dst bucket item count to get to: 0 curr count: 2591
2021/12/16 00:32:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 00:32:26 Post DELETE ops. Expected item count: 0 got 0
2021/12/16 00:32:26 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 00:32:31 Waiting for app: TestKVRebOutWithOps to get un-deployed
2021/12/16 00:32:36 Waiting for app: TestKVRebOutWithOps to get un-deployed
2021/12/16 00:32:41 Waiting for app: TestKVRebOutWithOps to get un-deployed
2021/12/16 00:32:46 Waiting for app: TestKVRebOutWithOps to get un-deployed
2021/12/16 00:32:46 App: TestKVRebOutWithOps got un-deployed
2021/12/16 00:32:46 No eventing-con process running
2021/12/16 00:32:46 Function: TestKVRebOutWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVRebOutWithOps deleting in the background"
}
2021/12/16 00:32:48 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 00:32:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 00:32:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 00:32:55 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 00:33:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 00:33:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebOutWithOps (463.34s)
=== RUN   TestKVSwapRebWithOps
2021/12/16 00:33:10 Waiting for app: TestKVSwapRebWithOps to get un-deployed
2021/12/16 00:33:10 App: TestKVSwapRebWithOps got un-deployed
2021/12/16 00:33:10 No eventing-con process running
2021/12/16 00:33:10 Function: TestKVSwapRebWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVSwapRebWithOps not found"
 }
}
2021/12/16 00:33:11 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 00:33:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 00:33:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 00:33:18 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 00:33:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 00:33:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 00:33:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 00:33:28 Deploying app: TestKVSwapRebWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/16 00:33:29 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVSwapRebWithOps' in metakv",
  "warnings": null
 }
}
2021/12/16 00:33:34 Waiting for app: TestKVSwapRebWithOps to get deployed
2021/12/16 00:33:39 Waiting for app: TestKVSwapRebWithOps to get deployed
2021/12/16 00:33:44 Waiting for app: TestKVSwapRebWithOps to get deployed
2021/12/16 00:33:49 Waiting for app: TestKVSwapRebWithOps to get deployed
2021/12/16 00:33:49 App: TestKVSwapRebWithOps got deployed
2021/12/16 00:33:49 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/16 00:33:49 Starting bucket ops to source bucket
2021/12/16 00:33:50 Removing node(s): [] from the cluster
2021/12/16 00:33:55 Rebalance progress: 12.5
2021/12/16 00:34:00 Rebalance progress: 12.5
2021/12/16 00:34:05 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 00:34:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 00:34:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 00:34:05 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/16 00:34:05 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/16 00:34:10 Rebalance progress: 12.5
2021/12/16 00:34:15 Rebalance progress: 12.5
2021/12/16 00:34:20 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 00:34:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 00:34:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 00:34:20 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/16 00:34:25 Rebalance progress: 12.5
2021/12/16 00:34:30 Rebalance progress: 12.5
2021/12/16 00:34:35 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 00:34:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 00:34:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 00:34:35 Waiting for src bucket item count to get to: 200000 curr count: 61226
2021/12/16 00:34:40 Waiting for src bucket item count to get to: 200000 curr count: 61226
2021/12/16 00:34:45 Waiting for src bucket item count to get to: 200000 curr count: 75979
2021/12/16 00:34:50 Waiting for src bucket item count to get to: 200000 curr count: 91328
2021/12/16 00:34:55 Waiting for src bucket item count to get to: 200000 curr count: 91328
2021/12/16 00:35:00 Waiting for src bucket item count to get to: 200000 curr count: 107375
2021/12/16 00:35:05 Waiting for src bucket item count to get to: 200000 curr count: 107375
2021/12/16 00:35:11 Waiting for src bucket item count to get to: 200000 curr count: 123587
2021/12/16 00:35:16 Waiting for src bucket item count to get to: 200000 curr count: 123587
2021/12/16 00:35:21 Waiting for src bucket item count to get to: 200000 curr count: 139785
2021/12/16 00:35:26 Waiting for src bucket item count to get to: 200000 curr count: 139785
2021/12/16 00:35:31 Waiting for src bucket item count to get to: 200000 curr count: 155959
2021/12/16 00:35:36 Waiting for src bucket item count to get to: 200000 curr count: 155959
2021/12/16 00:35:41 Waiting for src bucket item count to get to: 200000 curr count: 171249
2021/12/16 00:35:46 Waiting for src bucket item count to get to: 200000 curr count: 171249
2021/12/16 00:35:51 Waiting for src bucket item count to get to: 200000 curr count: 171249
2021/12/16 00:35:56 Waiting for src bucket item count to get to: 200000 curr count: 187183
2021/12/16 00:36:01 Waiting for src bucket item count to get to: 200000 curr count: 187183
2021/12/16 00:36:06 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/16 00:36:06 Post SET ops. Expected item count: 200000 got 200000
2021/12/16 00:36:06 Starting bucket ops to source bucket
2021/12/16 00:39:55 Waiting for dst bucket item count to get to: 0 curr count: 1209
2021/12/16 00:40:00 Waiting for dst bucket item count to get to: 0 curr count: 1209
2021/12/16 00:40:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 00:40:00 Post DELETE ops. Expected item count: 0 got 0
2021/12/16 00:40:00 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 00:40:05 Waiting for app: TestKVSwapRebWithOps to get un-deployed
2021/12/16 00:40:10 Waiting for app: TestKVSwapRebWithOps to get un-deployed
2021/12/16 00:40:15 Waiting for app: TestKVSwapRebWithOps to get un-deployed
2021/12/16 00:40:20 Waiting for app: TestKVSwapRebWithOps to get un-deployed
2021/12/16 00:40:20 App: TestKVSwapRebWithOps got un-deployed
2021/12/16 00:40:20 No eventing-con process running
2021/12/16 00:40:20 Function: TestKVSwapRebWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVSwapRebWithOps deleting in the background"
}
2021/12/16 00:40:22 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 00:40:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 00:40:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 00:40:29 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 00:40:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 00:40:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVSwapRebWithOps (454.18s)
=== RUN   TestEventingRebInWithTimers
2021/12/16 00:40:44 Waiting for app: TestEventingRebInWithTimers to get un-deployed
2021/12/16 00:40:44 App: TestEventingRebInWithTimers got un-deployed
2021/12/16 00:40:44 No eventing-con process running
2021/12/16 00:40:44 Function: TestEventingRebInWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebInWithTimers not found"
 }
}
2021/12/16 00:40:46 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 00:40:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 00:40:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 00:40:52 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 00:40:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 00:40:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 00:41:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 00:41:02 Deploying app: TestEventingRebInWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/16 00:41:03 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebInWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/16 00:41:08 Waiting for app: TestEventingRebInWithTimers to get deployed
2021/12/16 00:41:13 Waiting for app: TestEventingRebInWithTimers to get deployed
2021/12/16 00:41:18 Waiting for app: TestEventingRebInWithTimers to get deployed
2021/12/16 00:41:23 Waiting for app: TestEventingRebInWithTimers to get deployed
2021/12/16 00:41:23 App: TestEventingRebInWithTimers got deployed
2021/12/16 00:41:23 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/16 00:41:23 Starting bucket ops to source bucket
2021/12/16 00:41:24 Removing node(s): [] from the cluster
2021/12/16 00:41:29 Rebalance progress: 12.5
2021/12/16 00:41:34 Rebalance progress: 12.5
2021/12/16 00:41:39 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 00:41:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 00:41:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 00:41:39 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/16 00:41:44 Rebalance progress: 12.5
2021/12/16 00:41:49 Rebalance progress: 12.5
2021/12/16 00:41:54 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 00:41:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 00:41:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 00:41:54 Waiting for src bucket item count to get to: 50000 curr count: 30414
2021/12/16 00:41:59 Waiting for src bucket item count to get to: 50000 curr count: 30414
2021/12/16 00:42:04 Waiting for src bucket item count to get to: 50000 curr count: 41583
2021/12/16 00:42:09 Waiting for src bucket item count to get to: 50000 curr count: 41583
2021/12/16 00:42:19 Waiting for dst bucket item count to get to: 50000 curr count: 40215
2021/12/16 00:42:24 Waiting for dst bucket item count to get to: 50000 curr count: 40215
2021/12/16 00:42:29 Waiting for dst bucket item count to get to: 50000 curr count: 49996
2021/12/16 00:42:34 Waiting for dst bucket item count to get to: 50000 curr count: 49996
2021/12/16 00:42:34 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/16 00:42:34 Post SET ops. Expected item count: 50000 got 50000
2021/12/16 00:42:34 Starting bucket ops to source bucket
2021/12/16 00:43:44 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/16 00:43:49 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/16 00:43:54 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/16 00:43:59 Waiting for dst bucket item count to get to: 0 curr count: 46522
2021/12/16 00:44:04 Waiting for dst bucket item count to get to: 0 curr count: 46522
2021/12/16 00:44:09 Waiting for dst bucket item count to get to: 0 curr count: 24622
2021/12/16 00:44:14 Waiting for dst bucket item count to get to: 0 curr count: 24622
2021/12/16 00:44:19 Waiting for dst bucket item count to get to: 0 curr count: 9629
2021/12/16 00:44:24 Waiting for dst bucket item count to get to: 0 curr count: 9629
2021/12/16 00:44:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 00:44:24 Post DELETE ops. Expected item count: 0 got 0
2021/12/16 00:44:24 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 00:44:29 Waiting for app: TestEventingRebInWithTimers to get un-deployed
2021/12/16 00:44:34 Waiting for app: TestEventingRebInWithTimers to get un-deployed
2021/12/16 00:44:39 Waiting for app: TestEventingRebInWithTimers to get un-deployed
2021/12/16 00:44:44 Waiting for app: TestEventingRebInWithTimers to get un-deployed
2021/12/16 00:44:44 App: TestEventingRebInWithTimers got un-deployed
2021/12/16 00:44:44 No eventing-con process running
2021/12/16 00:44:44 Function: TestEventingRebInWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebInWithTimers deleting in the background"
}
2021/12/16 00:44:46 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 00:44:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 00:44:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 00:44:53 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 00:44:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 00:44:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebInWithTimers (264.28s)
=== RUN   TestEventingRebOutWithTimers
2021/12/16 00:45:03 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/16 00:45:03 Removing node(s): [] from the cluster
2021/12/16 00:45:09 Rebalance progress: 100
2021/12/16 00:45:14 Waiting for app: TestEventingRebOutWithTimers to get un-deployed
2021/12/16 00:45:14 App: TestEventingRebOutWithTimers got un-deployed
2021/12/16 00:45:14 No eventing-con process running
2021/12/16 00:45:14 Function: TestEventingRebOutWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebOutWithTimers not found"
 }
}
2021/12/16 00:45:15 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 00:45:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 00:45:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 00:45:22 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 00:45:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 00:45:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 00:45:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 00:45:32 Deploying app: TestEventingRebOutWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/16 00:45:33 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebOutWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/16 00:45:38 Waiting for app: TestEventingRebOutWithTimers to get deployed
2021/12/16 00:45:43 Waiting for app: TestEventingRebOutWithTimers to get deployed
2021/12/16 00:45:48 Waiting for app: TestEventingRebOutWithTimers to get deployed
2021/12/16 00:45:53 Waiting for app: TestEventingRebOutWithTimers to get deployed
2021/12/16 00:45:53 App: TestEventingRebOutWithTimers got deployed
2021/12/16 00:45:53 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/16 00:45:53 Starting bucket ops to source bucket
2021/12/16 00:45:58 Rebalance progress: 12.5
2021/12/16 00:46:03 Rebalance progress: 12.5
2021/12/16 00:46:08 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 00:46:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 00:46:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 00:46:08 Waiting for src bucket item count to get to: 50000 curr count: 7878
2021/12/16 00:46:13 Waiting for src bucket item count to get to: 50000 curr count: 21601
2021/12/16 00:46:18 Waiting for src bucket item count to get to: 50000 curr count: 21601
2021/12/16 00:46:23 Waiting for src bucket item count to get to: 50000 curr count: 33297
2021/12/16 00:46:28 Waiting for src bucket item count to get to: 50000 curr count: 33297
2021/12/16 00:46:33 Waiting for src bucket item count to get to: 50000 curr count: 43867
2021/12/16 00:46:38 Waiting for src bucket item count to get to: 50000 curr count: 43867
2021/12/16 00:46:43 Waiting for src bucket item count to get to: 50000 curr count: 43867
2021/12/16 00:46:53 Waiting for dst bucket item count to get to: 50000 curr count: 35233
2021/12/16 00:46:58 Waiting for dst bucket item count to get to: 50000 curr count: 35233
2021/12/16 00:47:03 Waiting for dst bucket item count to get to: 50000 curr count: 48395
2021/12/16 00:47:08 Waiting for dst bucket item count to get to: 50000 curr count: 48395
2021/12/16 00:47:08 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/16 00:47:08 Post SET ops. Expected item count: 50000 got 50000
2021/12/16 00:47:08 Starting bucket ops to source bucket
2021/12/16 00:48:18 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/16 00:48:23 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/16 00:48:28 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/16 00:48:33 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/16 00:48:38 Waiting for dst bucket item count to get to: 0 curr count: 35107
2021/12/16 00:48:43 Waiting for dst bucket item count to get to: 0 curr count: 35107
2021/12/16 00:48:48 Waiting for dst bucket item count to get to: 0 curr count: 24607
2021/12/16 00:48:53 Waiting for dst bucket item count to get to: 0 curr count: 24607
2021/12/16 00:48:58 Waiting for dst bucket item count to get to: 0 curr count: 4281
2021/12/16 00:49:03 Waiting for dst bucket item count to get to: 0 curr count: 4281
2021/12/16 00:49:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 00:49:03 Post DELETE ops. Expected item count: 0 got 0
2021/12/16 00:49:04 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 00:49:09 Waiting for app: TestEventingRebOutWithTimers to get un-deployed
2021/12/16 00:49:14 Waiting for app: TestEventingRebOutWithTimers to get un-deployed
2021/12/16 00:49:19 Waiting for app: TestEventingRebOutWithTimers to get un-deployed
2021/12/16 00:49:24 Waiting for app: TestEventingRebOutWithTimers to get un-deployed
2021/12/16 00:49:24 App: TestEventingRebOutWithTimers got un-deployed
2021/12/16 00:49:24 No eventing-con process running
2021/12/16 00:49:24 Function: TestEventingRebOutWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebOutWithTimers deleting in the background"
}
2021/12/16 00:49:25 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 00:49:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 00:49:30 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 00:49:32 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 00:49:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 00:49:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebOutWithTimers (279.08s)
=== RUN   TestEventingSwapRebWithTimers
2021/12/16 00:49:47 Waiting for app: TestEventingSwapRebWithTimers to get un-deployed
2021/12/16 00:49:47 App: TestEventingSwapRebWithTimers got un-deployed
2021/12/16 00:49:47 No eventing-con process running
2021/12/16 00:49:47 Function: TestEventingSwapRebWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingSwapRebWithTimers not found"
 }
}
2021/12/16 00:49:49 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 00:49:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 00:49:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 00:49:56 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 00:50:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 00:50:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 00:50:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 00:50:06 Deploying app: TestEventingSwapRebWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/16 00:50:07 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingSwapRebWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/16 00:50:12 Waiting for app: TestEventingSwapRebWithTimers to get deployed
2021/12/16 00:50:17 Waiting for app: TestEventingSwapRebWithTimers to get deployed
2021/12/16 00:50:22 Waiting for app: TestEventingSwapRebWithTimers to get deployed
2021/12/16 00:50:27 Waiting for app: TestEventingSwapRebWithTimers to get deployed
2021/12/16 00:50:27 App: TestEventingSwapRebWithTimers got deployed
2021/12/16 00:50:27 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/16 00:50:27 Starting bucket ops to source bucket
2021/12/16 00:50:27 Removing node(s): [] from the cluster
2021/12/16 00:50:32 Rebalance progress: 12.5
2021/12/16 00:50:37 Rebalance progress: 12.5
2021/12/16 00:50:42 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 00:50:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 00:50:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 00:50:42 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/16 00:50:43 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/16 00:50:48 Rebalance progress: 12.5
2021/12/16 00:50:53 Rebalance progress: 12.5
2021/12/16 00:50:58 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 00:50:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 00:50:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 00:50:58 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/16 00:51:03 Rebalance progress: 12.5
2021/12/16 00:51:08 Rebalance progress: 12.5
2021/12/16 00:51:13 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 00:51:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 00:51:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 00:51:18 Waiting for dst bucket item count to get to: 50000 curr count: 31199
2021/12/16 00:51:23 Waiting for dst bucket item count to get to: 50000 curr count: 31199
2021/12/16 00:51:28 Waiting for dst bucket item count to get to: 50000 curr count: 42611
2021/12/16 00:51:33 Waiting for dst bucket item count to get to: 50000 curr count: 42611
2021/12/16 00:51:33 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/16 00:51:33 Post SET ops. Expected item count: 50000 got 50000
2021/12/16 00:51:33 Starting bucket ops to source bucket
2021/12/16 00:52:44 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/16 00:52:49 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/16 00:52:54 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/16 00:52:59 Waiting for dst bucket item count to get to: 0 curr count: 41823
2021/12/16 00:53:04 Waiting for dst bucket item count to get to: 0 curr count: 41823
2021/12/16 00:53:09 Waiting for dst bucket item count to get to: 0 curr count: 33037
2021/12/16 00:53:14 Waiting for dst bucket item count to get to: 0 curr count: 33037
2021/12/16 00:53:19 Waiting for dst bucket item count to get to: 0 curr count: 14331
2021/12/16 00:53:24 Waiting for dst bucket item count to get to: 0 curr count: 14331
2021/12/16 00:53:29 Waiting for dst bucket item count to get to: 0 curr count: 14331
2021/12/16 00:53:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 00:53:29 Post DELETE ops. Expected item count: 0 got 0
2021/12/16 00:53:29 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 00:53:34 Waiting for app: TestEventingSwapRebWithTimers to get un-deployed
2021/12/16 00:53:39 Waiting for app: TestEventingSwapRebWithTimers to get un-deployed
2021/12/16 00:53:44 Waiting for app: TestEventingSwapRebWithTimers to get un-deployed
2021/12/16 00:53:49 Waiting for app: TestEventingSwapRebWithTimers to get un-deployed
2021/12/16 00:53:49 App: TestEventingSwapRebWithTimers got un-deployed
2021/12/16 00:53:49 No eventing-con process running
2021/12/16 00:53:49 Function: TestEventingSwapRebWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingSwapRebWithTimers deleting in the background"
}
2021/12/16 00:53:51 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 00:53:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 00:53:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 00:53:58 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 00:54:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 00:54:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingSwapRebWithTimers (265.73s)
=== RUN   TestKVRebInWithTimers
2021/12/16 00:54:13 Waiting for app: TestKVRebInWithTimers to get un-deployed
2021/12/16 00:54:13 App: TestKVRebInWithTimers got un-deployed
2021/12/16 00:54:13 No eventing-con process running
2021/12/16 00:54:13 Function: TestKVRebInWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVRebInWithTimers not found"
 }
}
2021/12/16 00:54:15 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 00:54:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 00:54:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 00:54:21 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 00:54:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 00:54:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 00:54:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 00:54:31 Deploying app: TestKVRebInWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/16 00:54:33 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVRebInWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/16 00:54:38 Waiting for app: TestKVRebInWithTimers to get deployed
2021/12/16 00:54:43 Waiting for app: TestKVRebInWithTimers to get deployed
2021/12/16 00:54:48 Waiting for app: TestKVRebInWithTimers to get deployed
2021/12/16 00:54:53 Waiting for app: TestKVRebInWithTimers to get deployed
2021/12/16 00:54:53 App: TestKVRebInWithTimers got deployed
2021/12/16 00:54:53 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/16 00:54:53 Starting bucket ops to source bucket
2021/12/16 00:54:53 Removing node(s): [] from the cluster
2021/12/16 00:54:58 Rebalance progress: 12.5
2021/12/16 00:55:03 Rebalance progress: 12.5
2021/12/16 00:55:08 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 00:55:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 00:55:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 00:55:08 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/16 00:55:13 Rebalance progress: 12.5
2021/12/16 00:55:18 Rebalance progress: 12.5
2021/12/16 00:55:23 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 00:55:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 00:55:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 00:55:23 Waiting for src bucket item count to get to: 50000 curr count: 33944
2021/12/16 00:55:28 Waiting for src bucket item count to get to: 50000 curr count: 33944
2021/12/16 00:55:33 Waiting for src bucket item count to get to: 50000 curr count: 43974
2021/12/16 00:55:38 Waiting for src bucket item count to get to: 50000 curr count: 43974
2021/12/16 00:55:48 Waiting for dst bucket item count to get to: 50000 curr count: 34855
2021/12/16 00:55:53 Waiting for dst bucket item count to get to: 50000 curr count: 34855
2021/12/16 00:55:58 Waiting for dst bucket item count to get to: 50000 curr count: 49488
2021/12/16 00:56:03 Waiting for dst bucket item count to get to: 50000 curr count: 49488
2021/12/16 00:56:03 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/16 00:56:03 Post SET ops. Expected item count: 50000 got 50000
2021/12/16 00:56:03 Starting bucket ops to source bucket
2021/12/16 00:57:14 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/16 00:57:19 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/16 00:57:24 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/16 00:57:29 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/16 00:57:34 Waiting for dst bucket item count to get to: 0 curr count: 47122
2021/12/16 00:57:39 Waiting for dst bucket item count to get to: 0 curr count: 29266
2021/12/16 00:57:44 Waiting for dst bucket item count to get to: 0 curr count: 29266
2021/12/16 00:57:49 Waiting for dst bucket item count to get to: 0 curr count: 17931
2021/12/16 00:57:54 Waiting for dst bucket item count to get to: 0 curr count: 17931
2021/12/16 00:57:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 00:57:54 Post DELETE ops. Expected item count: 0 got 0
2021/12/16 00:57:54 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 00:57:59 Waiting for app: TestKVRebInWithTimers to get un-deployed
2021/12/16 00:58:04 Waiting for app: TestKVRebInWithTimers to get un-deployed
2021/12/16 00:58:09 Waiting for app: TestKVRebInWithTimers to get un-deployed
2021/12/16 00:58:14 Waiting for app: TestKVRebInWithTimers to get un-deployed
2021/12/16 00:58:14 App: TestKVRebInWithTimers got un-deployed
2021/12/16 00:58:14 No eventing-con process running
2021/12/16 00:58:14 Function: TestKVRebInWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVRebInWithTimers deleting in the background"
}
2021/12/16 00:58:16 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 00:58:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 00:58:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 00:58:23 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 00:58:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 00:58:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebInWithTimers (265.19s)
=== RUN   TestKVRebOutWithTimers
2021/12/16 00:58:33 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/16 00:58:33 Removing node(s): [] from the cluster
2021/12/16 00:58:39 Rebalance progress: 100
2021/12/16 00:58:44 Waiting for app: TestKVRebOutWithTimers to get un-deployed
2021/12/16 00:58:44 App: TestKVRebOutWithTimers got un-deployed
2021/12/16 00:58:44 No eventing-con process running
2021/12/16 00:58:44 Function: TestKVRebOutWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVRebOutWithTimers not found"
 }
}
2021/12/16 00:58:45 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 00:58:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 00:58:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 00:58:52 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 00:58:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 00:58:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 00:59:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 00:59:02 Deploying app: TestKVRebOutWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/16 00:59:03 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVRebOutWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/16 00:59:08 Waiting for app: TestKVRebOutWithTimers to get deployed
2021/12/16 00:59:13 Waiting for app: TestKVRebOutWithTimers to get deployed
2021/12/16 00:59:18 Waiting for app: TestKVRebOutWithTimers to get deployed
2021/12/16 00:59:23 Waiting for app: TestKVRebOutWithTimers to get deployed
2021/12/16 00:59:23 App: TestKVRebOutWithTimers got deployed
2021/12/16 00:59:23 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/16 00:59:23 Starting bucket ops to source bucket
2021/12/16 00:59:28 Rebalance progress: 12.5
2021/12/16 00:59:33 Rebalance progress: 12.5
2021/12/16 00:59:38 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 00:59:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 00:59:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 00:59:38 Waiting for src bucket item count to get to: 50000 curr count: 7566
2021/12/16 00:59:43 Waiting for src bucket item count to get to: 50000 curr count: 20253
2021/12/16 00:59:48 Waiting for src bucket item count to get to: 50000 curr count: 20253
2021/12/16 00:59:53 Waiting for src bucket item count to get to: 50000 curr count: 31893
2021/12/16 00:59:58 Waiting for src bucket item count to get to: 50000 curr count: 31893
2021/12/16 01:00:03 Waiting for src bucket item count to get to: 50000 curr count: 31893
2021/12/16 01:00:08 Waiting for src bucket item count to get to: 50000 curr count: 42907
2021/12/16 01:00:13 Waiting for src bucket item count to get to: 50000 curr count: 42907
2021/12/16 01:00:23 Waiting for dst bucket item count to get to: 50000 curr count: 38589
2021/12/16 01:00:28 Waiting for dst bucket item count to get to: 50000 curr count: 38589
2021/12/16 01:00:33 Waiting for dst bucket item count to get to: 50000 curr count: 49343
2021/12/16 01:00:38 Waiting for dst bucket item count to get to: 50000 curr count: 49343
2021/12/16 01:00:38 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/16 01:00:38 Post SET ops. Expected item count: 50000 got 50000
2021/12/16 01:00:38 Starting bucket ops to source bucket
2021/12/16 01:01:48 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/16 01:01:53 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/16 01:01:58 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/16 01:02:03 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/16 01:02:08 Waiting for dst bucket item count to get to: 0 curr count: 35412
2021/12/16 01:02:14 Waiting for dst bucket item count to get to: 0 curr count: 35412
2021/12/16 01:02:19 Waiting for dst bucket item count to get to: 0 curr count: 20472
2021/12/16 01:02:24 Waiting for dst bucket item count to get to: 0 curr count: 20472
2021/12/16 01:02:29 Waiting for dst bucket item count to get to: 0 curr count: 1504
2021/12/16 01:02:34 Waiting for dst bucket item count to get to: 0 curr count: 1504
2021/12/16 01:02:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 01:02:34 Post DELETE ops. Expected item count: 0 got 0
2021/12/16 01:02:34 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 01:02:39 Waiting for app: TestKVRebOutWithTimers to get un-deployed
2021/12/16 01:02:44 Waiting for app: TestKVRebOutWithTimers to get un-deployed
2021/12/16 01:02:49 Waiting for app: TestKVRebOutWithTimers to get un-deployed
2021/12/16 01:02:54 Waiting for app: TestKVRebOutWithTimers to get un-deployed
2021/12/16 01:02:54 App: TestKVRebOutWithTimers got un-deployed
2021/12/16 01:02:54 No eventing-con process running
2021/12/16 01:02:54 Function: TestKVRebOutWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVRebOutWithTimers deleting in the background"
}
2021/12/16 01:02:55 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:03:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 01:03:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 01:03:02 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:03:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 01:03:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebOutWithTimers (279.25s)
=== RUN   TestKVSwapRebWithTimers
2021/12/16 01:03:17 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/16 01:03:17 App: TestKVSwapRebWithTimers got un-deployed
2021/12/16 01:03:17 No eventing-con process running
2021/12/16 01:03:17 Function: TestKVSwapRebWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVSwapRebWithTimers not found"
 }
}
2021/12/16 01:03:19 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:03:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 01:03:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 01:03:26 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:03:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 01:03:31 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:03:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 01:03:36 Deploying app: TestKVSwapRebWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/16 01:03:37 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVSwapRebWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/16 01:03:42 Waiting for app: TestKVSwapRebWithTimers to get deployed
2021/12/16 01:03:47 Waiting for app: TestKVSwapRebWithTimers to get deployed
2021/12/16 01:03:52 Waiting for app: TestKVSwapRebWithTimers to get deployed
2021/12/16 01:03:57 Waiting for app: TestKVSwapRebWithTimers to get deployed
2021/12/16 01:03:57 App: TestKVSwapRebWithTimers got deployed
2021/12/16 01:03:57 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/16 01:03:57 Starting bucket ops to source bucket
2021/12/16 01:03:58 Removing node(s): [] from the cluster
2021/12/16 01:04:03 Rebalance progress: 12.5
2021/12/16 01:04:08 Rebalance progress: 12.5
2021/12/16 01:04:13 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:04:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:04:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 01:04:13 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/16 01:04:13 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/16 01:04:18 Rebalance progress: 12.5
2021/12/16 01:04:23 Rebalance progress: 12.5
2021/12/16 01:04:28 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:04:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:04:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 01:04:28 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/16 01:04:33 Rebalance progress: 12.5
2021/12/16 01:04:38 Rebalance progress: 12.5
2021/12/16 01:04:43 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:04:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:04:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 01:04:43 Waiting for src bucket item count to get to: 50000 curr count: 48947
2021/12/16 01:04:48 Waiting for src bucket item count to get to: 50000 curr count: 48947
2021/12/16 01:04:58 Waiting for dst bucket item count to get to: 50000 curr count: 44281
2021/12/16 01:05:03 Waiting for dst bucket item count to get to: 50000 curr count: 44281
2021/12/16 01:05:03 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/16 01:05:03 Post SET ops. Expected item count: 50000 got 50000
2021/12/16 01:05:03 Starting bucket ops to source bucket
2021/12/16 01:06:14 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/16 01:06:19 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/16 01:06:24 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/16 01:06:29 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/16 01:06:34 Waiting for dst bucket item count to get to: 0 curr count: 41797
2021/12/16 01:06:39 Waiting for dst bucket item count to get to: 0 curr count: 41797
2021/12/16 01:06:44 Waiting for dst bucket item count to get to: 0 curr count: 32579
2021/12/16 01:06:49 Waiting for dst bucket item count to get to: 0 curr count: 32579
2021/12/16 01:06:54 Waiting for dst bucket item count to get to: 0 curr count: 11257
2021/12/16 01:06:59 Waiting for dst bucket item count to get to: 0 curr count: 11257
2021/12/16 01:06:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 01:06:59 Post DELETE ops. Expected item count: 0 got 0
2021/12/16 01:06:59 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 01:07:04 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/16 01:07:09 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/16 01:07:14 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/16 01:07:19 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/16 01:07:24 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/16 01:07:29 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/16 01:07:29 App: TestKVSwapRebWithTimers got un-deployed
2021/12/16 01:07:29 No eventing-con process running
2021/12/16 01:07:29 Function: TestKVSwapRebWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVSwapRebWithTimers deleting in the background"
}
2021/12/16 01:07:31 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:07:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 01:07:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 01:07:38 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:07:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 01:07:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVSwapRebWithTimers (275.25s)
=== RUN   TestDuplicateTimerCreationEventingRebIn
2021/12/16 01:07:53 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/16 01:07:53 App: TestDuplicateTimerCreationEventingRebIn got un-deployed
2021/12/16 01:07:53 No eventing-con process running
2021/12/16 01:07:53 Function: TestDuplicateTimerCreationEventingRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationEventingRebIn not found"
 }
}
2021/12/16 01:07:54 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:07:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 01:07:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 01:08:01 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:08:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 01:08:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:08:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 01:08:11 Deploying app: TestDuplicateTimerCreationEventingRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/16 01:08:12 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationEventingRebIn' in metakv",
  "warnings": null
 }
}
2021/12/16 01:08:17 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get deployed
2021/12/16 01:08:22 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get deployed
2021/12/16 01:08:27 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get deployed
2021/12/16 01:08:32 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get deployed
2021/12/16 01:08:32 App: TestDuplicateTimerCreationEventingRebIn got deployed
2021/12/16 01:08:32 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/16 01:08:32 Starting bucket ops to source bucket
2021/12/16 01:08:33 Removing node(s): [] from the cluster
2021/12/16 01:08:38 Rebalance progress: 12.5
2021/12/16 01:08:43 Rebalance progress: 12.5
2021/12/16 01:08:48 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:08:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:08:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 01:08:48 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/16 01:08:53 Rebalance progress: 12.5
2021/12/16 01:08:58 Rebalance progress: 12.5
2021/12/16 01:09:03 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:09:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:09:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 01:09:03 Waiting for src bucket item count to get to: 50000 curr count: 35528
2021/12/16 01:09:08 Waiting for src bucket item count to get to: 50000 curr count: 35528
2021/12/16 01:09:13 Waiting for src bucket item count to get to: 50000 curr count: 44919
2021/12/16 01:09:18 Waiting for src bucket item count to get to: 50000 curr count: 44919
2021/12/16 01:09:28 Waiting for dst bucket item count to get to: 50000 curr count: 27349
2021/12/16 01:09:33 Waiting for dst bucket item count to get to: 50000 curr count: 27349
2021/12/16 01:09:38 Waiting for dst bucket item count to get to: 50000 curr count: 42736
2021/12/16 01:09:43 Waiting for dst bucket item count to get to: 50000 curr count: 42736
2021/12/16 01:09:48 Waiting for dst bucket item count to get to: 50000 curr count: 49915
2021/12/16 01:09:53 Waiting for dst bucket item count to get to: 50000 curr count: 49915
2021/12/16 01:09:53 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/16 01:09:53 Post SET ops. Expected item count: 50000 got 50000
2021/12/16 01:09:53 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 01:09:58 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/16 01:10:03 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/16 01:10:08 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/16 01:10:13 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/16 01:10:18 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/16 01:10:23 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/16 01:10:23 App: TestDuplicateTimerCreationEventingRebIn got un-deployed
2021/12/16 01:10:23 No eventing-con process running
2021/12/16 01:10:23 Function: TestDuplicateTimerCreationEventingRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationEventingRebIn deleting in the background"
}
2021/12/16 01:10:25 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:10:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 01:10:35 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/16 01:10:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 01:10:37 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:10:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 01:10:47 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/16 01:10:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationEventingRebIn (184.54s)
=== RUN   TestDuplicateTimerCreationEventingRebOut
2021/12/16 01:10:52 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/16 01:10:52 Removing node(s): [] from the cluster
2021/12/16 01:10:57 Rebalance progress: 100
2021/12/16 01:11:02 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/16 01:11:02 App: TestDuplicateTimerCreationEventingRebOut got un-deployed
2021/12/16 01:11:02 No eventing-con process running
2021/12/16 01:11:02 Function: TestDuplicateTimerCreationEventingRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationEventingRebOut not found"
 }
}
2021/12/16 01:11:04 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:11:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 01:11:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 01:11:11 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:11:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 01:11:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:11:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 01:11:21 Deploying app: TestDuplicateTimerCreationEventingRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/16 01:11:22 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationEventingRebOut' in metakv",
  "warnings": null
 }
}
2021/12/16 01:11:27 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get deployed
2021/12/16 01:11:32 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get deployed
2021/12/16 01:11:37 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get deployed
2021/12/16 01:11:42 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get deployed
2021/12/16 01:11:42 App: TestDuplicateTimerCreationEventingRebOut got deployed
2021/12/16 01:11:42 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/16 01:11:42 Starting bucket ops to source bucket
2021/12/16 01:11:47 Rebalance progress: 12.5
2021/12/16 01:11:52 Rebalance progress: 12.5
2021/12/16 01:11:57 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:11:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:11:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 01:11:57 Waiting for src bucket item count to get to: 50000 curr count: 8831
2021/12/16 01:12:02 Waiting for src bucket item count to get to: 50000 curr count: 22555
2021/12/16 01:12:07 Waiting for src bucket item count to get to: 50000 curr count: 22555
2021/12/16 01:12:12 Waiting for src bucket item count to get to: 50000 curr count: 36379
2021/12/16 01:12:17 Waiting for src bucket item count to get to: 50000 curr count: 36379
2021/12/16 01:12:22 Waiting for src bucket item count to get to: 50000 curr count: 46038
2021/12/16 01:12:27 Waiting for src bucket item count to get to: 50000 curr count: 46038
2021/12/16 01:12:32 Waiting for src bucket item count to get to: 50000 curr count: 46038
2021/12/16 01:12:42 Waiting for dst bucket item count to get to: 50000 curr count: 28520
2021/12/16 01:12:47 Waiting for dst bucket item count to get to: 50000 curr count: 28520
2021/12/16 01:12:52 Waiting for dst bucket item count to get to: 50000 curr count: 44965
2021/12/16 01:12:57 Waiting for dst bucket item count to get to: 50000 curr count: 44965
2021/12/16 01:12:57 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/16 01:12:57 Post SET ops. Expected item count: 50000 got 50000
2021/12/16 01:12:57 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 01:13:02 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/16 01:13:07 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/16 01:13:13 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/16 01:13:18 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/16 01:13:23 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/16 01:13:28 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/16 01:13:28 App: TestDuplicateTimerCreationEventingRebOut got un-deployed
2021/12/16 01:13:28 No eventing-con process running
2021/12/16 01:13:28 Function: TestDuplicateTimerCreationEventingRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationEventingRebOut deleting in the background"
}
2021/12/16 01:13:29 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:13:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 01:13:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 01:13:36 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:13:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 01:13:46 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/16 01:13:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationEventingRebOut (179.16s)
=== RUN   TestDuplicateTimerCreationEventingSwapReb
2021/12/16 01:13:56 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/16 01:13:56 App: TestDuplicateTimerCreationEventingSwapReb got un-deployed
2021/12/16 01:13:56 No eventing-con process running
2021/12/16 01:13:56 Function: TestDuplicateTimerCreationEventingSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationEventingSwapReb not found"
 }
}
2021/12/16 01:13:58 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:14:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 01:14:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 01:14:05 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:14:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 01:14:10 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:14:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 01:14:15 Deploying app: TestDuplicateTimerCreationEventingSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/16 01:14:16 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationEventingSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/16 01:14:21 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get deployed
2021/12/16 01:14:26 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get deployed
2021/12/16 01:14:31 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get deployed
2021/12/16 01:14:36 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get deployed
2021/12/16 01:14:36 App: TestDuplicateTimerCreationEventingSwapReb got deployed
2021/12/16 01:14:36 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/16 01:14:36 Starting bucket ops to source bucket
2021/12/16 01:14:36 Removing node(s): [] from the cluster
2021/12/16 01:14:41 Rebalance progress: 12.5
2021/12/16 01:14:46 Rebalance progress: 12.5
2021/12/16 01:14:51 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:14:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:14:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 01:14:51 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/16 01:14:52 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/16 01:14:57 Rebalance progress: 12.5
2021/12/16 01:15:02 Rebalance progress: 12.5
2021/12/16 01:15:07 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:15:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:15:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 01:15:07 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/16 01:15:12 Rebalance progress: 12.5
2021/12/16 01:15:17 Rebalance progress: 12.5
2021/12/16 01:15:22 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:15:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:15:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 01:15:27 Waiting for dst bucket item count to get to: 50000 curr count: 26110
2021/12/16 01:15:32 Waiting for dst bucket item count to get to: 50000 curr count: 26110
2021/12/16 01:15:37 Waiting for dst bucket item count to get to: 50000 curr count: 35979
2021/12/16 01:15:42 Waiting for dst bucket item count to get to: 50000 curr count: 35979
2021/12/16 01:15:47 Waiting for dst bucket item count to get to: 50000 curr count: 46054
2021/12/16 01:15:52 Waiting for dst bucket item count to get to: 50000 curr count: 46054
2021/12/16 01:15:52 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/16 01:15:52 Post SET ops. Expected item count: 50000 got 50000
2021/12/16 01:15:52 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 01:15:57 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/16 01:16:02 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/16 01:16:07 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/16 01:16:12 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/16 01:16:17 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/16 01:16:22 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/16 01:16:22 App: TestDuplicateTimerCreationEventingSwapReb got un-deployed
2021/12/16 01:16:22 No eventing-con process running
2021/12/16 01:16:22 Function: TestDuplicateTimerCreationEventingSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationEventingSwapReb deleting in the background"
}
2021/12/16 01:16:24 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:16:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 01:16:34 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/16 01:16:39 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/16 01:16:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 01:16:41 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:16:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 01:16:51 Waiting for dst bucket item count to get to: 0 curr count: 46663
2021/12/16 01:16:56 Waiting for dst bucket item count to get to: 0 curr count: 46663
2021/12/16 01:16:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationEventingSwapReb (189.70s)
=== RUN   TestDuplicateTimerCreationKVRebIn
2021/12/16 01:17:06 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/16 01:17:06 App: TestDuplicateTimerCreationKVRebIn got un-deployed
2021/12/16 01:17:06 No eventing-con process running
2021/12/16 01:17:06 Function: TestDuplicateTimerCreationKVRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationKVRebIn not found"
 }
}
2021/12/16 01:17:08 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:17:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 01:17:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 01:17:15 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:17:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 01:17:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:17:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 01:17:25 Deploying app: TestDuplicateTimerCreationKVRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/16 01:17:26 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationKVRebIn' in metakv",
  "warnings": null
 }
}
2021/12/16 01:17:31 Waiting for app: TestDuplicateTimerCreationKVRebIn to get deployed
2021/12/16 01:17:36 Waiting for app: TestDuplicateTimerCreationKVRebIn to get deployed
2021/12/16 01:17:41 Waiting for app: TestDuplicateTimerCreationKVRebIn to get deployed
2021/12/16 01:17:46 Waiting for app: TestDuplicateTimerCreationKVRebIn to get deployed
2021/12/16 01:17:46 App: TestDuplicateTimerCreationKVRebIn got deployed
2021/12/16 01:17:46 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/16 01:17:46 Starting bucket ops to source bucket
2021/12/16 01:17:46 Removing node(s): [] from the cluster
2021/12/16 01:17:51 Rebalance progress: 12.5
2021/12/16 01:17:56 Rebalance progress: 12.5
2021/12/16 01:18:01 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:18:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:18:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 01:18:01 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/16 01:18:06 Rebalance progress: 12.5
2021/12/16 01:18:11 Rebalance progress: 12.5
2021/12/16 01:18:16 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:18:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:18:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 01:18:16 Waiting for src bucket item count to get to: 50000 curr count: 30571
2021/12/16 01:18:21 Waiting for src bucket item count to get to: 50000 curr count: 41843
2021/12/16 01:18:26 Waiting for src bucket item count to get to: 50000 curr count: 41843
2021/12/16 01:18:36 Waiting for dst bucket item count to get to: 50000 curr count: 26551
2021/12/16 01:18:41 Waiting for dst bucket item count to get to: 50000 curr count: 26551
2021/12/16 01:18:46 Waiting for dst bucket item count to get to: 50000 curr count: 35675
2021/12/16 01:18:51 Waiting for dst bucket item count to get to: 50000 curr count: 35675
2021/12/16 01:18:56 Waiting for dst bucket item count to get to: 50000 curr count: 47907
2021/12/16 01:19:01 Waiting for dst bucket item count to get to: 50000 curr count: 47907
2021/12/16 01:19:01 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/16 01:19:01 Post SET ops. Expected item count: 50000 got 50000
2021/12/16 01:19:01 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 01:19:06 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/16 01:19:12 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/16 01:19:17 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/16 01:19:22 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/16 01:19:27 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/16 01:19:32 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/16 01:19:32 App: TestDuplicateTimerCreationKVRebIn got un-deployed
2021/12/16 01:19:32 No eventing-con process running
2021/12/16 01:19:32 Function: TestDuplicateTimerCreationKVRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationKVRebIn deleting in the background"
}
2021/12/16 01:19:34 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:19:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 01:19:44 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/16 01:19:49 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/16 01:19:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 01:19:50 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:19:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 01:20:00 Waiting for dst bucket item count to get to: 0 curr count: 19236
2021/12/16 01:20:05 Waiting for dst bucket item count to get to: 0 curr count: 19236
2021/12/16 01:20:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationKVRebIn (189.35s)
=== RUN   TestDuplicateTimerCreationKVRebOut
2021/12/16 01:20:10 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/16 01:20:11 Removing node(s): [] from the cluster
2021/12/16 01:20:16 Rebalance progress: 100
2021/12/16 01:20:21 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/16 01:20:21 App: TestDuplicateTimerCreationKVRebOut got un-deployed
2021/12/16 01:20:21 No eventing-con process running
2021/12/16 01:20:21 Function: TestDuplicateTimerCreationKVRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationKVRebOut not found"
 }
}
2021/12/16 01:20:22 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:20:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 01:20:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 01:20:29 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:20:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 01:20:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:20:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 01:20:39 Deploying app: TestDuplicateTimerCreationKVRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/16 01:20:40 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationKVRebOut' in metakv",
  "warnings": null
 }
}
2021/12/16 01:20:45 Waiting for app: TestDuplicateTimerCreationKVRebOut to get deployed
2021/12/16 01:20:50 Waiting for app: TestDuplicateTimerCreationKVRebOut to get deployed
2021/12/16 01:20:55 Waiting for app: TestDuplicateTimerCreationKVRebOut to get deployed
2021/12/16 01:21:00 Waiting for app: TestDuplicateTimerCreationKVRebOut to get deployed
2021/12/16 01:21:00 App: TestDuplicateTimerCreationKVRebOut got deployed
2021/12/16 01:21:00 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/16 01:21:00 Starting bucket ops to source bucket
2021/12/16 01:21:06 Rebalance progress: 12.5
2021/12/16 01:21:11 Rebalance progress: 12.5
2021/12/16 01:21:16 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:21:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:21:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 01:21:16 Waiting for src bucket item count to get to: 50000 curr count: 11503
2021/12/16 01:21:21 Waiting for src bucket item count to get to: 50000 curr count: 25204
2021/12/16 01:21:26 Waiting for src bucket item count to get to: 50000 curr count: 25204
2021/12/16 01:21:31 Waiting for src bucket item count to get to: 50000 curr count: 37787
2021/12/16 01:21:36 Waiting for src bucket item count to get to: 50000 curr count: 37787
2021/12/16 01:21:41 Waiting for src bucket item count to get to: 50000 curr count: 47907
2021/12/16 01:21:46 Waiting for src bucket item count to get to: 50000 curr count: 47907
2021/12/16 01:21:51 Waiting for src bucket item count to get to: 50000 curr count: 47907
2021/12/16 01:22:01 Waiting for dst bucket item count to get to: 50000 curr count: 31772
2021/12/16 01:22:06 Waiting for dst bucket item count to get to: 50000 curr count: 31772
2021/12/16 01:22:11 Waiting for dst bucket item count to get to: 50000 curr count: 44036
2021/12/16 01:22:16 Waiting for dst bucket item count to get to: 50000 curr count: 44036
2021/12/16 01:22:16 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/16 01:22:16 Post SET ops. Expected item count: 50000 got 50000
2021/12/16 01:22:16 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 01:22:21 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/16 01:22:26 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/16 01:22:31 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/16 01:22:36 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/16 01:22:41 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/16 01:22:46 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/16 01:22:46 App: TestDuplicateTimerCreationKVRebOut got un-deployed
2021/12/16 01:22:46 No eventing-con process running
2021/12/16 01:22:46 Function: TestDuplicateTimerCreationKVRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationKVRebOut deleting in the background"
}
2021/12/16 01:22:48 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:22:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 01:22:58 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/16 01:22:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 01:23:00 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:23:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 01:23:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationKVRebOut (179.42s)
=== RUN   TestDuplicateTimerCreationKVSwapReb
2021/12/16 01:23:15 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/16 01:23:15 App: TestDuplicateTimerCreationKVSwapReb got un-deployed
2021/12/16 01:23:15 No eventing-con process running
2021/12/16 01:23:15 Function: TestDuplicateTimerCreationKVSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationKVSwapReb not found"
 }
}
2021/12/16 01:23:17 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:23:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 01:23:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 01:23:23 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:23:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 01:23:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:23:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 01:23:33 Deploying app: TestDuplicateTimerCreationKVSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/16 01:23:34 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationKVSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/16 01:23:39 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get deployed
2021/12/16 01:23:44 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get deployed
2021/12/16 01:23:49 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get deployed
2021/12/16 01:23:54 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get deployed
2021/12/16 01:23:54 App: TestDuplicateTimerCreationKVSwapReb got deployed
2021/12/16 01:23:54 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/16 01:23:54 Starting bucket ops to source bucket
2021/12/16 01:23:55 Removing node(s): [] from the cluster
2021/12/16 01:24:00 Rebalance progress: 12.5
2021/12/16 01:24:05 Rebalance progress: 12.5
2021/12/16 01:24:10 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:24:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:24:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 01:24:10 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/16 01:24:10 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/16 01:24:15 Rebalance progress: 12.5
2021/12/16 01:24:20 Rebalance progress: 12.5
2021/12/16 01:24:25 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:24:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:24:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 01:24:25 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/16 01:24:30 Rebalance progress: 12.5
2021/12/16 01:24:35 Rebalance progress: 12.5
2021/12/16 01:24:40 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:24:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:24:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 01:24:45 Waiting for dst bucket item count to get to: 50000 curr count: 29554
2021/12/16 01:24:50 Waiting for dst bucket item count to get to: 50000 curr count: 29554
2021/12/16 01:24:55 Waiting for dst bucket item count to get to: 50000 curr count: 35423
2021/12/16 01:25:00 Waiting for dst bucket item count to get to: 50000 curr count: 35423
2021/12/16 01:25:05 Waiting for dst bucket item count to get to: 50000 curr count: 49415
2021/12/16 01:25:10 Waiting for dst bucket item count to get to: 50000 curr count: 49415
2021/12/16 01:25:10 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/16 01:25:10 Post SET ops. Expected item count: 50000 got 50000
2021/12/16 01:25:10 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 01:25:15 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/16 01:25:20 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/16 01:25:26 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/16 01:25:31 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/16 01:25:36 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/16 01:25:41 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/16 01:25:41 App: TestDuplicateTimerCreationKVSwapReb got un-deployed
2021/12/16 01:25:41 No eventing-con process running
2021/12/16 01:25:41 Function: TestDuplicateTimerCreationKVSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationKVSwapReb deleting in the background"
}
2021/12/16 01:25:42 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:25:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 01:25:52 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/16 01:25:57 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/16 01:25:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 01:25:59 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:26:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 01:26:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationKVSwapReb (179.68s)
=== RUN   TestDuplicateTimerFiringEventingRebIn
2021/12/16 01:26:14 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/16 01:26:14 App: TestDuplicateTimerFiringEventingRebIn got un-deployed
2021/12/16 01:26:14 No eventing-con process running
2021/12/16 01:26:14 Function: TestDuplicateTimerFiringEventingRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringEventingRebIn not found"
 }
}
2021/12/16 01:26:16 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:26:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 01:26:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 01:26:23 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:26:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 01:26:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:26:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 01:26:33 Deploying app: TestDuplicateTimerFiringEventingRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/16 01:26:34 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringEventingRebIn' in metakv",
  "warnings": null
 }
}
2021/12/16 01:26:39 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get deployed
2021/12/16 01:26:44 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get deployed
2021/12/16 01:26:49 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get deployed
2021/12/16 01:26:54 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get deployed
2021/12/16 01:26:54 App: TestDuplicateTimerFiringEventingRebIn got deployed
2021/12/16 01:26:54 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/16 01:26:54 Starting bucket ops to source bucket
2021/12/16 01:26:55 Removing node(s): [] from the cluster
2021/12/16 01:27:00 Rebalance progress: 12.5
2021/12/16 01:27:05 Rebalance progress: 12.5
2021/12/16 01:27:10 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:27:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:27:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 01:27:10 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/16 01:27:15 Rebalance progress: 12.5
2021/12/16 01:27:20 Rebalance progress: 12.5
2021/12/16 01:27:25 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:27:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:27:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 01:27:25 Waiting for src bucket item count to get to: 50000 curr count: 29064
2021/12/16 01:27:30 Waiting for src bucket item count to get to: 50000 curr count: 29064
2021/12/16 01:27:35 Waiting for src bucket item count to get to: 50000 curr count: 40818
2021/12/16 01:27:40 Waiting for src bucket item count to get to: 50000 curr count: 40818
2021/12/16 01:27:50 Waiting for dst bucket item count to get to: 50000 curr count: 38837
2021/12/16 01:27:55 Waiting for dst bucket item count to get to: 50000 curr count: 38837
2021/12/16 01:28:00 Waiting for dst bucket item count to get to: 50000 curr count: 46853
2021/12/16 01:28:00 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/16 01:28:00 Post SET ops. Expected item count: 50000 got 50000
2021/12/16 01:28:00 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 01:28:05 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/16 01:28:10 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/16 01:28:15 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/16 01:28:20 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/16 01:28:25 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/16 01:28:30 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/16 01:28:30 App: TestDuplicateTimerFiringEventingRebIn got un-deployed
2021/12/16 01:28:30 No eventing-con process running
2021/12/16 01:28:30 Function: TestDuplicateTimerFiringEventingRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringEventingRebIn deleting in the background"
}
2021/12/16 01:28:32 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:28:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 01:28:42 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/16 01:28:47 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/16 01:28:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 01:28:49 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:28:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 01:28:59 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/16 01:28:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringEventingRebIn (174.29s)
=== RUN   TestDuplicateTimerFiringEventingRebOut
2021/12/16 01:29:04 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/16 01:29:04 Removing node(s): [] from the cluster
2021/12/16 01:29:09 Rebalance progress: 100
2021/12/16 01:29:14 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/16 01:29:14 App: TestDuplicateTimerFiringEventingRebOut got un-deployed
2021/12/16 01:29:14 No eventing-con process running
2021/12/16 01:29:14 Function: TestDuplicateTimerFiringEventingRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringEventingRebOut not found"
 }
}
2021/12/16 01:29:16 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:29:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 01:29:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 01:29:23 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:29:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 01:29:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:29:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 01:29:33 Deploying app: TestDuplicateTimerFiringEventingRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/16 01:29:34 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringEventingRebOut' in metakv",
  "warnings": null
 }
}
2021/12/16 01:29:39 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get deployed
2021/12/16 01:29:44 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get deployed
2021/12/16 01:29:49 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get deployed
2021/12/16 01:29:54 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get deployed
2021/12/16 01:29:54 App: TestDuplicateTimerFiringEventingRebOut got deployed
2021/12/16 01:29:54 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/16 01:29:54 Starting bucket ops to source bucket
2021/12/16 01:29:59 Rebalance progress: 12.5
2021/12/16 01:30:04 Rebalance progress: 12.5
2021/12/16 01:30:09 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:30:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:30:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 01:30:09 Waiting for src bucket item count to get to: 50000 curr count: 7060
2021/12/16 01:30:14 Waiting for src bucket item count to get to: 50000 curr count: 21384
2021/12/16 01:30:19 Waiting for src bucket item count to get to: 50000 curr count: 21384
2021/12/16 01:30:24 Waiting for src bucket item count to get to: 50000 curr count: 32671
2021/12/16 01:30:29 Waiting for src bucket item count to get to: 50000 curr count: 32671
2021/12/16 01:30:34 Waiting for src bucket item count to get to: 50000 curr count: 32671
2021/12/16 01:30:39 Waiting for src bucket item count to get to: 50000 curr count: 43516
2021/12/16 01:30:44 Waiting for src bucket item count to get to: 50000 curr count: 43516
2021/12/16 01:30:54 Waiting for dst bucket item count to get to: 50000 curr count: 35244
2021/12/16 01:30:59 Waiting for dst bucket item count to get to: 50000 curr count: 35244
2021/12/16 01:31:04 Waiting for dst bucket item count to get to: 50000 curr count: 43479
2021/12/16 01:31:09 Waiting for dst bucket item count to get to: 50000 curr count: 43479
2021/12/16 01:31:09 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/16 01:31:09 Post SET ops. Expected item count: 50000 got 50000
2021/12/16 01:31:09 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 01:31:14 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/16 01:31:19 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/16 01:31:24 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/16 01:31:30 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/16 01:31:35 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/16 01:31:35 App: TestDuplicateTimerFiringEventingRebOut got un-deployed
2021/12/16 01:31:35 No eventing-con process running
2021/12/16 01:31:35 Function: TestDuplicateTimerFiringEventingRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringEventingRebOut deleting in the background"
}
2021/12/16 01:31:36 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:31:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 01:31:46 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/16 01:31:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 01:31:48 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:31:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 01:31:58 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/16 01:31:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringEventingRebOut (179.48s)
=== RUN   TestDuplicateTimerFiringEventingSwapReb
2021/12/16 01:32:08 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/16 01:32:08 App: TestDuplicateTimerFiringEventingSwapReb got un-deployed
2021/12/16 01:32:08 No eventing-con process running
2021/12/16 01:32:08 Function: TestDuplicateTimerFiringEventingSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringEventingSwapReb not found"
 }
}
2021/12/16 01:32:10 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:32:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 01:32:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 01:32:17 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:32:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 01:32:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:32:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 01:32:27 Deploying app: TestDuplicateTimerFiringEventingSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/16 01:32:28 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringEventingSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/16 01:32:33 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get deployed
2021/12/16 01:32:38 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get deployed
2021/12/16 01:32:43 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get deployed
2021/12/16 01:32:48 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get deployed
2021/12/16 01:32:48 App: TestDuplicateTimerFiringEventingSwapReb got deployed
2021/12/16 01:32:48 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/16 01:32:48 Starting bucket ops to source bucket
2021/12/16 01:32:48 Removing node(s): [] from the cluster
2021/12/16 01:32:53 Rebalance progress: 12.5
2021/12/16 01:32:58 Rebalance progress: 12.5
2021/12/16 01:33:03 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:33:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:33:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 01:33:03 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/16 01:33:04 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/16 01:33:09 Rebalance progress: 12.5
2021/12/16 01:33:14 Rebalance progress: 12.5
2021/12/16 01:33:19 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:33:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:33:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 01:33:19 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/16 01:33:24 Rebalance progress: 12.5
2021/12/16 01:33:29 Rebalance progress: 12.5
2021/12/16 01:33:34 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:33:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:33:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 01:33:34 Waiting for src bucket item count to get to: 50000 curr count: 48199
2021/12/16 01:33:39 Waiting for src bucket item count to get to: 50000 curr count: 48199
2021/12/16 01:33:49 Waiting for dst bucket item count to get to: 50000 curr count: 40763
2021/12/16 01:33:54 Waiting for dst bucket item count to get to: 50000 curr count: 40763
2021/12/16 01:33:59 Waiting for dst bucket item count to get to: 50000 curr count: 48298
2021/12/16 01:34:04 Waiting for dst bucket item count to get to: 50000 curr count: 48298
2021/12/16 01:34:04 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/16 01:34:04 Post SET ops. Expected item count: 50000 got 50000
2021/12/16 01:34:04 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 01:34:09 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/16 01:34:14 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/16 01:34:19 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/16 01:34:24 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/16 01:34:29 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/16 01:34:34 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/16 01:34:34 App: TestDuplicateTimerFiringEventingSwapReb got un-deployed
2021/12/16 01:34:34 No eventing-con process running
2021/12/16 01:34:34 Function: TestDuplicateTimerFiringEventingSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringEventingSwapReb deleting in the background"
}
2021/12/16 01:34:36 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:34:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 01:34:46 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/16 01:34:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 01:34:48 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:34:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 01:34:58 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/16 01:34:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringEventingSwapReb (179.67s)
=== RUN   TestDuplicateTimerFiringKVRebIn
2021/12/16 01:35:08 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/16 01:35:08 App: TestDuplicateTimerFiringKVRebIn got un-deployed
2021/12/16 01:35:08 No eventing-con process running
2021/12/16 01:35:08 Function: TestDuplicateTimerFiringKVRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringKVRebIn not found"
 }
}
2021/12/16 01:35:10 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:35:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 01:35:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 01:35:16 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:35:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 01:35:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:35:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 01:35:26 Deploying app: TestDuplicateTimerFiringKVRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/16 01:35:28 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringKVRebIn' in metakv",
  "warnings": null
 }
}
2021/12/16 01:35:33 Waiting for app: TestDuplicateTimerFiringKVRebIn to get deployed
2021/12/16 01:35:38 Waiting for app: TestDuplicateTimerFiringKVRebIn to get deployed
2021/12/16 01:35:43 Waiting for app: TestDuplicateTimerFiringKVRebIn to get deployed
2021/12/16 01:35:48 Waiting for app: TestDuplicateTimerFiringKVRebIn to get deployed
2021/12/16 01:35:48 App: TestDuplicateTimerFiringKVRebIn got deployed
2021/12/16 01:35:48 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/16 01:35:48 Starting bucket ops to source bucket
2021/12/16 01:35:48 Removing node(s): [] from the cluster
2021/12/16 01:35:53 Rebalance progress: 12.5
2021/12/16 01:35:58 Rebalance progress: 12.5
2021/12/16 01:36:03 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:36:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:36:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 01:36:03 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/16 01:36:08 Rebalance progress: 12.5
2021/12/16 01:36:13 Rebalance progress: 12.5
2021/12/16 01:36:18 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:36:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:36:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 01:36:18 Waiting for src bucket item count to get to: 50000 curr count: 26877
2021/12/16 01:36:23 Waiting for src bucket item count to get to: 50000 curr count: 36861
2021/12/16 01:36:28 Waiting for src bucket item count to get to: 50000 curr count: 36861
2021/12/16 01:36:33 Waiting for src bucket item count to get to: 50000 curr count: 48804
2021/12/16 01:36:38 Waiting for src bucket item count to get to: 50000 curr count: 48804
2021/12/16 01:36:48 Waiting for dst bucket item count to get to: 50000 curr count: 44972
2021/12/16 01:36:53 Waiting for dst bucket item count to get to: 50000 curr count: 44972
2021/12/16 01:36:53 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/16 01:36:53 Post SET ops. Expected item count: 50000 got 50000
2021/12/16 01:36:53 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 01:36:58 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/16 01:37:04 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/16 01:37:09 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/16 01:37:14 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/16 01:37:19 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/16 01:37:24 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/16 01:37:24 App: TestDuplicateTimerFiringKVRebIn got un-deployed
2021/12/16 01:37:24 No eventing-con process running
2021/12/16 01:37:24 Function: TestDuplicateTimerFiringKVRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringKVRebIn deleting in the background"
}
2021/12/16 01:37:26 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:37:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 01:37:36 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/16 01:37:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 01:37:37 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:37:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 01:37:47 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/16 01:37:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringKVRebIn (169.48s)
=== RUN   TestDuplicateTimerFiringKVRebOut
2021/12/16 01:37:52 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/16 01:37:52 Removing node(s): [] from the cluster
2021/12/16 01:37:58 Rebalance progress: 100
2021/12/16 01:38:03 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/16 01:38:03 App: TestDuplicateTimerFiringKVRebOut got un-deployed
2021/12/16 01:38:03 No eventing-con process running
2021/12/16 01:38:03 Function: TestDuplicateTimerFiringKVRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringKVRebOut not found"
 }
}
2021/12/16 01:38:04 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:38:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 01:38:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 01:38:11 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:38:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 01:38:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:38:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 01:38:21 Deploying app: TestDuplicateTimerFiringKVRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/16 01:38:22 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringKVRebOut' in metakv",
  "warnings": null
 }
}
2021/12/16 01:38:27 Waiting for app: TestDuplicateTimerFiringKVRebOut to get deployed
2021/12/16 01:38:32 Waiting for app: TestDuplicateTimerFiringKVRebOut to get deployed
2021/12/16 01:38:37 Waiting for app: TestDuplicateTimerFiringKVRebOut to get deployed
2021/12/16 01:38:42 Waiting for app: TestDuplicateTimerFiringKVRebOut to get deployed
2021/12/16 01:38:42 App: TestDuplicateTimerFiringKVRebOut got deployed
2021/12/16 01:38:42 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/16 01:38:42 Starting bucket ops to source bucket
2021/12/16 01:38:47 Rebalance progress: 12.5
2021/12/16 01:38:52 Rebalance progress: 12.5
2021/12/16 01:38:57 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:38:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:38:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 01:38:57 Waiting for src bucket item count to get to: 50000 curr count: 8987
2021/12/16 01:39:02 Waiting for src bucket item count to get to: 50000 curr count: 21337
2021/12/16 01:39:07 Waiting for src bucket item count to get to: 50000 curr count: 21337
2021/12/16 01:39:12 Waiting for src bucket item count to get to: 50000 curr count: 32901
2021/12/16 01:39:17 Waiting for src bucket item count to get to: 50000 curr count: 32901
2021/12/16 01:39:23 Waiting for src bucket item count to get to: 50000 curr count: 32901
2021/12/16 01:39:28 Waiting for src bucket item count to get to: 50000 curr count: 42710
2021/12/16 01:39:33 Waiting for src bucket item count to get to: 50000 curr count: 42710
2021/12/16 01:39:43 Waiting for dst bucket item count to get to: 50000 curr count: 37715
2021/12/16 01:39:48 Waiting for dst bucket item count to get to: 50000 curr count: 37715
2021/12/16 01:39:48 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/16 01:39:48 Post SET ops. Expected item count: 50000 got 50000
2021/12/16 01:39:48 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 01:39:53 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/16 01:39:58 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/16 01:40:03 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/16 01:40:08 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/16 01:40:13 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/16 01:40:18 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/16 01:40:18 App: TestDuplicateTimerFiringKVRebOut got un-deployed
2021/12/16 01:40:18 No eventing-con process running
2021/12/16 01:40:18 Function: TestDuplicateTimerFiringKVRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringKVRebOut deleting in the background"
}
2021/12/16 01:40:20 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:40:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 01:40:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 01:40:26 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:40:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 01:40:36 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/16 01:40:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringKVRebOut (169.13s)
=== RUN   TestDuplicateTimerFiringKVSwapReb
2021/12/16 01:40:46 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/16 01:40:46 App: TestDuplicateTimerFiringKVSwapReb got un-deployed
2021/12/16 01:40:46 No eventing-con process running
2021/12/16 01:40:46 Function: TestDuplicateTimerFiringKVSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringKVSwapReb not found"
 }
}
2021/12/16 01:40:48 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:40:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 01:40:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 01:40:55 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:41:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 01:41:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:41:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 01:41:05 Deploying app: TestDuplicateTimerFiringKVSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/16 01:41:06 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringKVSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/16 01:41:11 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get deployed
2021/12/16 01:41:16 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get deployed
2021/12/16 01:41:21 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get deployed
2021/12/16 01:41:26 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get deployed
2021/12/16 01:41:26 App: TestDuplicateTimerFiringKVSwapReb got deployed
2021/12/16 01:41:26 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/16 01:41:26 Starting bucket ops to source bucket
2021/12/16 01:41:26 Removing node(s): [] from the cluster
2021/12/16 01:41:32 Rebalance progress: 12.5
2021/12/16 01:41:37 Rebalance progress: 12.5
2021/12/16 01:41:42 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:41:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:41:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 01:41:42 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/16 01:41:42 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/16 01:41:47 Rebalance progress: 12.5
2021/12/16 01:41:52 Rebalance progress: 12.5
2021/12/16 01:41:57 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:41:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:41:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 01:41:57 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/16 01:42:02 Rebalance progress: 12.5
2021/12/16 01:42:07 Rebalance progress: 12.5
2021/12/16 01:42:12 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:42:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:42:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 01:42:12 Waiting for src bucket item count to get to: 50000 curr count: 49297
2021/12/16 01:42:17 Waiting for src bucket item count to get to: 50000 curr count: 49297
2021/12/16 01:42:27 Waiting for dst bucket item count to get to: 50000 curr count: 39967
2021/12/16 01:42:32 Waiting for dst bucket item count to get to: 50000 curr count: 39967
2021/12/16 01:42:32 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/16 01:42:32 Post SET ops. Expected item count: 50000 got 50000
2021/12/16 01:42:33 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 01:42:38 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/16 01:42:43 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/16 01:42:48 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/16 01:42:53 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/16 01:42:58 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/16 01:43:03 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/16 01:43:03 App: TestDuplicateTimerFiringKVSwapReb got un-deployed
2021/12/16 01:43:03 No eventing-con process running
2021/12/16 01:43:03 Function: TestDuplicateTimerFiringKVSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringKVSwapReb deleting in the background"
}
2021/12/16 01:43:05 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:43:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 01:43:15 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/16 01:43:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 01:43:16 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:43:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 01:43:26 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/16 01:43:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringKVSwapReb (169.95s)
=== RUN   TestKVRebOnUpdateBucketOpOneByOne
2021/12/16 01:43:36 Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 01:43:36 App: bucket_op_on_update got un-deployed
2021/12/16 01:43:36 No eventing-con process running
2021/12/16 01:43:36 Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/16 01:43:38 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:43:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 01:43:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 01:43:45 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:43:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 01:43:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:43:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 01:43:55 Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/16 01:43:56 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/16 01:44:01 Starting bucket ops to source bucket
2021/12/16 01:44:06 Waiting for app: bucket_op_on_update to get deployed
2021/12/16 01:44:11 Waiting for app: bucket_op_on_update to get deployed
2021/12/16 01:44:16 Waiting for app: bucket_op_on_update to get deployed
2021/12/16 01:44:16 App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:44:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:44:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 01:44:16 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/16 01:44:17 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/16 01:44:22 Rebalance progress: 12.5
2021/12/16 01:44:27 Rebalance progress: 12.5
2021/12/16 01:44:32 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:44:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:44:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 01:44:32 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/16 01:44:32 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/16 01:44:37 Rebalance progress: 12.5
2021/12/16 01:44:42 Rebalance progress: 12.5
2021/12/16 01:44:47 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:44:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:44:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 01:44:47 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: kv to the cluster
2021/12/16 01:44:48 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/16 01:44:53 Rebalance progress: 12.5
2021/12/16 01:44:58 Rebalance progress: 12.5
2021/12/16 01:45:03 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:45:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:45:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 01:45:03 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/16 01:45:08 Rebalance progress: 12.5
2021/12/16 01:45:13 Rebalance progress: 12.5
2021/12/16 01:45:18 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:45:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:45:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 01:45:18 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/16 01:45:23 Rebalance progress: 12.5
2021/12/16 01:45:28 Rebalance progress: 12.5
2021/12/16 01:45:33 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:45:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:45:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 01:45:33 clusterSetup.go:253: Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/16 01:45:38 Rebalance progress: 12.5
2021/12/16 01:45:43 Rebalance progress: 12.5
2021/12/16 01:45:48 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:45:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:45:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 01:45:48 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 01:45:53 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 01:45:58 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 01:46:03 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 01:46:08 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 01:46:13 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 01:46:13 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/16 01:46:13 eventingSetup.go:46: No eventing-con process running
2021/12/16 01:46:13 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/16 01:46:15 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:46:20 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 01:46:25 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 10668
2021/12/16 01:46:25 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 01:46:27 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:46:32 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 01:46:37 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 10668
2021/12/16 01:46:37 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebOnUpdateBucketOpOneByOne (190.61s)
=== RUN   TestKVForceFailover
2021/12/16 01:46:52 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 01:46:52 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/16 01:46:52 eventingSetup.go:46: No eventing-con process running
2021/12/16 01:46:52 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/16 01:46:54 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:46:59 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 01:46:59 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 01:47:01 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:47:06 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 01:47:06 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 01:47:06 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/16 01:47:06 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/16 01:47:11 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:47:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 01:47:11 Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/16 01:47:12 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/16 01:47:12 Starting bucket ops to source bucket
2021/12/16 01:47:17 Waiting for app: bucket_op_on_update to get deployed
2021/12/16 01:47:22 Waiting for app: bucket_op_on_update to get deployed
2021/12/16 01:47:27 Waiting for app: bucket_op_on_update to get deployed
2021/12/16 01:47:32 Waiting for app: bucket_op_on_update to get deployed
2021/12/16 01:47:32 App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:47:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:47:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 01:47:32 clusterSetup.go:229: Failing over: [127.0.0.1:19001]
2021/12/16 01:47:52 clusterSetup.go:237: Kicking off failover recovery, type: full
2021/12/16 01:47:52 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/16 01:47:58 Rebalance progress: 12.5
2021/12/16 01:48:03 Rebalance progress: 12.5
2021/12/16 01:48:08 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:48:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:48:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 01:48:08 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002] from the cluster
2021/12/16 01:48:13 Rebalance progress: 12.5
2021/12/16 01:48:18 Rebalance progress: 12.5
2021/12/16 01:48:23 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:48:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:48:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 01:48:23 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 01:48:28 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 01:48:33 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 01:48:38 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 01:48:43 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 01:48:48 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 01:48:53 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 01:48:53 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/16 01:48:53 eventingSetup.go:46: No eventing-con process running
2021/12/16 01:48:53 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/16 01:48:55 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:49:00 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 01:49:05 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7037
2021/12/16 01:49:05 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 01:49:07 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:49:12 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 01:49:17 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7037
2021/12/16 01:49:17 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVForceFailover (159.64s)
=== RUN   TestKVHardFailoverRecovery
2021/12/16 01:49:27 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 01:49:27 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/16 01:49:27 eventingSetup.go:46: No eventing-con process running
2021/12/16 01:49:27 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/16 01:49:28 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:49:33 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 01:49:34 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 01:49:35 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:49:40 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 01:49:40 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:49:45 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 01:49:45 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/16 01:49:46 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/16 01:49:51 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/16 01:49:56 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/16 01:50:01 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/16 01:50:06 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/16 01:50:06 clusterSetup.go:354: App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:50:06 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:50:06 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 01:50:06 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/16 01:50:07 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/16 01:50:12 Rebalance progress: 12.5
2021/12/16 01:50:17 Rebalance progress: 12.5
2021/12/16 01:50:22 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:50:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:50:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 01:50:22 clusterSetup.go:229: Failing over: [127.0.0.1:19001]
2021/12/16 01:51:22 clusterSetup.go:237: Kicking off failover recovery, type: full
2021/12/16 01:51:27 Rebalance progress: 12.5
2021/12/16 01:51:32 Rebalance progress: 12.5
2021/12/16 01:51:37 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:51:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:51:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 01:51:37 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/16 01:51:42 Rebalance progress: 12.5
2021/12/16 01:51:47 Rebalance progress: 12.5
2021/12/16 01:51:52 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:51:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:51:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 01:51:52 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 01:51:57 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 01:52:02 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 01:52:07 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 01:52:12 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 01:52:17 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 01:52:22 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 01:52:22 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/16 01:52:22 eventingSetup.go:46: No eventing-con process running
2021/12/16 01:52:23 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/16 01:52:24 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:52:29 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 01:52:34 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12071
2021/12/16 01:52:34 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 01:52:36 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:52:41 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 01:52:46 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12071
2021/12/16 01:52:46 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVHardFailoverRecovery (209.51s)
=== RUN   TestKVRebOnUpdateBucketOpAllAtOnce
2021/12/16 01:52:56 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 01:52:56 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/16 01:52:56 eventingSetup.go:46: No eventing-con process running
2021/12/16 01:52:56 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/16 01:52:58 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:53:03 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 01:53:03 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 01:53:05 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:53:10 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 01:53:10 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:53:15 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 01:53:15 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/16 01:53:16 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/16 01:53:21 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/16 01:53:26 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/16 01:53:31 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/16 01:53:36 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/16 01:53:36 clusterSetup.go:354: App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:53:36 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:53:36 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 01:53:36 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/16 01:53:36 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/16 01:53:37 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: kv to the cluster
2021/12/16 01:53:37 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/16 01:53:42 Rebalance progress: 12.5
2021/12/16 01:53:47 Rebalance progress: 12.5
2021/12/16 01:53:52 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:53:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:53:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 01:53:52 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002 127.0.0.1:19003] from the cluster
2021/12/16 01:53:58 Rebalance progress: 12.5
2021/12/16 01:54:03 Rebalance progress: 12.5
2021/12/16 01:54:08 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:54:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:54:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 01:54:08 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 01:54:13 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 01:54:18 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 01:54:23 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 01:54:28 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 01:54:33 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 01:54:33 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/16 01:54:33 eventingSetup.go:46: No eventing-con process running
2021/12/16 01:54:33 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/16 01:54:35 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:54:40 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 01:54:45 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4666
2021/12/16 01:54:45 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 01:54:46 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:54:51 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 01:54:56 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4666
2021/12/16 01:54:56 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebOnUpdateBucketOpAllAtOnce (130.34s)
=== RUN   TestKVSwapRebOnUpdateBucketOp
2021/12/16 01:55:06 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 01:55:07 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/16 01:55:07 eventingSetup.go:46: No eventing-con process running
2021/12/16 01:55:07 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/16 01:55:08 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:55:13 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 01:55:13 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 01:55:15 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:55:20 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 01:55:20 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:55:20 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 01:55:20 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/16 01:55:21 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/16 01:55:26 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/16 01:55:31 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/16 01:55:36 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/16 01:55:41 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/16 01:55:41 clusterSetup.go:354: App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:55:41 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:55:41 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 01:55:41 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/16 01:55:42 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/16 01:55:47 Rebalance progress: 12.5
2021/12/16 01:55:52 Rebalance progress: 12.5
2021/12/16 01:55:57 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:55:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:55:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 01:55:57 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/16 01:55:57 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/16 01:56:02 Rebalance progress: 12.5
2021/12/16 01:56:07 Rebalance progress: 12.5
2021/12/16 01:56:12 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:56:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:56:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 01:56:12 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/16 01:56:17 Rebalance progress: 12.5
2021/12/16 01:56:22 Rebalance progress: 12.5
2021/12/16 01:56:27 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:56:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:56:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 01:56:28 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 01:56:33 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 01:56:38 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 01:56:43 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 01:56:48 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 01:56:53 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 01:56:58 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 01:56:58 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/16 01:56:58 eventingSetup.go:46: No eventing-con process running
2021/12/16 01:56:58 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/16 01:57:00 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:57:05 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 01:57:05 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 01:57:06 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:57:11 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 01:57:16 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6114
2021/12/16 01:57:16 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 01:57:21 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 01:57:21 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/16 01:57:21 eventingSetup.go:46: No eventing-con process running
2021/12/16 01:57:21 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/16 01:57:23 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:57:28 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 01:57:28 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 01:57:30 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:57:35 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 01:57:35 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVSwapRebOnUpdateBucketOp (158.43s)
=== RUN   TestKVRebalanceWithMultipleHandlers
2021/12/16 01:57:45 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 01:57:45 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/16 01:57:45 eventingSetup.go:46: No eventing-con process running
2021/12/16 01:57:45 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/16 01:57:47 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:57:52 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 01:57:52 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 01:57:54 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:57:59 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 01:57:59 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 01:58:04 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/16 01:58:04 clusterSetup.go:405: App: sys_test_bucket_op got un-deployed
2021/12/16 01:58:04 eventingSetup.go:46: No eventing-con process running
2021/12/16 01:58:04 eventingSetup.go:418: Function: sys_test_bucket_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: sys_test_bucket_op not found"
 }
}
2021/12/16 01:58:05 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:58:10 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 01:58:10 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 01:58:12 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:58:17 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 01:58:17 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:58:17 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 01:58:17 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/16 01:58:18 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:58:18 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 01:58:18 eventingSetup.go:125: Deploying app: sys_test_bucket_op. Item count src bucket: 0 dst bucket: 0
2021/12/16 01:58:19 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'sys_test_bucket_op' in metakv",
  "warnings": null
 }
}
2021/12/16 01:58:24 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/16 01:58:29 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/16 01:58:34 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/16 01:58:39 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/16 01:58:39 clusterSetup.go:354: App: bucket_op_on_update got deployed
2021/12/16 01:58:44 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/16 01:58:49 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/16 01:58:54 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/16 01:58:54 clusterSetup.go:354: App: sys_test_bucket_op got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:58:54 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:58:54 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 01:58:55 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/16 01:58:55 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/16 01:59:00 Rebalance progress: 12.5
2021/12/16 01:59:05 Rebalance progress: 12.5
2021/12/16 01:59:10 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:59:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:59:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 01:59:10 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/16 01:59:10 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/16 01:59:15 Rebalance progress: 12.5
2021/12/16 01:59:20 Rebalance progress: 12.5
2021/12/16 01:59:25 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:59:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:59:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 01:59:26 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: kv to the cluster
2021/12/16 01:59:26 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/16 01:59:31 Rebalance progress: 12.5
2021/12/16 01:59:36 Rebalance progress: 12.5
2021/12/16 01:59:41 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:59:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:59:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 01:59:41 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/16 01:59:46 Rebalance progress: 12.5
2021/12/16 01:59:51 Rebalance progress: 12.5
2021/12/16 01:59:56 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:59:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 01:59:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 01:59:56 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/16 02:00:01 Rebalance progress: 12.5
2021/12/16 02:00:06 Rebalance progress: 12.5
2021/12/16 02:00:11 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 02:00:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 02:00:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 02:00:11 clusterSetup.go:253: Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/16 02:00:16 Rebalance progress: 12.5
2021/12/16 02:00:21 Rebalance progress: 12.5
2021/12/16 02:00:26 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 02:00:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 02:00:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 02:00:26 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 02:00:31 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 02:00:36 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 02:00:41 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 02:00:47 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 02:00:52 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 02:00:57 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 02:01:02 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 02:01:02 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/16 02:01:02 eventingSetup.go:46: No eventing-con process running
2021/12/16 02:01:02 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/16 02:01:05 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 02:01:10 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 02:01:15 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12164
2021/12/16 02:01:15 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 02:01:17 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 02:01:22 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 02:01:27 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12164
2021/12/16 02:01:27 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 02:01:27 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 02:01:32 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/16 02:01:37 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/16 02:01:42 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/16 02:01:47 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/16 02:01:52 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/16 02:01:57 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/16 02:01:57 clusterSetup.go:405: App: sys_test_bucket_op got un-deployed
2021/12/16 02:01:57 eventingSetup.go:46: No eventing-con process running
2021/12/16 02:01:57 eventingSetup.go:418: Function: sys_test_bucket_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: sys_test_bucket_op deleting in the background"
}
2021/12/16 02:01:59 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 02:02:04 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 02:02:04 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 02:02:06 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 02:02:11 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 02:02:11 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebalanceWithMultipleHandlers (275.68s)
=== RUN   TestKVSwapRebalanceWithMultipleHandlers
2021/12/16 02:02:21 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 02:02:21 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/16 02:02:21 eventingSetup.go:46: No eventing-con process running
2021/12/16 02:02:21 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/16 02:02:22 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 02:02:27 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 02:02:27 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 02:02:29 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 02:02:34 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 02:02:34 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 02:02:39 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/16 02:02:39 clusterSetup.go:405: App: sys_test_bucket_op got un-deployed
2021/12/16 02:02:39 eventingSetup.go:46: No eventing-con process running
2021/12/16 02:02:39 eventingSetup.go:418: Function: sys_test_bucket_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: sys_test_bucket_op not found"
 }
}
2021/12/16 02:02:41 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 02:02:46 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 02:02:46 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 02:02:48 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 02:02:53 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 02:02:53 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 02:02:53 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 02:02:53 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/16 02:02:54 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 02:02:54 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 02:02:54 eventingSetup.go:125: Deploying app: sys_test_bucket_op. Item count src bucket: 0 dst bucket: 0
2021/12/16 02:02:55 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'sys_test_bucket_op' in metakv",
  "warnings": null
 }
}
2021/12/16 02:03:00 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/16 02:03:05 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/16 02:03:10 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/16 02:03:15 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/16 02:03:15 clusterSetup.go:354: App: bucket_op_on_update got deployed
2021/12/16 02:03:20 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/16 02:03:25 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/16 02:03:30 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/16 02:03:30 clusterSetup.go:354: App: sys_test_bucket_op got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 02:03:30 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 02:03:30 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 02:03:30 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/16 02:03:30 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/16 02:03:35 Rebalance progress: 12.5
2021/12/16 02:03:40 Rebalance progress: 12.5
2021/12/16 02:03:45 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 02:03:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 02:03:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 02:03:45 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/16 02:03:46 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/16 02:03:51 Rebalance progress: 12.5
2021/12/16 02:03:56 Rebalance progress: 12.5
2021/12/16 02:04:01 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 02:04:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 02:04:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 02:04:01 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/16 02:04:06 Rebalance progress: 12.5
2021/12/16 02:04:11 Rebalance progress: 12.5
2021/12/16 02:04:16 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 02:04:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 02:04:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 02:04:16 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 02:04:21 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 02:04:26 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 02:04:31 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 02:04:36 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 02:04:41 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 02:04:46 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 02:04:46 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/16 02:04:46 eventingSetup.go:46: No eventing-con process running
2021/12/16 02:04:46 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/16 02:04:50 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 02:04:55 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 02:04:55 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 02:04:57 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 02:05:02 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 02:05:07 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7582
2021/12/16 02:05:07 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 02:05:07 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 02:05:12 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/16 02:05:17 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/16 02:05:22 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/16 02:05:27 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/16 02:05:32 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/16 02:05:37 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/16 02:05:37 clusterSetup.go:405: App: sys_test_bucket_op got un-deployed
2021/12/16 02:05:37 eventingSetup.go:46: No eventing-con process running
2021/12/16 02:05:37 eventingSetup.go:418: Function: sys_test_bucket_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: sys_test_bucket_op deleting in the background"
}
2021/12/16 02:05:39 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 02:05:44 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 02:05:44 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 02:05:46 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 02:05:51 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 02:05:51 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVSwapRebalanceWithMultipleHandlers (219.93s)
=== RUN   TestKVRebStopStartKVOpsOnUpdateBucketOpOneByOne
2021/12/16 02:06:01 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 02:06:01 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/16 02:06:01 eventingSetup.go:46: No eventing-con process running
2021/12/16 02:06:01 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/16 02:06:03 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 02:06:08 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 02:06:08 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 02:06:09 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 02:06:14 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 02:06:14 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 02:06:19 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 02:06:19 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/16 02:06:20 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/16 02:06:25 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/16 02:06:30 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/16 02:06:35 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/16 02:06:40 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/16 02:06:40 clusterSetup.go:354: App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 02:06:40 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 02:06:40 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 02:06:40 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/16 02:06:41 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/16 02:07:01 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 02:07:01 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 02:07:01 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 02:07:11 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 02:07:11 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 02:07:11 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/16 02:07:16 Rebalance progress: 12.5
2021/12/16 02:07:21 Rebalance progress: 12.5
2021/12/16 02:07:26 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 02:07:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 02:07:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 02:07:26 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/16 02:07:27 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/16 02:07:32 Rebalance progress: 12.5
2021/12/16 02:07:37 Rebalance progress: 12.5
2021/12/16 02:07:42 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 02:07:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 02:07:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 02:07:42 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/16 02:07:47 Rebalance progress: 12.5
2021/12/16 02:07:52 Rebalance progress: 12.5
2021/12/16 02:07:57 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 02:07:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 02:07:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 02:07:57 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 02:08:02 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 02:08:07 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 02:08:12 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 02:08:17 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 02:08:22 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 02:08:27 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 02:08:27 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/16 02:08:27 eventingSetup.go:46: No eventing-con process running
2021/12/16 02:08:27 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/16 02:08:29 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 02:08:34 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 02:08:39 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 9125
2021/12/16 02:08:39 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 02:08:41 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 02:08:46 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 02:08:51 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6370
2021/12/16 02:08:56 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6370
2021/12/16 02:08:56 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebStopStartKVOpsOnUpdateBucketOpOneByOne (185.11s)
=== RUN   TestKVFailoverOnUpdateBucketOp
2021/12/16 02:09:06 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 02:09:06 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/16 02:09:06 eventingSetup.go:46: No eventing-con process running
2021/12/16 02:09:06 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/16 02:09:07 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 02:09:12 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 02:09:12 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 02:09:14 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 02:09:19 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 02:09:19 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 02:09:24 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 02:09:24 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/16 02:09:25 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/16 02:09:30 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/16 02:09:35 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/16 02:09:40 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/16 02:09:45 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/16 02:09:45 clusterSetup.go:354: App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 02:09:45 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 02:09:45 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 02:09:45 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/16 02:09:46 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/16 02:09:51 Rebalance progress: 12.5
2021/12/16 02:09:56 Rebalance progress: 12.5
2021/12/16 02:10:01 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 02:10:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 02:10:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 02:10:01 clusterSetup.go:229: Failing over: [127.0.0.1:19001]
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 02:10:11 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 02:10:11 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 02:10:11 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/16 02:10:16 Rebalance progress: 12.5
2021/12/16 02:10:21 Rebalance progress: 12.5
2021/12/16 02:10:26 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 02:10:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 02:10:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 02:10:26 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/16 02:10:26 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/16 02:10:32 Rebalance progress: 12.5
2021/12/16 02:10:37 Rebalance progress: 12.5
2021/12/16 02:10:42 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 02:10:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 02:10:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 02:10:42 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/16 02:10:47 Rebalance progress: 12.5
2021/12/16 02:10:52 Rebalance progress: 12.5
2021/12/16 02:10:57 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 02:10:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 02:10:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 02:10:57 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 02:11:02 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 02:11:07 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 02:11:12 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 02:11:17 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 02:11:22 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 02:11:27 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 02:11:27 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/16 02:11:27 eventingSetup.go:46: No eventing-con process running
2021/12/16 02:11:27 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/16 02:11:29 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 02:11:34 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 02:11:39 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 8619
2021/12/16 02:11:39 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 02:11:41 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 02:11:46 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 02:11:51 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6242
2021/12/16 02:11:56 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6242
2021/12/16 02:11:56 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVFailoverOnUpdateBucketOp (180.07s)
=== RUN   TestBootstrapAfterKVHardFailover
2021/12/16 02:12:01 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/16 02:12:01 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/16 02:12:06 Rebalance progress: 100
2021/12/16 02:12:06 Failing over: [127.0.0.1:19001]
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 02:12:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 02:12:36 Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/16 02:12:37 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/16 02:12:42 Removing node(s): [] from the cluster
2021/12/16 02:12:42 Starting bucket ops to source bucket
2021/12/16 02:12:47 Rebalance failed. See logs for detailed reason. You can try again.
2021/12/16 02:12:52 Waiting for app: bucket_op_on_update to get deployed
2021/12/16 02:12:57 Waiting for app: bucket_op_on_update to get deployed
2021/12/16 02:12:57 App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 02:12:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 02:12:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 02:12:57 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 02:13:02 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 02:13:07 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 02:13:12 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 02:13:17 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 02:13:23 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 02:13:28 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 02:13:28 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/16 02:13:28 eventingSetup.go:46: No eventing-con process running
2021/12/16 02:13:28 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/16 02:13:29 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 02:13:34 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 02:13:34 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 02:13:36 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 02:13:41 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 02:13:46 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1503
2021/12/16 02:13:46 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestBootstrapAfterKVHardFailover (110.44s)
PASS
ok  	github.com/couchbase/eventing/tests/functional_tests	12083.833s

Rebalance tests took 3 hours, 21 minutes and 26 seconds.

Testing Succeeded


Version: versions-15.12.2021-18.29.cfg
Build Log: make-15.12.2021-18.29.log
Server Log: logs-15.12.2021-18.29.tar.gz

Finished