Building

Started building at 2021/12/03 11:39:19
Calculating base
Updating mirror
Basing run on 7.1.0-1832 0355326df
Updating tree for run 03.12.2021-11.39
gocb is at 635fa6c, changes since last good build: none
libcouchbase is at f6822fc, changes since last good build: none
indexing is at a00b35f, changes since last good build: none
kv_engine is at 7e709d9, changes since last good build: 
 7e709d911 MB-47300: Log open() errors at caller
 bf66aad32 MB-47300: Add CouchKVStore::logOpenError
 f0ebb1b64 MB-49036: Allow ReviveDocument to be used with durability
ns_server is at ef29789, changes since last good build: 
 ef297895c Merge branch 'cheshire-cat'
 88f038101 MB-49735: Merge branch 'mad-hatter' into cheshire-cat
 71a005f8f MB-49735: Fix incorrect return in ns_cluster:perform_actual_join
 7bd6bef9c Don't show 'nodes' key and 'unused' warning in GET /trustedCAs...
 e34a2a30c MB-49637: Treat self-generated CA and custom CA as trusted...
 c8b93da4f remove not needed module prefix
 c59652bf8 slightly simplify code in menelaus_users
 415da1a31 unblock replicator after menelaus_users:init is executed
 c3329dfa7 MB-34703 Add Framing info extras in mc_binary to impersonate user
 77d01b73d MB-49864: Prevent log trimming for logs in rebalance_observer
 cedfbcb6e fix typo
 0d78faf4b MB-49871: Rename kv data_size limit to data_size_mib
 7a4fa3654 MB-49619: Server groups are only supported on Enterprise Edition.
 56a400c22 Merge remote-tracking branch 'couchbase/mad-hatter'
 e08212c1f MB-49843:[BP] Set GODEBUG=madvdontneed=1 for indexer & projector
 7208d0198 MB-49620: Declaration of bare specifier for mn.pools.service
couchstore is at 05c4320, changes since last good build: none
couchdb is at fca2e6b, changes since last good build: none
testrunner is at 633e830, changes since last good build: 
 633e83017 CBQE-7352 add semi colon to separate command
 bcdc46ae7 CBQE-7352 delete binaries cached in suse vms.
 4312f2714 Add TC for n1ql js udf
 31db4f394 use https to add node in cli.
Switching eventing to unstable
eventing is at 9fce322, changes since last good build: 
 9fce322f MB-47385: Move eventing-producer and cbevent go version to 1.16.5
Switching eventing-ee to unstable
eventing-ee is at 7060ef8, changes since last good build: none
Pulling in uncommitted change 160397 at refs/changes/97/160397/3
Total 23 (delta 15), reused 17 (delta 15)
[unstable f044c325] MB-100 : Do not fail n1ql timeout test if we are catching timeout exceptions successfully
 Author: abhijpes 
 Date: Mon Aug 30 08:24:38 2021 +0530
 1 file changed, 2 insertions(+), 2 deletions(-)
Pulling in uncommitted change 163449 at refs/changes/49/163449/1
Total 48 (delta 27), reused 39 (delta 27)
[unstable 65df4fbb] MB-100: collect data when test fails
 Author: AnkitPrabhu 
 Date: Tue Oct 12 14:02:31 2021 +0530
 1 file changed, 2 insertions(+)
Pulling in uncommitted change 165976 at refs/changes/76/165976/16
Total 5 (delta 3), reused 5 (delta 3)
[unstable b636a868] MB-48538: Add new audit ids for different events
 Author: AnkitPrabhu 
 Date: Tue Nov 23 14:29:10 2021 +0530
 1 file changed, 337 insertions(+), 61 deletions(-)
Pulling in uncommitted change 166697 at refs/changes/97/166697/4
Total 10 (delta 9), reused 10 (delta 9)
[unstable 6a12df44] MB-49547: Check bucket instance to get kv node for dcp request
 Author: AnkitPrabhu 
 Date: Thu Dec 2 12:22:17 2021 +0530
 6 files changed, 38 insertions(+), 109 deletions(-)
Pulling in uncommitted change 166698 at refs/changes/98/166698/3
Total 4 (delta 3), reused 4 (delta 3)
[unstable 644a8b22] MB-48334 : Fail rebalance unless all functions deployed on current encryption level
 Author: abhijpes 
 Date: Wed Dec 1 14:51:29 2021 +0530
 1 file changed, 9 insertions(+)
Building cmakefiles and deps
Building main product

Testing

Started testing at 2021/12/03 11:55:29
Testing mode: rebalance,n1ql,functional
Starting server
Starting server: attempt 1

Functional tests

2021/12/03 11:57:09 Started functional
2021/12/03 11:57:20 Adding node: https://127.0.0.1:19001 with role: kv,index,n1ql to the cluster
2021/12/03 11:57:33 Error in adding nodes : , response: {"otpNode":"n_1@127.0.0.1"}
2021/12/03 11:57:33 Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/03 11:57:38 Removing node(s): [] from the cluster
2021/12/03 11:57:43 Rebalance progress: 2.604166666666667
2021/12/03 11:57:48 Rebalance progress: 7.161458333333333
2021/12/03 11:57:53 Rebalance progress: 10.41666666666667
2021/12/03 11:57:58 Rebalance progress: 14.95587384259259
2021/12/03 11:58:03 Rebalance progress: 18.51851851851852
2021/12/03 11:58:08 Rebalance progress: 22.49710648148148
2021/12/03 11:58:13 Rebalance progress: 27.00014467592592
2021/12/03 11:58:18 Rebalance progress: 100
=== RUN   TestAdvancedGetOps
2021/12/03 11:58:24 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/03 11:58:39 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/03 11:58:54 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/03 11:59:09 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/03 11:59:24 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/03 11:59:39 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/03 11:59:54 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/03 12:00:09 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/03 12:00:24 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/03 12:00:39 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/03 12:00:54 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/03 12:01:09 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/03 12:01:24 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/03 12:01:39 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/03 12:01:54 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/03 12:02:09 Deploying app: TestAdvancedGetOps. Item count src bucket: 0 dst bucket: 0
2021/12/03 12:02:10 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedGetOps' in metakv",
  "warnings": null
 }
}
2021/12/03 12:02:15 Waiting for app: TestAdvancedGetOps to get deployed
2021/12/03 12:02:20 Waiting for app: TestAdvancedGetOps to get deployed
2021/12/03 12:02:25 Waiting for app: TestAdvancedGetOps to get deployed
2021/12/03 12:02:30 Waiting for app: TestAdvancedGetOps to get deployed
2021/12/03 12:02:30 App: TestAdvancedGetOps got deployed
2021/12/03 12:02:30 Starting bucket ops to source bucket
2021/12/03 12:02:35 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/03 12:02:40 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/03 12:02:45 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/03 12:02:45 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/03 12:02:45 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 12:02:45 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/03 12:02:45 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 12:02:50 Waiting for app: TestAdvancedGetOps to get un-deployed
2021/12/03 12:02:55 Waiting for app: TestAdvancedGetOps to get un-deployed
2021/12/03 12:03:00 Waiting for app: TestAdvancedGetOps to get un-deployed
2021/12/03 12:03:05 Waiting for app: TestAdvancedGetOps to get un-deployed
2021/12/03 12:03:05 App: TestAdvancedGetOps got un-deployed
2021/12/03 12:03:05 No eventing-con process running
2021/12/03 12:03:05 Function: TestAdvancedGetOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedGetOps deleting in the background"
}
2021/12/03 12:03:07 Context: Bucket flush, response code: 200 dump: 
2021/12/03 12:03:17 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/03 12:03:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 12:03:19 Context: Bucket flush, response code: 200 dump: 
2021/12/03 12:03:29 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/03 12:03:34 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/03 12:03:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedGetOps (310.33s)
=== RUN   TestAdvancedInsertOps
2021/12/03 12:03:34 Deploying app: TestAdvancedInsertOps. Item count src bucket: 0 dst bucket: 0
2021/12/03 12:03:35 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedInsertOps' in metakv",
  "warnings": null
 }
}
2021/12/03 12:03:40 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/03 12:03:45 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/03 12:03:50 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/03 12:03:55 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/03 12:03:55 App: TestAdvancedInsertOps got deployed
2021/12/03 12:03:55 Starting bucket ops to source bucket
2021/12/03 12:04:01 Waiting for dst bucket item count to get to: 1024 curr count: 0
2021/12/03 12:04:01 src & dst bucket item count matched up. src bucket count: 1024 dst bucket count: 1024
2021/12/03 12:04:01 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 12:04:01 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/03 12:04:01 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 12:04:06 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/03 12:04:11 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/03 12:04:16 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/03 12:04:21 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/03 12:04:21 App: TestAdvancedInsertOps got un-deployed
2021/12/03 12:04:21 No eventing-con process running
2021/12/03 12:04:21 Function: TestAdvancedInsertOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedInsertOps deleting in the background"
}
2021/12/03 12:04:23 Context: Bucket flush, response code: 200 dump: 
2021/12/03 12:04:33 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/03 12:04:38 Waiting for dst bucket item count to get to: 0 curr count: 512
2021/12/03 12:04:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 12:04:40 Context: Bucket flush, response code: 200 dump: 
2021/12/03 12:04:50 Waiting for dst bucket item count to get to: 0 curr count: 512
2021/12/03 12:04:55 Waiting for dst bucket item count to get to: 0 curr count: 512
2021/12/03 12:04:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 12:04:55 Testing insert operation on source bucket
2021/12/03 12:04:55 Deploying app: TestAdvancedInsertOps. Item count src bucket: 0 dst bucket: 0
2021/12/03 12:04:56 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedInsertOps' in metakv",
  "warnings": null
 }
}
2021/12/03 12:05:01 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/03 12:05:06 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/03 12:05:11 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/03 12:05:16 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/03 12:05:16 App: TestAdvancedInsertOps got deployed
2021/12/03 12:05:16 Starting bucket ops to source bucket
2021/12/03 12:05:22 Waiting for dst bucket item count to get to: 2048 curr count: 0
2021/12/03 12:05:22 src & dst bucket item count matched up. src bucket count: 2048 dst bucket count: 2048
2021/12/03 12:05:22 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 12:05:22 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/03 12:05:22 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 12:05:27 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/03 12:05:32 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/03 12:05:37 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/03 12:05:42 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/03 12:05:42 App: TestAdvancedInsertOps got un-deployed
2021/12/03 12:05:42 No eventing-con process running
2021/12/03 12:05:42 Function: TestAdvancedInsertOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedInsertOps deleting in the background"
}
2021/12/03 12:05:44 Context: Bucket flush, response code: 200 dump: 
2021/12/03 12:05:54 Waiting for dst bucket item count to get to: 0 curr count: 2048
2021/12/03 12:05:59 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/03 12:05:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 12:06:01 Context: Bucket flush, response code: 200 dump: 
2021/12/03 12:06:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedInsertOps (151.79s)
=== RUN   TestAdvancedUpsertOps
2021/12/03 12:06:06 Starting bucket ops to source bucket
2021/12/03 12:06:06 Deploying app: TestAdvancedUpsertOps. Item count src bucket: 0 dst bucket: 0
2021/12/03 12:06:08 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedUpsertOps' in metakv",
  "warnings": null
 }
}
2021/12/03 12:06:13 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/03 12:06:18 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/03 12:06:23 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/03 12:06:28 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/03 12:06:28 App: TestAdvancedUpsertOps got deployed
2021/12/03 12:06:28 Starting bucket ops to source bucket
2021/12/03 12:06:34 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/03 12:06:39 Waiting for dst bucket item count to get to: 0 curr count: 512
2021/12/03 12:06:44 Waiting for dst bucket item count to get to: 0 curr count: 512
2021/12/03 12:06:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 12:06:44 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 12:06:44 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/03 12:06:44 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 12:06:49 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/03 12:06:54 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/03 12:06:59 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/03 12:07:04 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/03 12:07:04 App: TestAdvancedUpsertOps got un-deployed
2021/12/03 12:07:04 No eventing-con process running
2021/12/03 12:07:04 Function: TestAdvancedUpsertOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedUpsertOps deleting in the background"
}
2021/12/03 12:07:05 Context: Bucket flush, response code: 200 dump: 
2021/12/03 12:07:16 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/03 12:07:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 12:07:17 Context: Bucket flush, response code: 200 dump: 
2021/12/03 12:07:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 12:07:22 Testing upsert operation on source bucket
2021/12/03 12:07:22 Deploying app: TestAdvancedUpsertOps. Item count src bucket: 0 dst bucket: 0
2021/12/03 12:07:24 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedUpsertOps' in metakv",
  "warnings": null
 }
}
2021/12/03 12:07:29 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/03 12:07:34 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/03 12:07:39 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/03 12:07:44 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/03 12:07:44 App: TestAdvancedUpsertOps got deployed
2021/12/03 12:07:44 Starting bucket ops to source bucket
2021/12/03 12:07:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 12:07:45 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 12:07:45 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/03 12:07:45 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 12:07:50 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/03 12:07:55 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/03 12:08:00 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/03 12:08:05 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/03 12:08:05 App: TestAdvancedUpsertOps got un-deployed
2021/12/03 12:08:05 No eventing-con process running
2021/12/03 12:08:05 Function: TestAdvancedUpsertOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedUpsertOps deleting in the background"
}
2021/12/03 12:08:07 Context: Bucket flush, response code: 200 dump: 
2021/12/03 12:08:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 12:08:14 Context: Bucket flush, response code: 200 dump: 
2021/12/03 12:08:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedUpsertOps (133.23s)
=== RUN   TestAdvancedReplaceOps
2021/12/03 12:08:19 Starting bucket ops to source bucket
2021/12/03 12:08:20 Deploying app: TestAdvancedReplaceOps. Item count src bucket: 0 dst bucket: 0
2021/12/03 12:08:21 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedReplaceOps' in metakv",
  "warnings": null
 }
}
2021/12/03 12:08:26 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/03 12:08:31 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/03 12:08:36 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/03 12:08:41 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/03 12:08:41 App: TestAdvancedReplaceOps got deployed
2021/12/03 12:08:41 Starting bucket ops to source bucket
2021/12/03 12:08:47 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/03 12:08:52 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/03 12:08:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 12:08:52 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 12:08:52 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/03 12:08:52 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 12:08:57 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/03 12:09:02 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/03 12:09:07 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/03 12:09:12 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/03 12:09:12 App: TestAdvancedReplaceOps got un-deployed
2021/12/03 12:09:12 No eventing-con process running
2021/12/03 12:09:12 Function: TestAdvancedReplaceOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedReplaceOps deleting in the background"
}
2021/12/03 12:09:14 Context: Bucket flush, response code: 200 dump: 
2021/12/03 12:09:24 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/03 12:09:29 Waiting for dst bucket item count to get to: 0 curr count: 512
2021/12/03 12:09:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 12:09:31 Context: Bucket flush, response code: 200 dump: 
2021/12/03 12:09:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 12:09:36 Testing upsert operation on source bucket
2021/12/03 12:09:36 Deploying app: TestAdvancedReplaceOps. Item count src bucket: 0 dst bucket: 0
2021/12/03 12:09:37 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedReplaceOps' in metakv",
  "warnings": null
 }
}
2021/12/03 12:09:42 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/03 12:09:47 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/03 12:09:52 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/03 12:09:57 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/03 12:09:57 App: TestAdvancedReplaceOps got deployed
2021/12/03 12:09:57 Starting bucket ops to source bucket
2021/12/03 12:09:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 12:09:58 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 12:09:58 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/03 12:09:58 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 12:10:03 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/03 12:10:09 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/03 12:10:14 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/03 12:10:19 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/03 12:10:19 App: TestAdvancedReplaceOps got un-deployed
2021/12/03 12:10:19 No eventing-con process running
2021/12/03 12:10:19 Function: TestAdvancedReplaceOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedReplaceOps deleting in the background"
}
2021/12/03 12:10:20 Context: Bucket flush, response code: 200 dump: 
2021/12/03 12:10:30 Waiting for dst bucket item count to get to: 0 curr count: 160
2021/12/03 12:10:35 Waiting for dst bucket item count to get to: 0 curr count: 160
2021/12/03 12:10:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 12:10:37 Context: Bucket flush, response code: 200 dump: 
2021/12/03 12:10:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedReplaceOps (143.09s)
=== RUN   TestAdvancedDeleteOps
2021/12/03 12:10:42 Starting bucket ops to source bucket
2021/12/03 12:10:42 Deploying app: TestAdvancedDeleteOps. Item count src bucket: 0 dst bucket: 0
2021/12/03 12:10:43 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedDeleteOps' in metakv",
  "warnings": null
 }
}
2021/12/03 12:10:48 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/03 12:10:53 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/03 12:10:58 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/03 12:11:03 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/03 12:11:03 App: TestAdvancedDeleteOps got deployed
2021/12/03 12:11:03 Starting bucket ops to source bucket
2021/12/03 12:11:09 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/03 12:11:14 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/03 12:11:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 12:11:14 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 12:11:14 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/03 12:11:14 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 12:11:19 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/03 12:11:24 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/03 12:11:29 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/03 12:11:34 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/03 12:11:34 App: TestAdvancedDeleteOps got un-deployed
2021/12/03 12:11:34 No eventing-con process running
2021/12/03 12:11:34 Function: TestAdvancedDeleteOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedDeleteOps deleting in the background"
}
2021/12/03 12:11:35 Context: Bucket flush, response code: 200 dump: 
2021/12/03 12:11:46 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/03 12:11:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 12:11:47 Context: Bucket flush, response code: 200 dump: 
2021/12/03 12:11:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 12:11:52 Testing delete operation on source bucket
2021/12/03 12:11:52 Deploying app: TestAdvancedDeleteOps. Item count src bucket: 0 dst bucket: 0
2021/12/03 12:11:53 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedDeleteOps' in metakv",
  "warnings": null
 }
}
2021/12/03 12:11:58 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/03 12:12:03 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/03 12:12:09 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/03 12:12:14 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/03 12:12:14 App: TestAdvancedDeleteOps got deployed
2021/12/03 12:12:14 Starting bucket ops to source bucket
2021/12/03 12:12:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 12:12:14 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 12:12:14 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/03 12:12:14 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 12:12:19 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/03 12:12:24 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/03 12:12:29 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/03 12:12:34 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/03 12:12:34 App: TestAdvancedDeleteOps got un-deployed
2021/12/03 12:12:34 No eventing-con process running
2021/12/03 12:12:34 Function: TestAdvancedDeleteOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedDeleteOps deleting in the background"
}
2021/12/03 12:12:36 Context: Bucket flush, response code: 200 dump: 
2021/12/03 12:12:41 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 12:12:43 Context: Bucket flush, response code: 200 dump: 
2021/12/03 12:12:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedDeleteOps (125.90s)
=== RUN   TestEnoentAdvancedGet
2021/12/03 12:12:48 Deploying app: TestEnoentAdvancedGet. Item count src bucket: 0 dst bucket: 0
2021/12/03 12:12:49 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedGet' in metakv",
  "warnings": null
 }
}
2021/12/03 12:12:54 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/03 12:12:59 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/03 12:13:04 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/03 12:13:09 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/03 12:13:09 App: TestEnoentAdvancedGet got deployed
2021/12/03 12:13:09 Starting bucket ops to source bucket
2021/12/03 12:13:14 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/03 12:13:19 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/03 12:13:19 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/03 12:13:19 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 12:13:19 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/03 12:13:19 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 12:13:24 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/03 12:13:29 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/03 12:13:34 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/03 12:13:39 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/03 12:13:39 App: TestEnoentAdvancedGet got un-deployed
2021/12/03 12:13:39 No eventing-con process running
2021/12/03 12:13:39 Function: TestEnoentAdvancedGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedGet deleting in the background"
}
2021/12/03 12:13:41 Context: Bucket flush, response code: 200 dump: 
2021/12/03 12:13:51 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/03 12:13:56 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/03 12:13:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 12:13:58 Context: Bucket flush, response code: 200 dump: 
2021/12/03 12:14:08 Waiting for dst bucket item count to get to: 0 curr count: 71
2021/12/03 12:14:13 Waiting for dst bucket item count to get to: 0 curr count: 21
2021/12/03 12:14:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 12:14:13 Testing get enoent operation on source bucket
2021/12/03 12:14:13 Deploying app: TestEnoentAdvancedGet. Item count src bucket: 0 dst bucket: 0
2021/12/03 12:14:14 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedGet' in metakv",
  "warnings": null
 }
}
2021/12/03 12:14:19 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/03 12:14:24 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/03 12:14:29 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/03 12:14:34 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/03 12:14:34 App: TestEnoentAdvancedGet got deployed
2021/12/03 12:14:34 Starting bucket ops to source bucket
2021/12/03 12:14:39 Waiting for dst bucket item count to get to: 200 curr count: 0
2021/12/03 12:14:44 Waiting for dst bucket item count to get to: 200 curr count: 100
2021/12/03 12:14:44 src & dst bucket item count matched up. src bucket count: 200 dst bucket count: 200
2021/12/03 12:14:44 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 12:14:44 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/03 12:14:44 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 12:14:49 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/03 12:14:54 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/03 12:14:59 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/03 12:15:04 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/03 12:15:04 App: TestEnoentAdvancedGet got un-deployed
2021/12/03 12:15:04 No eventing-con process running
2021/12/03 12:15:05 Function: TestEnoentAdvancedGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedGet deleting in the background"
}
2021/12/03 12:15:06 Context: Bucket flush, response code: 200 dump: 
2021/12/03 12:15:16 Waiting for dst bucket item count to get to: 0 curr count: 200
2021/12/03 12:15:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 12:15:18 Context: Bucket flush, response code: 200 dump: 
2021/12/03 12:15:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEnoentAdvancedGet (155.11s)
=== RUN   TestEnoentAdvancedReplace
2021/12/03 12:15:23 Deploying app: TestEnoentAdvancedReplace. Item count src bucket: 0 dst bucket: 0
2021/12/03 12:15:24 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedReplace' in metakv",
  "warnings": null
 }
}
2021/12/03 12:15:29 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/03 12:15:34 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/03 12:15:39 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/03 12:15:44 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/03 12:15:44 App: TestEnoentAdvancedReplace got deployed
2021/12/03 12:15:44 Starting bucket ops to source bucket
2021/12/03 12:15:49 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/03 12:15:54 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/03 12:15:54 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/03 12:15:54 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 12:15:54 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/03 12:15:54 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 12:15:59 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/03 12:16:05 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/03 12:16:10 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/03 12:16:15 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/03 12:16:15 App: TestEnoentAdvancedReplace got un-deployed
2021/12/03 12:16:15 No eventing-con process running
2021/12/03 12:16:15 Function: TestEnoentAdvancedReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedReplace deleting in the background"
}
2021/12/03 12:16:16 Context: Bucket flush, response code: 200 dump: 
2021/12/03 12:16:26 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/03 12:16:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 12:16:28 Context: Bucket flush, response code: 200 dump: 
2021/12/03 12:16:38 Waiting for dst bucket item count to get to: 0 curr count: 75
2021/12/03 12:16:43 Waiting for dst bucket item count to get to: 0 curr count: 25
2021/12/03 12:16:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 12:16:43 Testing get enoent operation on source bucket
2021/12/03 12:16:43 Deploying app: TestEnoentAdvancedReplace. Item count src bucket: 0 dst bucket: 0
2021/12/03 12:16:44 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedReplace' in metakv",
  "warnings": null
 }
}
2021/12/03 12:16:49 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/03 12:16:54 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/03 12:16:59 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/03 12:17:04 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/03 12:17:04 App: TestEnoentAdvancedReplace got deployed
2021/12/03 12:17:04 Starting bucket ops to source bucket
2021/12/03 12:17:09 Waiting for dst bucket item count to get to: 200 curr count: 0
2021/12/03 12:17:15 Waiting for dst bucket item count to get to: 200 curr count: 100
2021/12/03 12:17:15 src & dst bucket item count matched up. src bucket count: 200 dst bucket count: 200
2021/12/03 12:17:15 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 12:17:15 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/03 12:17:15 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 12:17:20 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/03 12:17:25 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/03 12:17:30 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/03 12:17:35 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/03 12:17:35 App: TestEnoentAdvancedReplace got un-deployed
2021/12/03 12:17:35 No eventing-con process running
2021/12/03 12:17:35 Function: TestEnoentAdvancedReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedReplace deleting in the background"
}
2021/12/03 12:17:36 Context: Bucket flush, response code: 200 dump: 
2021/12/03 12:17:46 Waiting for dst bucket item count to get to: 0 curr count: 200
2021/12/03 12:17:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 12:17:48 Context: Bucket flush, response code: 200 dump: 
2021/12/03 12:17:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEnoentAdvancedReplace (150.08s)
=== RUN   TestEnoentAdvancedDelete
2021/12/03 12:17:53 Deploying app: TestEnoentAdvancedDelete. Item count src bucket: 0 dst bucket: 0
2021/12/03 12:17:54 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedDelete' in metakv",
  "warnings": null
 }
}
2021/12/03 12:17:59 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/03 12:18:04 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/03 12:18:09 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/03 12:18:14 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/03 12:18:14 App: TestEnoentAdvancedDelete got deployed
2021/12/03 12:18:14 Starting bucket ops to source bucket
2021/12/03 12:18:20 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/03 12:18:25 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/03 12:18:25 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/03 12:18:25 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 12:18:25 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/03 12:18:25 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 12:18:30 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/03 12:18:35 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/03 12:18:40 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/03 12:18:45 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/03 12:18:45 App: TestEnoentAdvancedDelete got un-deployed
2021/12/03 12:18:45 No eventing-con process running
2021/12/03 12:18:45 Function: TestEnoentAdvancedDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedDelete deleting in the background"
}
2021/12/03 12:18:46 Context: Bucket flush, response code: 200 dump: 
2021/12/03 12:18:56 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/03 12:18:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 12:18:58 Context: Bucket flush, response code: 200 dump: 
2021/12/03 12:19:08 Waiting for dst bucket item count to get to: 0 curr count: 78
2021/12/03 12:19:13 Waiting for dst bucket item count to get to: 0 curr count: 28
2021/12/03 12:19:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 12:19:13 Testing get enoent operation on source bucket
2021/12/03 12:19:13 Deploying app: TestEnoentAdvancedDelete. Item count src bucket: 0 dst bucket: 0
2021/12/03 12:19:14 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedDelete' in metakv",
  "warnings": null
 }
}
2021/12/03 12:19:19 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/03 12:19:24 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/03 12:19:29 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/03 12:19:34 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/03 12:19:34 App: TestEnoentAdvancedDelete got deployed
2021/12/03 12:19:34 Starting bucket ops to source bucket
2021/12/03 12:19:40 Waiting for dst bucket item count to get to: 200 curr count: 0
2021/12/03 12:19:45 Waiting for dst bucket item count to get to: 200 curr count: 100
2021/12/03 12:19:45 src & dst bucket item count matched up. src bucket count: 200 dst bucket count: 200
2021/12/03 12:19:45 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 12:19:45 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/03 12:19:45 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 12:19:50 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/03 12:19:55 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/03 12:20:00 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/03 12:20:05 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/03 12:20:05 App: TestEnoentAdvancedDelete got un-deployed
2021/12/03 12:20:05 No eventing-con process running
2021/12/03 12:20:05 Function: TestEnoentAdvancedDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedDelete deleting in the background"
}
2021/12/03 12:20:06 Context: Bucket flush, response code: 200 dump: 
2021/12/03 12:20:17 Waiting for dst bucket item count to get to: 0 curr count: 200
2021/12/03 12:20:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 12:20:18 Context: Bucket flush, response code: 200 dump: 
2021/12/03 12:20:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEnoentAdvancedDelete (150.16s)
=== RUN   TestAdvancedInsertKeyExist
2021/12/03 12:20:23 Deploying app: TestAdvancedInsertKeyExist. Item count src bucket: 0 dst bucket: 0
2021/12/03 12:20:25 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedInsertKeyExist' in metakv",
  "warnings": null
 }
}
2021/12/03 12:20:30 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/03 12:20:35 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/03 12:20:40 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/03 12:20:45 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/03 12:20:45 App: TestAdvancedInsertKeyExist got deployed
2021/12/03 12:20:45 Starting bucket ops to source bucket
2021/12/03 12:20:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 12:20:45 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 12:20:45 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/03 12:20:45 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 12:20:50 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/03 12:20:55 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/03 12:21:00 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/03 12:21:05 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/03 12:21:05 App: TestAdvancedInsertKeyExist got un-deployed
2021/12/03 12:21:05 No eventing-con process running
2021/12/03 12:21:05 Function: TestAdvancedInsertKeyExist delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedInsertKeyExist deleting in the background"
}
2021/12/03 12:21:07 Context: Bucket flush, response code: 200 dump: 
2021/12/03 12:21:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 12:21:14 Context: Bucket flush, response code: 200 dump: 
2021/12/03 12:21:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 12:21:19 Pumping document in the destination bucket
2021/12/03 12:21:19 Starting bucket ops to source bucket
2021/12/03 12:21:19 Deploying app: TestAdvancedInsertKeyExist. Item count src bucket: 0 dst bucket: 0
2021/12/03 12:21:20 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedInsertKeyExist' in metakv",
  "warnings": null
 }
}
2021/12/03 12:21:25 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/03 12:21:30 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/03 12:21:35 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/03 12:21:40 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/03 12:21:40 App: TestAdvancedInsertKeyExist got deployed
2021/12/03 12:21:40 Starting bucket ops to source bucket
2021/12/03 12:21:45 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/03 12:21:50 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/03 12:21:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 12:21:50 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 12:21:50 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/03 12:21:50 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 12:21:55 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/03 12:22:00 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/03 12:22:05 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/03 12:22:11 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/03 12:22:11 App: TestAdvancedInsertKeyExist got un-deployed
2021/12/03 12:22:11 No eventing-con process running
2021/12/03 12:22:11 Function: TestAdvancedInsertKeyExist delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedInsertKeyExist deleting in the background"
}
2021/12/03 12:22:12 Context: Bucket flush, response code: 200 dump: 
2021/12/03 12:22:22 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/03 12:22:27 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/03 12:22:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 12:22:29 Context: Bucket flush, response code: 200 dump: 
2021/12/03 12:22:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedInsertKeyExist (130.76s)
=== RUN   TestExipryGet
2021/12/03 12:22:34 Deploying app: TestExipryGet. Item count src bucket: 0 dst bucket: 0
2021/12/03 12:22:35 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryGet' in metakv",
  "warnings": null
 }
}
2021/12/03 12:22:40 Waiting for app: TestExipryGet to get deployed
2021/12/03 12:22:45 Waiting for app: TestExipryGet to get deployed
2021/12/03 12:22:50 Waiting for app: TestExipryGet to get deployed
2021/12/03 12:22:55 Waiting for app: TestExipryGet to get deployed
2021/12/03 12:22:55 App: TestExipryGet got deployed
2021/12/03 12:22:55 Starting bucket ops to source bucket
2021/12/03 12:23:41 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/03 12:23:41 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 12:23:41 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 12:23:41 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/03 12:23:41 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 12:23:46 Waiting for app: TestExipryGet to get un-deployed
2021/12/03 12:23:51 Waiting for app: TestExipryGet to get un-deployed
2021/12/03 12:23:56 Waiting for app: TestExipryGet to get un-deployed
2021/12/03 12:24:01 Waiting for app: TestExipryGet to get un-deployed
2021/12/03 12:24:01 App: TestExipryGet got un-deployed
2021/12/03 12:24:01 No eventing-con process running
2021/12/03 12:24:01 Function: TestExipryGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryGet deleting in the background"
}
2021/12/03 12:24:02 Context: Bucket flush, response code: 200 dump: 
2021/12/03 12:24:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 12:24:10 Context: Bucket flush, response code: 200 dump: 
2021/12/03 12:24:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 12:24:15 Pumping document in the destination bucket for expiry get
2021/12/03 12:24:15 Starting bucket ops to source bucket
2021/12/03 12:24:15 Deploying app: TestExipryGet. Item count src bucket: 0 dst bucket: 0
2021/12/03 12:24:16 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryGet' in metakv",
  "warnings": null
 }
}
2021/12/03 12:24:21 Waiting for app: TestExipryGet to get deployed
2021/12/03 12:24:26 Waiting for app: TestExipryGet to get deployed
2021/12/03 12:24:31 Waiting for app: TestExipryGet to get deployed
2021/12/03 12:24:36 Waiting for app: TestExipryGet to get deployed
2021/12/03 12:24:36 App: TestExipryGet got deployed
2021/12/03 12:24:36 Starting bucket ops to source bucket
2021/12/03 12:25:21 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/03 12:25:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 12:25:21 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 12:25:21 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/03 12:25:21 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 12:25:26 Waiting for app: TestExipryGet to get un-deployed
2021/12/03 12:25:32 Waiting for app: TestExipryGet to get un-deployed
2021/12/03 12:25:37 Waiting for app: TestExipryGet to get un-deployed
2021/12/03 12:25:42 Waiting for app: TestExipryGet to get un-deployed
2021/12/03 12:25:42 App: TestExipryGet got un-deployed
2021/12/03 12:25:42 No eventing-con process running
2021/12/03 12:25:42 Function: TestExipryGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryGet deleting in the background"
}
2021/12/03 12:25:43 Context: Bucket flush, response code: 200 dump: 
2021/12/03 12:25:53 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/03 12:25:58 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/03 12:25:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 12:26:00 Context: Bucket flush, response code: 200 dump: 
2021/12/03 12:26:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestExipryGet (210.99s)
=== RUN   TestExipryInsert
2021/12/03 12:26:05 Deploying app: TestExipryInsert. Item count src bucket: 0 dst bucket: 0
2021/12/03 12:26:06 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryInsert' in metakv",
  "warnings": null
 }
}
2021/12/03 12:26:11 Waiting for app: TestExipryInsert to get deployed
2021/12/03 12:26:16 Waiting for app: TestExipryInsert to get deployed
2021/12/03 12:26:21 Waiting for app: TestExipryInsert to get deployed
2021/12/03 12:26:26 Waiting for app: TestExipryInsert to get deployed
2021/12/03 12:26:26 App: TestExipryInsert got deployed
2021/12/03 12:26:26 Starting bucket ops to source bucket
2021/12/03 12:27:12 Waiting for dst bucket item count to get to: 0 curr count: 200
2021/12/03 12:27:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 12:27:12 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 12:27:12 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/03 12:27:12 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 12:27:17 Waiting for app: TestExipryInsert to get un-deployed
2021/12/03 12:27:22 Waiting for app: TestExipryInsert to get un-deployed
2021/12/03 12:27:27 Waiting for app: TestExipryInsert to get un-deployed
2021/12/03 12:27:32 Waiting for app: TestExipryInsert to get un-deployed
2021/12/03 12:27:32 App: TestExipryInsert got un-deployed
2021/12/03 12:27:32 No eventing-con process running
2021/12/03 12:27:32 Function: TestExipryInsert delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryInsert deleting in the background"
}
2021/12/03 12:27:34 Context: Bucket flush, response code: 200 dump: 
2021/12/03 12:27:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 12:27:41 Context: Bucket flush, response code: 200 dump: 
2021/12/03 12:27:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 12:27:46 Testing on the destination bucket for expiry insert
2021/12/03 12:27:46 Deploying app: TestExipryInsert. Item count src bucket: 0 dst bucket: 0
2021/12/03 12:27:47 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryInsert' in metakv",
  "warnings": null
 }
}
2021/12/03 12:27:52 Waiting for app: TestExipryInsert to get deployed
2021/12/03 12:27:57 Waiting for app: TestExipryInsert to get deployed
2021/12/03 12:28:02 Waiting for app: TestExipryInsert to get deployed
2021/12/03 12:28:07 Waiting for app: TestExipryInsert to get deployed
2021/12/03 12:28:07 App: TestExipryInsert got deployed
2021/12/03 12:28:07 Starting bucket ops to source bucket
2021/12/03 12:28:52 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/03 12:28:57 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/03 12:29:02 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/03 12:29:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 12:29:02 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 12:29:02 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/03 12:29:02 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 12:29:07 Waiting for app: TestExipryInsert to get un-deployed
2021/12/03 12:29:12 Waiting for app: TestExipryInsert to get un-deployed
2021/12/03 12:29:17 Waiting for app: TestExipryInsert to get un-deployed
2021/12/03 12:29:22 Waiting for app: TestExipryInsert to get un-deployed
2021/12/03 12:29:22 App: TestExipryInsert got un-deployed
2021/12/03 12:29:22 No eventing-con process running
2021/12/03 12:29:22 Function: TestExipryInsert delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryInsert deleting in the background"
}
2021/12/03 12:29:24 Context: Bucket flush, response code: 200 dump: 
2021/12/03 12:29:34 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/03 12:29:39 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/03 12:29:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 12:29:41 Context: Bucket flush, response code: 200 dump: 
2021/12/03 12:29:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestExipryInsert (220.93s)
=== RUN   TestExipryReplace
2021/12/03 12:29:46 Deploying app: TestExipryReplace. Item count src bucket: 0 dst bucket: 0
2021/12/03 12:29:47 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryReplace' in metakv",
  "warnings": null
 }
}
2021/12/03 12:29:52 Waiting for app: TestExipryReplace to get deployed
2021/12/03 12:29:57 Waiting for app: TestExipryReplace to get deployed
2021/12/03 12:30:02 Waiting for app: TestExipryReplace to get deployed
2021/12/03 12:30:07 Waiting for app: TestExipryReplace to get deployed
2021/12/03 12:30:07 App: TestExipryReplace got deployed
2021/12/03 12:30:07 Starting bucket ops to source bucket
2021/12/03 12:30:52 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/03 12:30:57 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/03 12:31:02 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/03 12:31:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 12:31:02 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 12:31:02 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/03 12:31:03 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 12:31:08 Waiting for app: TestExipryReplace to get un-deployed
2021/12/03 12:31:13 Waiting for app: TestExipryReplace to get un-deployed
2021/12/03 12:31:18 Waiting for app: TestExipryReplace to get un-deployed
2021/12/03 12:31:23 Waiting for app: TestExipryReplace to get un-deployed
2021/12/03 12:31:23 App: TestExipryReplace got un-deployed
2021/12/03 12:31:23 No eventing-con process running
2021/12/03 12:31:23 Function: TestExipryReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryReplace deleting in the background"
}
2021/12/03 12:31:24 Context: Bucket flush, response code: 200 dump: 
2021/12/03 12:31:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 12:31:32 Context: Bucket flush, response code: 200 dump: 
2021/12/03 12:31:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 12:31:37 Testing on the destination bucket for expiry insert
2021/12/03 12:31:37 Deploying app: TestExipryReplace. Item count src bucket: 0 dst bucket: 0
2021/12/03 12:31:38 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryReplace' in metakv",
  "warnings": null
 }
}
2021/12/03 12:31:43 Waiting for app: TestExipryReplace to get deployed
2021/12/03 12:31:48 Waiting for app: TestExipryReplace to get deployed
2021/12/03 12:31:53 Waiting for app: TestExipryReplace to get deployed
2021/12/03 12:31:58 Waiting for app: TestExipryReplace to get deployed
2021/12/03 12:31:58 App: TestExipryReplace got deployed
2021/12/03 12:31:58 Starting bucket ops to source bucket
2021/12/03 12:32:43 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/03 12:32:48 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/03 12:32:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 12:32:48 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 12:32:48 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/03 12:32:48 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 12:32:53 Waiting for app: TestExipryReplace to get un-deployed
2021/12/03 12:32:58 Waiting for app: TestExipryReplace to get un-deployed
2021/12/03 12:33:03 Waiting for app: TestExipryReplace to get un-deployed
2021/12/03 12:33:08 Waiting for app: TestExipryReplace to get un-deployed
2021/12/03 12:33:08 App: TestExipryReplace got un-deployed
2021/12/03 12:33:08 No eventing-con process running
2021/12/03 12:33:08 Function: TestExipryReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryReplace deleting in the background"
}
2021/12/03 12:33:10 Context: Bucket flush, response code: 200 dump: 
2021/12/03 12:33:20 Waiting for dst bucket item count to get to: 0 curr count: 57
2021/12/03 12:33:25 Waiting for dst bucket item count to get to: 0 curr count: 57
2021/12/03 12:33:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 12:33:27 Context: Bucket flush, response code: 200 dump: 
2021/12/03 12:33:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestExipryReplace (225.67s)
=== RUN   TestExipryUpsert
2021/12/03 12:33:32 Deploying app: TestExipryUpsert. Item count src bucket: 0 dst bucket: 0
2021/12/03 12:33:33 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryUpsert' in metakv",
  "warnings": null
 }
}
2021/12/03 12:33:38 Waiting for app: TestExipryUpsert to get deployed
2021/12/03 12:33:43 Waiting for app: TestExipryUpsert to get deployed
2021/12/03 12:33:48 Waiting for app: TestExipryUpsert to get deployed
2021/12/03 12:33:53 Waiting for app: TestExipryUpsert to get deployed
2021/12/03 12:33:53 App: TestExipryUpsert got deployed
2021/12/03 12:33:53 Starting bucket ops to source bucket
2021/12/03 12:34:38 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/03 12:34:43 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/03 12:34:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 12:34:43 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 12:34:43 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/03 12:34:43 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 12:34:48 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/03 12:34:53 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/03 12:34:58 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/03 12:35:03 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/03 12:35:03 App: TestExipryUpsert got un-deployed
2021/12/03 12:35:03 No eventing-con process running
2021/12/03 12:35:03 Function: TestExipryUpsert delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryUpsert deleting in the background"
}
2021/12/03 12:35:05 Context: Bucket flush, response code: 200 dump: 
2021/12/03 12:35:10 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 12:35:12 Context: Bucket flush, response code: 200 dump: 
2021/12/03 12:35:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 12:35:17 Testing on the destination bucket for expiry insert
2021/12/03 12:35:17 Deploying app: TestExipryUpsert. Item count src bucket: 0 dst bucket: 0
2021/12/03 12:35:19 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryUpsert' in metakv",
  "warnings": null
 }
}
2021/12/03 12:35:24 Waiting for app: TestExipryUpsert to get deployed
2021/12/03 12:35:29 Waiting for app: TestExipryUpsert to get deployed
2021/12/03 12:35:34 Waiting for app: TestExipryUpsert to get deployed
2021/12/03 12:35:39 Waiting for app: TestExipryUpsert to get deployed
2021/12/03 12:35:39 App: TestExipryUpsert got deployed
2021/12/03 12:35:39 Starting bucket ops to source bucket
2021/12/03 12:36:24 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/03 12:36:29 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/03 12:36:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 12:36:29 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 12:36:29 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/03 12:36:29 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 12:36:34 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/03 12:36:39 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/03 12:36:44 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/03 12:36:49 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/03 12:36:49 App: TestExipryUpsert got un-deployed
2021/12/03 12:36:49 No eventing-con process running
2021/12/03 12:36:49 Function: TestExipryUpsert delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryUpsert deleting in the background"
}
2021/12/03 12:36:51 Context: Bucket flush, response code: 200 dump: 
2021/12/03 12:37:01 Waiting for dst bucket item count to get to: 0 curr count: 85
2021/12/03 12:37:06 Waiting for dst bucket item count to get to: 0 curr count: 85
2021/12/03 12:37:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 12:37:07 Context: Bucket flush, response code: 200 dump: 
2021/12/03 12:37:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestExipryUpsert (220.79s)
=== RUN   TestCasReplace
2021/12/03 12:37:13 Deploying app: TestCasReplace. Item count src bucket: 0 dst bucket: 0
2021/12/03 12:37:14 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCasReplace' in metakv",
  "warnings": null
 }
}
2021/12/03 12:37:19 Waiting for app: TestCasReplace to get deployed
2021/12/03 12:37:24 Waiting for app: TestCasReplace to get deployed
2021/12/03 12:37:29 Waiting for app: TestCasReplace to get deployed
2021/12/03 12:37:34 Waiting for app: TestCasReplace to get deployed
2021/12/03 12:37:34 App: TestCasReplace got deployed
2021/12/03 12:37:34 Starting bucket ops to source bucket
2021/12/03 12:37:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 12:37:34 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 12:37:34 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/03 12:37:34 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 12:37:39 Waiting for app: TestCasReplace to get un-deployed
2021/12/03 12:37:44 Waiting for app: TestCasReplace to get un-deployed
2021/12/03 12:37:49 Waiting for app: TestCasReplace to get un-deployed
2021/12/03 12:37:54 Waiting for app: TestCasReplace to get un-deployed
2021/12/03 12:37:54 App: TestCasReplace got un-deployed
2021/12/03 12:37:54 No eventing-con process running
2021/12/03 12:37:54 Function: TestCasReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCasReplace deleting in the background"
}
2021/12/03 12:37:56 Context: Bucket flush, response code: 200 dump: 
2021/12/03 12:38:06 Waiting for dst bucket item count to get to: 0 curr count: 11
2021/12/03 12:38:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 12:38:08 Context: Bucket flush, response code: 200 dump: 
2021/12/03 12:38:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 12:38:13 Testing on the destination bucket for upsert with cas
2021/12/03 12:38:13 Starting bucket ops to source bucket
2021/12/03 12:38:13 Deploying app: TestCasReplace. Item count src bucket: 0 dst bucket: 0
2021/12/03 12:38:14 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCasReplace' in metakv",
  "warnings": null
 }
}
2021/12/03 12:38:19 Waiting for app: TestCasReplace to get deployed
2021/12/03 12:38:24 Waiting for app: TestCasReplace to get deployed
2021/12/03 12:38:29 Waiting for app: TestCasReplace to get deployed
2021/12/03 12:38:34 Waiting for app: TestCasReplace to get deployed
2021/12/03 12:38:34 App: TestCasReplace got deployed
2021/12/03 12:38:34 Starting bucket ops to source bucket
2021/12/03 12:38:39 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/03 12:38:44 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/03 12:38:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 12:38:44 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 12:38:44 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/03 12:38:44 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 12:38:49 Waiting for app: TestCasReplace to get un-deployed
2021/12/03 12:38:54 Waiting for app: TestCasReplace to get un-deployed
2021/12/03 12:38:59 Waiting for app: TestCasReplace to get un-deployed
2021/12/03 12:39:04 Waiting for app: TestCasReplace to get un-deployed
2021/12/03 12:39:04 App: TestCasReplace got un-deployed
2021/12/03 12:39:04 No eventing-con process running
2021/12/03 12:39:04 Function: TestCasReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCasReplace deleting in the background"
}
2021/12/03 12:39:06 Context: Bucket flush, response code: 200 dump: 
2021/12/03 12:39:16 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/03 12:39:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 12:39:18 Context: Bucket flush, response code: 200 dump: 
2021/12/03 12:39:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCasReplace (130.22s)
=== RUN   TestCasDelete
2021/12/03 12:39:23 Deploying app: TestCasDelete. Item count src bucket: 0 dst bucket: 0
2021/12/03 12:39:24 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCasDelete' in metakv",
  "warnings": null
 }
}
2021/12/03 12:39:29 Waiting for app: TestCasDelete to get deployed
2021/12/03 12:39:34 Waiting for app: TestCasDelete to get deployed
2021/12/03 12:39:39 Waiting for app: TestCasDelete to get deployed
2021/12/03 12:39:44 Waiting for app: TestCasDelete to get deployed
2021/12/03 12:39:44 App: TestCasDelete got deployed
2021/12/03 12:39:44 Starting bucket ops to source bucket
2021/12/03 12:39:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 12:39:44 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 12:39:44 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/03 12:39:44 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 12:39:49 Waiting for app: TestCasDelete to get un-deployed
2021/12/03 12:39:54 Waiting for app: TestCasDelete to get un-deployed
2021/12/03 12:39:59 Waiting for app: TestCasDelete to get un-deployed
2021/12/03 12:40:04 Waiting for app: TestCasDelete to get un-deployed
2021/12/03 12:40:04 App: TestCasDelete got un-deployed
2021/12/03 12:40:04 No eventing-con process running
2021/12/03 12:40:04 Function: TestCasDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCasDelete deleting in the background"
}
2021/12/03 12:40:06 Context: Bucket flush, response code: 200 dump: 
2021/12/03 12:40:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 12:40:13 Context: Bucket flush, response code: 200 dump: 
2021/12/03 12:40:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 12:40:18 Testing on the destination bucket for delete with cas
2021/12/03 12:40:18 Starting bucket ops to source bucket
2021/12/03 12:40:19 Deploying app: TestCasDelete. Item count src bucket: 0 dst bucket: 0
2021/12/03 12:40:20 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCasDelete' in metakv",
  "warnings": null
 }
}
2021/12/03 12:40:25 Waiting for app: TestCasDelete to get deployed
2021/12/03 12:40:30 Waiting for app: TestCasDelete to get deployed
2021/12/03 12:40:35 Waiting for app: TestCasDelete to get deployed
2021/12/03 12:40:40 Waiting for app: TestCasDelete to get deployed
2021/12/03 12:40:40 App: TestCasDelete got deployed
2021/12/03 12:40:40 Starting bucket ops to source bucket
2021/12/03 12:40:45 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/03 12:40:50 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/03 12:40:55 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/03 12:40:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 12:40:55 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 12:40:55 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/03 12:40:55 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 12:41:00 Waiting for app: TestCasDelete to get un-deployed
2021/12/03 12:41:05 Waiting for app: TestCasDelete to get un-deployed
2021/12/03 12:41:10 Waiting for app: TestCasDelete to get un-deployed
2021/12/03 12:41:15 Waiting for app: TestCasDelete to get un-deployed
2021/12/03 12:41:15 App: TestCasDelete got un-deployed
2021/12/03 12:41:15 No eventing-con process running
2021/12/03 12:41:15 Function: TestCasDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCasDelete deleting in the background"
}
2021/12/03 12:41:17 Context: Bucket flush, response code: 200 dump: 
2021/12/03 12:41:27 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/03 12:41:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 12:41:29 Context: Bucket flush, response code: 200 dump: 
2021/12/03 12:41:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCasDelete (130.98s)
=== RUN   TestCountersIncrement
2021/12/03 12:41:34 Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/03 12:41:40 Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/03 12:41:40 Removing node(s): [] from the cluster
2021/12/03 12:41:45 Rebalance progress: 100
2021/12/03 12:41:45 Deploying app: TestCountersIncrement. Item count src bucket: 0 dst bucket: 0
2021/12/03 12:41:46 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCountersIncrement' in metakv",
  "warnings": null
 }
}
2021/12/03 12:41:51 Waiting for app: TestCountersIncrement to get deployed
2021/12/03 12:41:56 Waiting for app: TestCountersIncrement to get deployed
2021/12/03 12:42:01 Waiting for app: TestCountersIncrement to get deployed
2021/12/03 12:42:06 Waiting for app: TestCountersIncrement to get deployed
2021/12/03 12:42:06 App: TestCountersIncrement got deployed
2021/12/03 12:42:06 Starting bucket ops to source bucket
2021/12/03 12:42:12 Waiting for dst bucket item count to get to: 1024 curr count: 0
2021/12/03 12:42:17 Waiting for dst bucket item count to get to: 1024 curr count: 392
2021/12/03 12:42:22 Waiting for dst bucket item count to get to: 1024 curr count: 904
2021/12/03 12:42:22 src & dst bucket item count matched up. src bucket count: 1024 dst bucket count: 1024
2021/12/03 12:42:22 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 12:42:23 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 12:42:28 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/03 12:42:33 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/03 12:42:38 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/03 12:42:43 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/03 12:42:43 App: TestCountersIncrement got un-deployed
2021/12/03 12:42:43 No eventing-con process running
2021/12/03 12:42:43 Function: TestCountersIncrement delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCountersIncrement deleting in the background"
}
2021/12/03 12:42:44 Context: Bucket flush, response code: 200 dump: 
2021/12/03 12:42:54 Waiting for dst bucket item count to get to: 0 curr count: 1025
2021/12/03 12:42:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 12:42:56 Context: Bucket flush, response code: 200 dump: 
2021/12/03 12:43:06 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/03 12:43:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 12:43:06 Deploying app: TestCountersIncrement. Item count src bucket: 0 dst bucket: 0
2021/12/03 12:43:07 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCountersIncrement' in metakv",
  "warnings": null
 }
}
2021/12/03 12:43:12 Waiting for app: TestCountersIncrement to get deployed
2021/12/03 12:43:17 Waiting for app: TestCountersIncrement to get deployed
2021/12/03 12:43:22 Waiting for app: TestCountersIncrement to get deployed
2021/12/03 12:43:27 Waiting for app: TestCountersIncrement to get deployed
2021/12/03 12:43:27 App: TestCountersIncrement got deployed
2021/12/03 12:43:27 Starting bucket ops to source bucket
2021/12/03 12:43:34 Waiting for dst bucket item count to get to: 2048 curr count: 0
2021/12/03 12:43:39 Waiting for dst bucket item count to get to: 2048 curr count: 1024
2021/12/03 12:43:44 Waiting for dst bucket item count to get to: 2048 curr count: 1024
2021/12/03 12:43:44 src & dst bucket item count matched up. src bucket count: 2048 dst bucket count: 2048
2021/12/03 12:43:44 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 12:43:44 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 12:43:49 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/03 12:43:54 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/03 12:43:59 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/03 12:44:04 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/03 12:44:04 App: TestCountersIncrement got un-deployed
2021/12/03 12:44:04 No eventing-con process running
2021/12/03 12:44:04 Function: TestCountersIncrement delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCountersIncrement deleting in the background"
}
2021/12/03 12:44:06 Context: Bucket flush, response code: 200 dump: 
2021/12/03 12:44:16 Waiting for dst bucket item count to get to: 0 curr count: 2048
2021/12/03 12:44:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 12:44:18 Context: Bucket flush, response code: 200 dump: 
2021/12/03 12:44:28 Waiting for dst bucket item count to get to: 0 curr count: 1
2021/12/03 12:44:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 12:44:28 Removing node(s): [http://127.0.0.1:19002 http://127.0.0.1:19003] from the cluster
2021/12/03 12:44:33 Rebalance progress: 100
--- PASS: TestCountersIncrement (178.98s)
=== RUN   TestCountersDecrement
2021/12/03 12:44:33 Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/03 12:44:33 Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/03 12:44:33 Removing node(s): [] from the cluster
2021/12/03 12:44:38 Rebalance progress: 100
2021/12/03 12:44:38 Deploying app: TestCountersDecrement. Item count src bucket: 0 dst bucket: 0
2021/12/03 12:44:39 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCountersDecrement' in metakv",
  "warnings": null
 }
}
2021/12/03 12:44:44 Waiting for app: TestCountersDecrement to get deployed
2021/12/03 12:44:49 Waiting for app: TestCountersDecrement to get deployed
2021/12/03 12:44:54 Waiting for app: TestCountersDecrement to get deployed
2021/12/03 12:44:59 Waiting for app: TestCountersDecrement to get deployed
2021/12/03 12:44:59 App: TestCountersDecrement got deployed
2021/12/03 12:44:59 Starting bucket ops to source bucket
2021/12/03 12:45:05 Waiting for dst bucket item count to get to: 1024 curr count: 0
2021/12/03 12:45:10 Waiting for dst bucket item count to get to: 1024 curr count: 0
2021/12/03 12:45:15 Waiting for dst bucket item count to get to: 1024 curr count: 512
2021/12/03 12:45:15 src & dst bucket item count matched up. src bucket count: 1024 dst bucket count: 1024
2021/12/03 12:45:15 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 12:45:15 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 12:45:20 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/03 12:45:25 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/03 12:45:31 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/03 12:45:36 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/03 12:45:36 App: TestCountersDecrement got un-deployed
2021/12/03 12:45:36 No eventing-con process running
2021/12/03 12:45:36 Function: TestCountersDecrement delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCountersDecrement deleting in the background"
}
2021/12/03 12:45:37 Context: Bucket flush, response code: 200 dump: 
2021/12/03 12:45:47 Waiting for dst bucket item count to get to: 0 curr count: 513
2021/12/03 12:45:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 12:45:49 Context: Bucket flush, response code: 200 dump: 
2021/12/03 12:45:59 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/03 12:46:04 Waiting for dst bucket item count to get to: 0 curr count: 512
2021/12/03 12:46:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 12:46:04 Deploying app: TestCountersDecrement. Item count src bucket: 0 dst bucket: 0
2021/12/03 12:46:05 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCountersDecrement' in metakv",
  "warnings": null
 }
}
2021/12/03 12:46:10 Waiting for app: TestCountersDecrement to get deployed
2021/12/03 12:46:15 Waiting for app: TestCountersDecrement to get deployed
2021/12/03 12:46:20 Waiting for app: TestCountersDecrement to get deployed
2021/12/03 12:46:25 Waiting for app: TestCountersDecrement to get deployed
2021/12/03 12:46:25 App: TestCountersDecrement got deployed
2021/12/03 12:46:25 Starting bucket ops to source bucket
2021/12/03 12:46:32 Waiting for dst bucket item count to get to: 2048 curr count: 0
2021/12/03 12:46:32 src & dst bucket item count matched up. src bucket count: 2048 dst bucket count: 2048
2021/12/03 12:46:32 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 12:46:32 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 12:46:37 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/03 12:46:42 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/03 12:46:47 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/03 12:46:52 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/03 12:46:52 App: TestCountersDecrement got un-deployed
2021/12/03 12:46:52 No eventing-con process running
2021/12/03 12:46:52 Function: TestCountersDecrement delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCountersDecrement deleting in the background"
}
2021/12/03 12:46:54 Context: Bucket flush, response code: 200 dump: 
2021/12/03 12:47:04 Waiting for dst bucket item count to get to: 0 curr count: 2048
2021/12/03 12:47:09 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/03 12:47:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 12:47:10 Context: Bucket flush, response code: 200 dump: 
2021/12/03 12:47:20 Waiting for dst bucket item count to get to: 0 curr count: 1
2021/12/03 12:47:25 Waiting for dst bucket item count to get to: 0 curr count: 1
2021/12/03 12:47:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 12:47:25 Removing node(s): [http://127.0.0.1:19002 http://127.0.0.1:19003] from the cluster
2021/12/03 12:47:30 Rebalance progress: 100
--- PASS: TestCountersDecrement (177.75s)
=== RUN   TestCacheGet
2021/12/03 12:47:30 Deploying app: TestCacheGet. Item count src bucket: 0 dst bucket: 0
2021/12/03 12:47:32 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCacheGet' in metakv",
  "warnings": null
 }
}
2021/12/03 12:47:37 Waiting for app: TestCacheGet to get deployed
2021/12/03 12:47:42 Waiting for app: TestCacheGet to get deployed
2021/12/03 12:47:47 Waiting for app: TestCacheGet to get deployed
2021/12/03 12:47:52 Waiting for app: TestCacheGet to get deployed
2021/12/03 12:47:52 App: TestCacheGet got deployed
2021/12/03 12:47:52 Starting bucket ops to source bucket
2021/12/03 12:47:57 Waiting for dst bucket item count to get to: 20 curr count: 0
2021/12/03 12:48:02 Waiting for dst bucket item count to get to: 20 curr count: 0
2021/12/03 12:48:07 Waiting for dst bucket item count to get to: 20 curr count: 5
2021/12/03 12:48:12 Waiting for dst bucket item count to get to: 20 curr count: 5
2021/12/03 12:48:17 Waiting for dst bucket item count to get to: 20 curr count: 18
2021/12/03 12:48:22 Waiting for dst bucket item count to get to: 20 curr count: 18
2021/12/03 12:48:22 src & dst bucket item count matched up. src bucket count: 20 dst bucket count: 20
2021/12/03 12:48:22 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 12:48:22 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 12:48:27 Waiting for app: TestCacheGet to get un-deployed
2021/12/03 12:48:32 Waiting for app: TestCacheGet to get un-deployed
2021/12/03 12:48:37 Waiting for app: TestCacheGet to get un-deployed
2021/12/03 12:48:42 Waiting for app: TestCacheGet to get un-deployed
2021/12/03 12:48:42 App: TestCacheGet got un-deployed
2021/12/03 12:48:42 No eventing-con process running
2021/12/03 12:48:42 Function: TestCacheGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCacheGet deleting in the background"
}
2021/12/03 12:48:44 Context: Bucket flush, response code: 200 dump: 
2021/12/03 12:48:54 Waiting for dst bucket item count to get to: 0 curr count: 20
2021/12/03 12:48:59 Waiting for dst bucket item count to get to: 0 curr count: 10
2021/12/03 12:48:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 12:49:01 Context: Bucket flush, response code: 200 dump: 
2021/12/03 12:49:11 Waiting for dst bucket item count to get to: 0 curr count: 18
2021/12/03 12:49:16 Waiting for dst bucket item count to get to: 0 curr count: 18
2021/12/03 12:49:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCacheGet (105.33s)
=== RUN   TestCacheRYOW
2021/12/03 12:49:16 Deploying app: TestCacheRYOW. Item count src bucket: 0 dst bucket: 0
2021/12/03 12:49:17 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCacheRYOW' in metakv",
  "warnings": null
 }
}
2021/12/03 12:49:22 Waiting for app: TestCacheRYOW to get deployed
2021/12/03 12:49:27 Waiting for app: TestCacheRYOW to get deployed
2021/12/03 12:49:32 Waiting for app: TestCacheRYOW to get deployed
2021/12/03 12:49:37 Waiting for app: TestCacheRYOW to get deployed
2021/12/03 12:49:37 App: TestCacheRYOW got deployed
2021/12/03 12:49:37 Starting bucket ops to source bucket
2021/12/03 12:49:42 Waiting for dst bucket item count to get to: 50 curr count: 0
2021/12/03 12:49:47 Waiting for dst bucket item count to get to: 50 curr count: 33
2021/12/03 12:49:52 Waiting for dst bucket item count to get to: 50 curr count: 33
2021/12/03 12:49:52 src & dst bucket item count matched up. src bucket count: 50 dst bucket count: 50
2021/12/03 12:49:52 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 12:49:52 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 12:49:57 Waiting for app: TestCacheRYOW to get un-deployed
2021/12/03 12:50:02 Waiting for app: TestCacheRYOW to get un-deployed
2021/12/03 12:50:07 Waiting for app: TestCacheRYOW to get un-deployed
2021/12/03 12:50:12 Waiting for app: TestCacheRYOW to get un-deployed
2021/12/03 12:50:12 App: TestCacheRYOW got un-deployed
2021/12/03 12:50:12 No eventing-con process running
2021/12/03 12:50:12 Function: TestCacheRYOW delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCacheRYOW deleting in the background"
}
2021/12/03 12:50:14 Context: Bucket flush, response code: 200 dump: 
2021/12/03 12:50:24 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/03 12:50:29 Waiting for dst bucket item count to get to: 0 curr count: 25
2021/12/03 12:50:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 12:50:31 Context: Bucket flush, response code: 200 dump: 
2021/12/03 12:50:41 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/03 12:50:46 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/03 12:50:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCacheRYOW (90.31s)
=== RUN   TestCacheExpiry
2021/12/03 12:50:46 Deploying app: TestCacheExpiry. Item count src bucket: 0 dst bucket: 0
2021/12/03 12:50:47 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCacheExpiry' in metakv",
  "warnings": null
 }
}
2021/12/03 12:50:52 Waiting for app: TestCacheExpiry to get deployed
2021/12/03 12:50:57 Waiting for app: TestCacheExpiry to get deployed
2021/12/03 12:51:02 Waiting for app: TestCacheExpiry to get deployed
2021/12/03 12:51:07 Waiting for app: TestCacheExpiry to get deployed
2021/12/03 12:51:07 App: TestCacheExpiry got deployed
2021/12/03 12:51:07 Starting bucket ops to source bucket
2021/12/03 12:51:12 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/03 12:51:18 Waiting for dst bucket item count to get to: 100 curr count: 3
2021/12/03 12:51:23 Waiting for dst bucket item count to get to: 100 curr count: 3
2021/12/03 12:51:28 Waiting for dst bucket item count to get to: 100 curr count: 86
2021/12/03 12:51:33 Waiting for dst bucket item count to get to: 100 curr count: 86
2021/12/03 12:51:33 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/03 12:51:33 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 12:51:33 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 12:51:38 Waiting for app: TestCacheExpiry to get un-deployed
2021/12/03 12:51:43 Waiting for app: TestCacheExpiry to get un-deployed
2021/12/03 12:51:48 Waiting for app: TestCacheExpiry to get un-deployed
2021/12/03 12:51:53 Waiting for app: TestCacheExpiry to get un-deployed
2021/12/03 12:51:53 App: TestCacheExpiry got un-deployed
2021/12/03 12:51:53 No eventing-con process running
2021/12/03 12:51:53 Function: TestCacheExpiry delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCacheExpiry deleting in the background"
}
2021/12/03 12:51:55 Context: Bucket flush, response code: 200 dump: 
2021/12/03 12:52:05 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/03 12:52:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 12:52:06 Context: Bucket flush, response code: 200 dump: 
2021/12/03 12:52:16 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/03 12:52:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCacheExpiry (90.32s)
=== RUN   TestCacheExpiryCustom
2021/12/03 12:52:16 Deploying app: TestCacheExpiryCustom. Item count src bucket: 0 dst bucket: 0
2021/12/03 12:52:18 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCacheExpiryCustom' in metakv",
  "warnings": null
 }
}
2021/12/03 12:52:23 Waiting for app: TestCacheExpiryCustom to get deployed
2021/12/03 12:52:28 Waiting for app: TestCacheExpiryCustom to get deployed
2021/12/03 12:52:33 Waiting for app: TestCacheExpiryCustom to get deployed
2021/12/03 12:52:38 Waiting for app: TestCacheExpiryCustom to get deployed
2021/12/03 12:52:38 App: TestCacheExpiryCustom got deployed
2021/12/03 12:52:38 Starting bucket ops to source bucket
2021/12/03 12:52:43 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/03 12:52:48 Waiting for dst bucket item count to get to: 100 curr count: 3
2021/12/03 12:52:53 Waiting for dst bucket item count to get to: 100 curr count: 3
2021/12/03 12:52:58 Waiting for dst bucket item count to get to: 100 curr count: 86
2021/12/03 12:53:03 Waiting for dst bucket item count to get to: 100 curr count: 86
2021/12/03 12:53:03 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/03 12:53:03 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 12:53:03 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 12:53:08 Waiting for app: TestCacheExpiryCustom to get un-deployed
2021/12/03 12:53:13 Waiting for app: TestCacheExpiryCustom to get un-deployed
2021/12/03 12:53:18 Waiting for app: TestCacheExpiryCustom to get un-deployed
2021/12/03 12:53:23 Waiting for app: TestCacheExpiryCustom to get un-deployed
2021/12/03 12:53:23 App: TestCacheExpiryCustom got un-deployed
2021/12/03 12:53:23 No eventing-con process running
2021/12/03 12:53:23 Function: TestCacheExpiryCustom delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCacheExpiryCustom deleting in the background"
}
2021/12/03 12:53:25 Context: Bucket flush, response code: 200 dump: 
2021/12/03 12:53:35 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/03 12:53:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 12:53:37 Context: Bucket flush, response code: 200 dump: 
2021/12/03 12:53:47 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/03 12:53:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCacheExpiryCustom (90.19s)
=== RUN   TestCacheOverflowCustom
2021/12/03 12:53:47 Deploying app: TestCacheOverflowCustom. Item count src bucket: 0 dst bucket: 0
2021/12/03 12:53:48 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCacheOverflowCustom' in metakv",
  "warnings": null
 }
}
2021/12/03 12:53:53 Waiting for app: TestCacheOverflowCustom to get deployed
2021/12/03 12:53:58 Waiting for app: TestCacheOverflowCustom to get deployed
2021/12/03 12:54:03 Waiting for app: TestCacheOverflowCustom to get deployed
2021/12/03 12:54:08 Waiting for app: TestCacheOverflowCustom to get deployed
2021/12/03 12:54:08 App: TestCacheOverflowCustom got deployed
2021/12/03 12:54:08 Starting bucket ops to source bucket
2021/12/03 12:54:13 Waiting for dst bucket item count to get to: 35 curr count: 0
2021/12/03 12:54:18 Waiting for dst bucket item count to get to: 35 curr count: 3
2021/12/03 12:54:23 Waiting for dst bucket item count to get to: 35 curr count: 3
2021/12/03 12:54:28 Waiting for dst bucket item count to get to: 35 curr count: 25
2021/12/03 12:54:33 Waiting for dst bucket item count to get to: 35 curr count: 25
2021/12/03 12:54:38 Waiting for dst bucket item count to get to: 35 curr count: 34
2021/12/03 12:54:43 Waiting for dst bucket item count to get to: 35 curr count: 34
2021/12/03 12:54:43 src & dst bucket item count matched up. src bucket count: 35 dst bucket count: 35
2021/12/03 12:54:43 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 12:54:43 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 12:54:48 Waiting for app: TestCacheOverflowCustom to get un-deployed
2021/12/03 12:54:53 Waiting for app: TestCacheOverflowCustom to get un-deployed
2021/12/03 12:54:58 Waiting for app: TestCacheOverflowCustom to get un-deployed
2021/12/03 12:55:03 Waiting for app: TestCacheOverflowCustom to get un-deployed
2021/12/03 12:55:03 App: TestCacheOverflowCustom got un-deployed
2021/12/03 12:55:03 No eventing-con process running
2021/12/03 12:55:03 Function: TestCacheOverflowCustom delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCacheOverflowCustom deleting in the background"
}
2021/12/03 12:55:05 Context: Bucket flush, response code: 200 dump: 
2021/12/03 12:55:15 Waiting for dst bucket item count to get to: 0 curr count: 5
2021/12/03 12:55:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 12:55:17 Context: Bucket flush, response code: 200 dump: 
2021/12/03 12:55:27 Waiting for dst bucket item count to get to: 0 curr count: 35
2021/12/03 12:55:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCacheOverflowCustom (100.28s)
=== RUN   TestStrictMode
2021/12/03 12:55:27 Deploying app: TestStrictMode. Item count src bucket: 0 dst bucket: 0
2021/12/03 12:55:28 Context: Post to main store, response code: 422 dump: {
 "name": "ERR_HANDLER_COMPILATION",
 "code": 27,
 "description": "Function compilation failed",
 "attributes": null,
 "runtime_info": {
  "code": 27,
  "info": {
   "area": "handlerCode",
   "column_number": 63,
   "compile_success": false,
   "description": "        destination[meta.id] = 00000000000000000000000000000101;\n                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nSyntaxError: Octal literals are not allowed in strict mode.\n",
   "index": 132,
   "language": "JavaScript",
   "line_number": 4
  }
 }
}
--- PASS: TestStrictMode (1.06s)
=== RUN   TestDataTypes
2021/12/03 12:55:28 Deploying app: TestDataTypes. Item count src bucket: 0 dst bucket: 0
2021/12/03 12:55:29 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDataTypes' in metakv",
  "warnings": null
 }
}
2021/12/03 12:55:34 Waiting for app: TestDataTypes to get deployed
2021/12/03 12:55:39 Waiting for app: TestDataTypes to get deployed
2021/12/03 12:55:44 Waiting for app: TestDataTypes to get deployed
2021/12/03 12:55:49 Waiting for app: TestDataTypes to get deployed
2021/12/03 12:55:49 App: TestDataTypes got deployed
2021/12/03 12:55:49 Starting bucket ops to source bucket
2021/12/03 12:55:54 Waiting for dst bucket item count to get to: 18 curr count: 0
2021/12/03 12:55:59 Waiting for dst bucket item count to get to: 18 curr count: 10
2021/12/03 12:55:59 src & dst bucket item count matched up. src bucket count: 18 dst bucket count: 18
2021/12/03 12:55:59 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 12:55:59 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 12:56:04 Waiting for app: TestDataTypes to get un-deployed
2021/12/03 12:56:09 Waiting for app: TestDataTypes to get un-deployed
2021/12/03 12:56:14 Waiting for app: TestDataTypes to get un-deployed
2021/12/03 12:56:19 Waiting for app: TestDataTypes to get un-deployed
2021/12/03 12:56:19 App: TestDataTypes got un-deployed
2021/12/03 12:56:19 No eventing-con process running
2021/12/03 12:56:19 Function: TestDataTypes delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDataTypes deleting in the background"
}
2021/12/03 12:56:21 Context: Bucket flush, response code: 200 dump: 
2021/12/03 12:56:31 Waiting for dst bucket item count to get to: 0 curr count: 1
2021/12/03 12:56:36 Waiting for dst bucket item count to get to: 0 curr count: 1
2021/12/03 12:56:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 12:56:38 Context: Bucket flush, response code: 200 dump: 
2021/12/03 12:56:48 Waiting for dst bucket item count to get to: 0 curr count: 13
2021/12/03 12:56:53 Waiting for dst bucket item count to get to: 0 curr count: 3
2021/12/03 12:56:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDataTypes (85.13s)
=== RUN   TestEnoentGet
2021/12/03 12:56:53 Deploying app: TestEnoentGet. Item count src bucket: 0 dst bucket: 0
2021/12/03 12:56:54 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentGet' in metakv",
  "warnings": null
 }
}
2021/12/03 12:56:59 Waiting for app: TestEnoentGet to get deployed
2021/12/03 12:57:04 Waiting for app: TestEnoentGet to get deployed
2021/12/03 12:57:09 Waiting for app: TestEnoentGet to get deployed
2021/12/03 12:57:14 Waiting for app: TestEnoentGet to get deployed
2021/12/03 12:57:14 App: TestEnoentGet got deployed
2021/12/03 12:57:14 Starting bucket ops to source bucket
2021/12/03 12:57:19 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/03 12:57:24 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/03 12:57:24 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/03 12:57:24 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 12:57:25 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 12:57:30 Waiting for app: TestEnoentGet to get un-deployed
2021/12/03 12:57:35 Waiting for app: TestEnoentGet to get un-deployed
2021/12/03 12:57:40 Waiting for app: TestEnoentGet to get un-deployed
2021/12/03 12:57:45 Waiting for app: TestEnoentGet to get un-deployed
2021/12/03 12:57:45 App: TestEnoentGet got un-deployed
2021/12/03 12:57:45 No eventing-con process running
2021/12/03 12:57:45 Function: TestEnoentGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentGet deleting in the background"
}
2021/12/03 12:57:46 Context: Bucket flush, response code: 200 dump: 
2021/12/03 12:57:56 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/03 12:57:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 12:57:58 Context: Bucket flush, response code: 200 dump: 
2021/12/03 12:58:08 Waiting for dst bucket item count to get to: 0 curr count: 79
2021/12/03 12:58:13 Waiting for dst bucket item count to get to: 0 curr count: 29
2021/12/03 12:58:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 12:58:13 Deploying app: TestEnoentGet. Item count src bucket: 0 dst bucket: 0
2021/12/03 12:58:14 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentGet' in metakv",
  "warnings": null
 }
}
2021/12/03 12:58:19 Waiting for app: TestEnoentGet to get deployed
2021/12/03 12:58:24 Waiting for app: TestEnoentGet to get deployed
2021/12/03 12:58:29 Waiting for app: TestEnoentGet to get deployed
2021/12/03 12:58:34 Waiting for app: TestEnoentGet to get deployed
2021/12/03 12:58:34 App: TestEnoentGet got deployed
2021/12/03 12:58:34 Starting bucket ops to source bucket
2021/12/03 12:58:40 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/03 12:58:45 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/03 12:58:45 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/03 12:58:45 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 12:58:45 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 12:58:50 Waiting for app: TestEnoentGet to get un-deployed
2021/12/03 12:58:55 Waiting for app: TestEnoentGet to get un-deployed
2021/12/03 12:59:00 Waiting for app: TestEnoentGet to get un-deployed
2021/12/03 12:59:05 Waiting for app: TestEnoentGet to get un-deployed
2021/12/03 12:59:05 App: TestEnoentGet got un-deployed
2021/12/03 12:59:05 No eventing-con process running
2021/12/03 12:59:05 Function: TestEnoentGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentGet deleting in the background"
}
2021/12/03 12:59:07 Context: Bucket flush, response code: 200 dump: 
2021/12/03 12:59:17 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/03 12:59:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 12:59:18 Context: Bucket flush, response code: 200 dump: 
2021/12/03 12:59:28 Waiting for dst bucket item count to get to: 0 curr count: 83
2021/12/03 12:59:33 Waiting for dst bucket item count to get to: 0 curr count: 33
2021/12/03 12:59:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEnoentGet (160.44s)
=== RUN   TestEnoentDelete
2021/12/03 12:59:34 Deploying app: TestEnoentDelete. Item count src bucket: 0 dst bucket: 0
2021/12/03 12:59:35 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentDelete' in metakv",
  "warnings": null
 }
}
2021/12/03 12:59:40 Waiting for app: TestEnoentDelete to get deployed
2021/12/03 12:59:45 Waiting for app: TestEnoentDelete to get deployed
2021/12/03 12:59:50 Waiting for app: TestEnoentDelete to get deployed
2021/12/03 12:59:55 Waiting for app: TestEnoentDelete to get deployed
2021/12/03 12:59:55 App: TestEnoentDelete got deployed
2021/12/03 12:59:55 Starting bucket ops to source bucket
2021/12/03 13:00:00 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/03 13:00:05 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/03 13:00:05 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/03 13:00:05 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 13:00:05 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 13:00:10 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/03 13:00:15 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/03 13:00:20 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/03 13:00:25 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/03 13:00:25 App: TestEnoentDelete got un-deployed
2021/12/03 13:00:25 No eventing-con process running
2021/12/03 13:00:25 Function: TestEnoentDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentDelete deleting in the background"
}
2021/12/03 13:00:27 Context: Bucket flush, response code: 200 dump: 
2021/12/03 13:00:37 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/03 13:00:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 13:00:39 Context: Bucket flush, response code: 200 dump: 
2021/12/03 13:00:49 Waiting for dst bucket item count to get to: 0 curr count: 89
2021/12/03 13:00:54 Waiting for dst bucket item count to get to: 0 curr count: 39
2021/12/03 13:00:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 13:00:54 Deploying app: TestEnoentDelete. Item count src bucket: 0 dst bucket: 0
2021/12/03 13:00:55 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentDelete' in metakv",
  "warnings": null
 }
}
2021/12/03 13:01:00 Waiting for app: TestEnoentDelete to get deployed
2021/12/03 13:01:05 Waiting for app: TestEnoentDelete to get deployed
2021/12/03 13:01:10 Waiting for app: TestEnoentDelete to get deployed
2021/12/03 13:01:15 Waiting for app: TestEnoentDelete to get deployed
2021/12/03 13:01:15 App: TestEnoentDelete got deployed
2021/12/03 13:01:15 Starting bucket ops to source bucket
2021/12/03 13:01:20 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/03 13:01:25 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/03 13:01:25 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/03 13:01:25 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 13:01:25 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 13:01:30 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/03 13:01:35 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/03 13:01:40 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/03 13:01:45 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/03 13:01:45 App: TestEnoentDelete got un-deployed
2021/12/03 13:01:45 No eventing-con process running
2021/12/03 13:01:45 Function: TestEnoentDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentDelete deleting in the background"
}
2021/12/03 13:01:47 Context: Bucket flush, response code: 200 dump: 
2021/12/03 13:01:57 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/03 13:01:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 13:01:59 Context: Bucket flush, response code: 200 dump: 
2021/12/03 13:02:09 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/03 13:02:14 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/03 13:02:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEnoentDelete (160.47s)
=== RUN   TestError
2021/12/03 13:02:24 Waiting for app: TestError to get un-deployed
2021/12/03 13:02:24 App: TestError got un-deployed
2021/12/03 13:02:24 No eventing-con process running
2021/12/03 13:02:24 Function: TestError delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestError not found"
 }
}
2021/12/03 13:02:26 Context: Bucket flush, response code: 200 dump: 
2021/12/03 13:02:31 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 13:02:33 Context: Bucket flush, response code: 200 dump: 
2021/12/03 13:02:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 13:02:38 Deploying app: TestError. Item count src bucket: 0 dst bucket: 0
2021/12/03 13:02:39 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestError' in metakv",
  "warnings": null
 }
}
2021/12/03 13:02:44 Waiting for app: TestError to get deployed
2021/12/03 13:02:49 Waiting for app: TestError to get deployed
2021/12/03 13:02:54 Waiting for app: TestError to get deployed
2021/12/03 13:02:59 Waiting for app: TestError to get deployed
2021/12/03 13:02:59 App: TestError got deployed
2021/12/03 13:02:59 Starting bucket ops to source bucket
2021/12/03 13:03:04 Waiting for dst bucket item count to get to: 300 curr count: 0
2021/12/03 13:03:09 Waiting for dst bucket item count to get to: 300 curr count: 144
2021/12/03 13:03:09 src & dst bucket item count matched up. src bucket count: 300 dst bucket count: 300
2021/12/03 13:03:09 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 13:03:09 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 13:03:14 Waiting for app: TestError to get un-deployed
2021/12/03 13:03:19 Waiting for app: TestError to get un-deployed
2021/12/03 13:03:24 Waiting for app: TestError to get un-deployed
2021/12/03 13:03:29 Waiting for app: TestError to get un-deployed
2021/12/03 13:03:29 App: TestError got un-deployed
2021/12/03 13:03:29 No eventing-con process running
2021/12/03 13:03:29 Function: TestError delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestError deleting in the background"
}
2021/12/03 13:03:31 Context: Bucket flush, response code: 200 dump: 
2021/12/03 13:03:41 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/03 13:03:46 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/03 13:03:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 13:03:48 Context: Bucket flush, response code: 200 dump: 
2021/12/03 13:03:58 Waiting for dst bucket item count to get to: 0 curr count: 189
2021/12/03 13:04:03 Waiting for dst bucket item count to get to: 0 curr count: 45
2021/12/03 13:04:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestError (108.86s)
=== RUN   TestCRLF
2021/12/03 13:04:13 Waiting for app: TestCRLF to get un-deployed
2021/12/03 13:04:13 App: TestCRLF got un-deployed
2021/12/03 13:04:13 No eventing-con process running
2021/12/03 13:04:13 Function: TestCRLF delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestCRLF not found"
 }
}
2021/12/03 13:04:15 Context: Bucket flush, response code: 200 dump: 
2021/12/03 13:04:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 13:04:22 Context: Bucket flush, response code: 200 dump: 
2021/12/03 13:04:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 13:04:27 Deploying app: TestCRLF. Item count src bucket: 0 dst bucket: 0
2021/12/03 13:04:28 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCRLF' in metakv",
  "warnings": null
 }
}
2021/12/03 13:04:33 Waiting for app: TestCRLF to get deployed
2021/12/03 13:04:38 Waiting for app: TestCRLF to get deployed
2021/12/03 13:04:43 Waiting for app: TestCRLF to get deployed
2021/12/03 13:04:48 Waiting for app: TestCRLF to get deployed
2021/12/03 13:04:48 App: TestCRLF got deployed
2021/12/03 13:04:48 Starting bucket ops to source bucket
2021/12/03 13:04:53 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/03 13:04:58 Waiting for dst bucket item count to get to: 100 curr count: 52
2021/12/03 13:05:03 Waiting for dst bucket item count to get to: 100 curr count: 52
2021/12/03 13:05:03 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/03 13:05:03 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 13:05:03 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 13:05:08 Waiting for app: TestCRLF to get un-deployed
2021/12/03 13:05:13 Waiting for app: TestCRLF to get un-deployed
2021/12/03 13:05:18 Waiting for app: TestCRLF to get un-deployed
2021/12/03 13:05:23 Waiting for app: TestCRLF to get un-deployed
2021/12/03 13:05:23 App: TestCRLF got un-deployed
2021/12/03 13:05:23 No eventing-con process running
2021/12/03 13:05:23 Function: TestCRLF delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCRLF deleting in the background"
}
2021/12/03 13:05:25 Context: Bucket flush, response code: 200 dump: 
2021/12/03 13:05:35 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/03 13:05:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 13:05:37 Context: Bucket flush, response code: 200 dump: 
2021/12/03 13:05:47 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/03 13:05:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCRLF (104.06s)
=== RUN   TestImportExport
2021/12/03 13:05:57 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/03 13:05:57 App: TestImportExport_n1ql_insert_on_update got un-deployed
2021/12/03 13:05:57 No eventing-con process running
2021/12/03 13:05:57 Function: TestImportExport_n1ql_insert_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestImportExport_n1ql_insert_on_update not found"
 }
}
2021/12/03 13:05:59 Context: Bucket flush, response code: 200 dump: 
2021/12/03 13:06:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 13:06:06 Context: Bucket flush, response code: 200 dump: 
2021/12/03 13:06:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 13:06:16 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/03 13:06:16 App: TestImportExport_bucket_op_on_update got un-deployed
2021/12/03 13:06:16 No eventing-con process running
2021/12/03 13:06:16 Function: TestImportExport_bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestImportExport_bucket_op_on_update not found"
 }
}
2021/12/03 13:06:17 Context: Bucket flush, response code: 200 dump: 
2021/12/03 13:06:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 13:06:24 Context: Bucket flush, response code: 200 dump: 
2021/12/03 13:06:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 13:06:29 Deploying app: TestImportExport_n1ql_insert_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/03 13:06:30 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestImportExport_n1ql_insert_on_update' in metakv",
  "warnings": null
 }
}
2021/12/03 13:06:30 Deploying app: TestImportExport_bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/03 13:06:32 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestImportExport_bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/03 13:06:37 Waiting for app: TestImportExport_n1ql_insert_on_update to get deployed
2021/12/03 13:06:42 Waiting for app: TestImportExport_n1ql_insert_on_update to get deployed
2021/12/03 13:06:47 Waiting for app: TestImportExport_n1ql_insert_on_update to get deployed
2021/12/03 13:06:52 Waiting for app: TestImportExport_n1ql_insert_on_update to get deployed
2021/12/03 13:06:52 App: TestImportExport_n1ql_insert_on_update got deployed
2021/12/03 13:06:57 Waiting for app: TestImportExport_bucket_op_on_update to get deployed
2021/12/03 13:07:02 Waiting for app: TestImportExport_bucket_op_on_update to get deployed
2021/12/03 13:07:07 Waiting for app: TestImportExport_bucket_op_on_update to get deployed
2021/12/03 13:07:07 App: TestImportExport_bucket_op_on_update got deployed
2021/12/03 13:07:07 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 13:07:12 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/03 13:07:17 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/03 13:07:22 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/03 13:07:27 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/03 13:07:27 App: TestImportExport_n1ql_insert_on_update got un-deployed
2021/12/03 13:07:27 No eventing-con process running
2021/12/03 13:07:27 Function: TestImportExport_n1ql_insert_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestImportExport_n1ql_insert_on_update deleting in the background"
}
2021/12/03 13:07:27 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 13:07:32 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/03 13:07:37 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/03 13:07:42 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/03 13:07:47 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/03 13:07:47 App: TestImportExport_bucket_op_on_update got un-deployed
2021/12/03 13:07:47 No eventing-con process running
2021/12/03 13:07:47 Function: TestImportExport_bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestImportExport_bucket_op_on_update deleting in the background"
}
2021/12/03 13:08:04 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/03 13:08:04 App: TestImportExport_bucket_op_on_update got un-deployed
2021/12/03 13:08:04 No eventing-con process running
2021/12/03 13:08:04 Function: TestImportExport_bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestImportExport_bucket_op_on_update deleting in the background"
}
2021/12/03 13:08:06 Context: Bucket flush, response code: 200 dump: 
2021/12/03 13:08:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 13:08:13 Context: Bucket flush, response code: 200 dump: 
2021/12/03 13:08:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 13:08:23 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/03 13:08:23 App: TestImportExport_n1ql_insert_on_update got un-deployed
2021/12/03 13:08:23 No eventing-con process running
2021/12/03 13:08:23 Function: TestImportExport_n1ql_insert_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestImportExport_n1ql_insert_on_update deleting in the background"
}
2021/12/03 13:08:25 Context: Bucket flush, response code: 200 dump: 
2021/12/03 13:08:30 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 13:08:32 Context: Bucket flush, response code: 200 dump: 
2021/12/03 13:08:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestImportExport (169.99s)
=== RUN   TestDeployUndeployLoopNonDefaultSettings
2021/12/03 13:08:47 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/03 13:08:47 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/03 13:08:47 No eventing-con process running
2021/12/03 13:08:47 Function: TestDeployUndeployLoopNonDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDeployUndeployLoopNonDefaultSettings not found"
 }
}
2021/12/03 13:08:49 Context: Bucket flush, response code: 200 dump: 
2021/12/03 13:08:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 13:08:56 Context: Bucket flush, response code: 200 dump: 
2021/12/03 13:09:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 13:09:01 Deploying app: TestDeployUndeployLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/03 13:09:02 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/03 13:09:02 Starting bucket ops to source bucket
2021/12/03 13:09:10 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 13:09:15 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 13:09:20 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 13:09:25 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 13:09:25 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/03 13:09:30 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get deployed
2021/12/03 13:09:30 App: TestDeployUndeployLoopNonDefaultSettings got deployed
2021/12/03 13:09:30 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 13:09:30 Undeploying app: bucket_op_on_update
2021/12/03 13:09:30 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 13:09:35 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/03 13:09:40 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/03 13:09:45 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/03 13:09:50 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/03 13:09:50 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/03 13:09:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 13:09:50 No eventing-con process running
2021/12/03 13:09:52 Context: Bucket flush, response code: 200 dump: 
2021/12/03 13:09:59 Context: Bucket flush, response code: 200 dump: 
2021/12/03 13:10:34 Deploying app: TestDeployUndeployLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/03 13:10:35 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/03 13:10:35 Starting bucket ops to source bucket
2021/12/03 13:10:43 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 13:10:48 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 13:10:53 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 13:10:58 Waiting for dst bucket item count to get to: 5000 curr count: 2530
2021/12/03 13:11:03 Waiting for dst bucket item count to get to: 5000 curr count: 2530
2021/12/03 13:11:03 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/03 13:11:08 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get deployed
2021/12/03 13:11:08 App: TestDeployUndeployLoopNonDefaultSettings got deployed
2021/12/03 13:11:08 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 13:11:08 Undeploying app: bucket_op_on_update
2021/12/03 13:11:09 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 13:11:14 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/03 13:11:19 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/03 13:11:24 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/03 13:11:29 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/03 13:11:29 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/03 13:11:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 13:11:29 No eventing-con process running
2021/12/03 13:11:30 Context: Bucket flush, response code: 200 dump: 
2021/12/03 13:11:38 Context: Bucket flush, response code: 200 dump: 
2021/12/03 13:12:13 Deploying app: TestDeployUndeployLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/03 13:12:14 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/03 13:12:14 Starting bucket ops to source bucket
2021/12/03 13:12:22 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 13:12:27 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 13:12:32 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 13:12:37 Waiting for dst bucket item count to get to: 5000 curr count: 1764
2021/12/03 13:12:42 Waiting for dst bucket item count to get to: 5000 curr count: 4264
2021/12/03 13:12:42 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/03 13:12:47 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get deployed
2021/12/03 13:12:47 App: TestDeployUndeployLoopNonDefaultSettings got deployed
2021/12/03 13:12:47 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 13:12:47 Undeploying app: bucket_op_on_update
2021/12/03 13:12:47 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 13:12:52 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/03 13:12:57 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/03 13:13:02 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/03 13:13:07 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/03 13:13:07 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/03 13:13:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 13:13:07 No eventing-con process running
2021/12/03 13:13:09 Context: Bucket flush, response code: 200 dump: 
2021/12/03 13:13:16 Context: Bucket flush, response code: 200 dump: 
2021/12/03 13:13:51 Deploying app: TestDeployUndeployLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/03 13:13:52 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/03 13:13:52 Starting bucket ops to source bucket
2021/12/03 13:14:00 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 13:14:05 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 13:14:10 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 13:14:15 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 13:14:15 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/03 13:14:20 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get deployed
2021/12/03 13:14:20 App: TestDeployUndeployLoopNonDefaultSettings got deployed
2021/12/03 13:14:20 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 13:14:20 Undeploying app: bucket_op_on_update
2021/12/03 13:14:20 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 13:14:25 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/03 13:14:30 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/03 13:14:35 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/03 13:14:40 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/03 13:14:40 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/03 13:14:41 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 13:14:41 No eventing-con process running
2021/12/03 13:14:42 Context: Bucket flush, response code: 200 dump: 
2021/12/03 13:14:50 Context: Bucket flush, response code: 200 dump: 
2021/12/03 13:15:25 Deploying app: TestDeployUndeployLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/03 13:15:26 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/03 13:15:26 Starting bucket ops to source bucket
2021/12/03 13:15:34 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 13:15:39 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 13:15:44 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 13:15:49 Waiting for dst bucket item count to get to: 5000 curr count: 2294
2021/12/03 13:15:54 Waiting for dst bucket item count to get to: 5000 curr count: 2294
2021/12/03 13:15:54 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/03 13:15:59 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get deployed
2021/12/03 13:15:59 App: TestDeployUndeployLoopNonDefaultSettings got deployed
2021/12/03 13:15:59 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 13:15:59 Undeploying app: bucket_op_on_update
2021/12/03 13:15:59 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 13:16:04 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/03 13:16:09 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/03 13:16:14 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/03 13:16:19 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/03 13:16:19 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/03 13:16:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 13:16:19 No eventing-con process running
2021/12/03 13:16:21 Context: Bucket flush, response code: 200 dump: 
2021/12/03 13:16:28 Context: Bucket flush, response code: 200 dump: 
2021/12/03 13:17:03 Function: TestDeployUndeployLoopNonDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDeployUndeployLoopNonDefaultSettings deleting in the background"
}
--- PASS: TestDeployUndeployLoopNonDefaultSettings (506.12s)
=== RUN   TestOnUpdateN1QLOp
2021/12/03 13:17:13 Waiting for app: TestOnUpdateN1QLOp to get un-deployed
2021/12/03 13:17:13 App: TestOnUpdateN1QLOp got un-deployed
2021/12/03 13:17:13 No eventing-con process running
2021/12/03 13:17:13 Function: TestOnUpdateN1QLOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateN1QLOp not found"
 }
}
2021/12/03 13:17:15 Context: Bucket flush, response code: 200 dump: 
2021/12/03 13:17:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 13:17:22 Context: Bucket flush, response code: 200 dump: 
2021/12/03 13:17:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 13:17:27 Deploying app: TestOnUpdateN1QLOp. Item count src bucket: 0 dst bucket: 0
2021/12/03 13:17:28 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateN1QLOp' in metakv",
  "warnings": null
 }
}
2021/12/03 13:17:28 Starting bucket ops to source bucket
2021/12/03 13:17:36 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 13:17:41 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 13:17:46 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 13:17:51 Waiting for dst bucket item count to get to: 5000 curr count: 54
2021/12/03 13:17:56 Waiting for dst bucket item count to get to: 5000 curr count: 54
2021/12/03 13:17:56 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/03 13:17:56 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 13:17:56 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 13:18:01 Waiting for app: TestOnUpdateN1QLOp to get un-deployed
2021/12/03 13:18:06 Waiting for app: TestOnUpdateN1QLOp to get un-deployed
2021/12/03 13:18:11 Waiting for app: TestOnUpdateN1QLOp to get un-deployed
2021/12/03 13:18:16 Waiting for app: TestOnUpdateN1QLOp to get un-deployed
2021/12/03 13:18:16 App: TestOnUpdateN1QLOp got un-deployed
2021/12/03 13:18:16 No eventing-con process running
2021/12/03 13:18:16 Function: TestOnUpdateN1QLOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateN1QLOp deleting in the background"
}
2021/12/03 13:18:18 Context: Bucket flush, response code: 200 dump: 
2021/12/03 13:18:28 Waiting for dst bucket item count to get to: 0 curr count: 3196
2021/12/03 13:18:33 Waiting for dst bucket item count to get to: 0 curr count: 696
2021/12/03 13:18:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 13:18:35 Context: Bucket flush, response code: 200 dump: 
2021/12/03 13:18:45 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/03 13:18:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateN1QLOp (101.84s)
=== RUN   TestOnUpdateBucketOpDefaultSettings
2021/12/03 13:18:55 Waiting for app: TestOnUpdateBucketOpDefaultSettings to get un-deployed
2021/12/03 13:18:55 App: TestOnUpdateBucketOpDefaultSettings got un-deployed
2021/12/03 13:18:55 No eventing-con process running
2021/12/03 13:18:55 Function: TestOnUpdateBucketOpDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateBucketOpDefaultSettings not found"
 }
}
2021/12/03 13:18:57 Context: Bucket flush, response code: 200 dump: 
2021/12/03 13:19:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 13:19:04 Context: Bucket flush, response code: 200 dump: 
2021/12/03 13:19:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 13:19:09 Deploying app: TestOnUpdateBucketOpDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/03 13:19:10 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateBucketOpDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/03 13:19:10 Starting bucket ops to source bucket
2021/12/03 13:19:18 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 13:19:23 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 13:19:28 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 13:19:33 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 13:19:38 Waiting for dst bucket item count to get to: 5000 curr count: 2500
2021/12/03 13:19:38 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/03 13:19:38 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 13:19:38 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 13:19:43 Waiting for app: TestOnUpdateBucketOpDefaultSettings to get un-deployed
2021/12/03 13:19:48 Waiting for app: TestOnUpdateBucketOpDefaultSettings to get un-deployed
2021/12/03 13:19:53 Waiting for app: TestOnUpdateBucketOpDefaultSettings to get un-deployed
2021/12/03 13:19:58 Waiting for app: TestOnUpdateBucketOpDefaultSettings to get un-deployed
2021/12/03 13:19:58 App: TestOnUpdateBucketOpDefaultSettings got un-deployed
2021/12/03 13:19:58 No eventing-con process running
2021/12/03 13:19:58 Function: TestOnUpdateBucketOpDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateBucketOpDefaultSettings deleting in the background"
}
2021/12/03 13:20:00 Context: Bucket flush, response code: 200 dump: 
2021/12/03 13:20:10 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/03 13:20:15 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/03 13:20:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 13:20:17 Context: Bucket flush, response code: 200 dump: 
2021/12/03 13:20:27 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/03 13:20:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateBucketOpDefaultSettings (101.82s)
=== RUN   TestOnUpdateBucketOpNonDefaultSettings
2021/12/03 13:20:37 Waiting for app: TestOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/03 13:20:37 App: TestOnUpdateBucketOpNonDefaultSettings got un-deployed
2021/12/03 13:20:37 No eventing-con process running
2021/12/03 13:20:37 Function: TestOnUpdateBucketOpNonDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateBucketOpNonDefaultSettings not found"
 }
}
2021/12/03 13:20:39 Context: Bucket flush, response code: 200 dump: 
2021/12/03 13:20:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 13:20:45 Context: Bucket flush, response code: 200 dump: 
2021/12/03 13:20:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 13:20:50 Deploying app: TestOnUpdateBucketOpNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/03 13:20:51 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateBucketOpNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/03 13:20:51 Starting bucket ops to source bucket
2021/12/03 13:21:00 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 13:21:05 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 13:21:10 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 13:21:15 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 13:21:15 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/03 13:21:15 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 13:21:15 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 13:21:20 Waiting for app: TestOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/03 13:21:25 Waiting for app: TestOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/03 13:21:30 Waiting for app: TestOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/03 13:21:35 Waiting for app: TestOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/03 13:21:35 App: TestOnUpdateBucketOpNonDefaultSettings got un-deployed
2021/12/03 13:21:35 No eventing-con process running
2021/12/03 13:21:35 Function: TestOnUpdateBucketOpNonDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateBucketOpNonDefaultSettings deleting in the background"
}
2021/12/03 13:21:37 Context: Bucket flush, response code: 200 dump: 
2021/12/03 13:21:47 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/03 13:21:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 13:21:48 Context: Bucket flush, response code: 200 dump: 
2021/12/03 13:21:59 Waiting for dst bucket item count to get to: 0 curr count: 4328
2021/12/03 13:22:04 Waiting for dst bucket item count to get to: 0 curr count: 1828
2021/12/03 13:22:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateBucketOpNonDefaultSettings (96.87s)
=== RUN   TestOnUpdateBucketOpDefaultSettings10K
2021/12/03 13:22:14 Waiting for app: TestOnUpdateBucketOpDefaultSettings10K to get un-deployed
2021/12/03 13:22:14 App: TestOnUpdateBucketOpDefaultSettings10K got un-deployed
2021/12/03 13:22:14 No eventing-con process running
2021/12/03 13:22:14 Function: TestOnUpdateBucketOpDefaultSettings10K delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateBucketOpDefaultSettings10K not found"
 }
}
2021/12/03 13:22:15 Context: Bucket flush, response code: 200 dump: 
2021/12/03 13:22:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 13:22:22 Context: Bucket flush, response code: 200 dump: 
2021/12/03 13:22:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 13:22:27 Deploying app: TestOnUpdateBucketOpDefaultSettings10K. Item count src bucket: 0 dst bucket: 0
2021/12/03 13:22:28 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateBucketOpDefaultSettings10K' in metakv",
  "warnings": null
 }
}
2021/12/03 13:22:28 Starting bucket ops to source bucket
2021/12/03 13:22:36 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 13:22:41 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 13:22:46 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 13:22:52 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 13:22:57 Waiting for dst bucket item count to get to: 5000 curr count: 2500
2021/12/03 13:22:57 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/03 13:22:57 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 13:22:57 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 13:23:02 Waiting for app: TestOnUpdateBucketOpDefaultSettings10K to get un-deployed
2021/12/03 13:23:07 Waiting for app: TestOnUpdateBucketOpDefaultSettings10K to get un-deployed
2021/12/03 13:23:12 Waiting for app: TestOnUpdateBucketOpDefaultSettings10K to get un-deployed
2021/12/03 13:23:17 Waiting for app: TestOnUpdateBucketOpDefaultSettings10K to get un-deployed
2021/12/03 13:23:17 App: TestOnUpdateBucketOpDefaultSettings10K got un-deployed
2021/12/03 13:23:17 No eventing-con process running
2021/12/03 13:23:17 Function: TestOnUpdateBucketOpDefaultSettings10K delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateBucketOpDefaultSettings10K deleting in the background"
}
2021/12/03 13:23:19 Context: Bucket flush, response code: 200 dump: 
2021/12/03 13:23:29 Waiting for dst bucket item count to get to: 0 curr count: 4270
2021/12/03 13:23:34 Waiting for dst bucket item count to get to: 0 curr count: 1770
2021/12/03 13:23:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 13:23:35 Context: Bucket flush, response code: 200 dump: 
2021/12/03 13:23:45 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/03 13:23:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateBucketOpDefaultSettings10K (101.89s)
=== RUN   TestOnUpdateBucketOpDefaultSettings100K
2021/12/03 13:23:55 Waiting for app: TestOnUpdateBucketOpDefaultSettings100K to get un-deployed
2021/12/03 13:23:55 App: TestOnUpdateBucketOpDefaultSettings100K got un-deployed
2021/12/03 13:23:55 No eventing-con process running
2021/12/03 13:23:55 Function: TestOnUpdateBucketOpDefaultSettings100K delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateBucketOpDefaultSettings100K not found"
 }
}
2021/12/03 13:23:57 Context: Bucket flush, response code: 200 dump: 
2021/12/03 13:24:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 13:24:04 Context: Bucket flush, response code: 200 dump: 
2021/12/03 13:24:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 13:24:09 Deploying app: TestOnUpdateBucketOpDefaultSettings100K. Item count src bucket: 0 dst bucket: 0
2021/12/03 13:24:11 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateBucketOpDefaultSettings100K' in metakv",
  "warnings": null
 }
}
2021/12/03 13:24:11 Starting bucket ops to source bucket
2021/12/03 13:24:19 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 13:24:24 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 13:24:29 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 13:24:34 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 13:24:34 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/03 13:24:34 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 13:24:34 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 13:24:39 Waiting for app: TestOnUpdateBucketOpDefaultSettings100K to get un-deployed
2021/12/03 13:24:44 Waiting for app: TestOnUpdateBucketOpDefaultSettings100K to get un-deployed
2021/12/03 13:24:49 Waiting for app: TestOnUpdateBucketOpDefaultSettings100K to get un-deployed
2021/12/03 13:24:54 Waiting for app: TestOnUpdateBucketOpDefaultSettings100K to get un-deployed
2021/12/03 13:24:54 App: TestOnUpdateBucketOpDefaultSettings100K got un-deployed
2021/12/03 13:24:54 No eventing-con process running
2021/12/03 13:24:54 Function: TestOnUpdateBucketOpDefaultSettings100K delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateBucketOpDefaultSettings100K deleting in the background"
}
2021/12/03 13:24:56 Context: Bucket flush, response code: 200 dump: 
2021/12/03 13:25:06 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/03 13:25:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 13:25:08 Context: Bucket flush, response code: 200 dump: 
2021/12/03 13:25:18 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/03 13:25:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateBucketOpDefaultSettings100K (92.22s)
=== RUN   TestOnDeleteBucketOp
2021/12/03 13:25:28 Waiting for app: TestOnDeleteBucketOp to get un-deployed
2021/12/03 13:25:28 App: TestOnDeleteBucketOp got un-deployed
2021/12/03 13:25:28 No eventing-con process running
2021/12/03 13:25:28 Function: TestOnDeleteBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnDeleteBucketOp not found"
 }
}
2021/12/03 13:25:29 Context: Bucket flush, response code: 200 dump: 
2021/12/03 13:25:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 13:25:36 Context: Bucket flush, response code: 200 dump: 
2021/12/03 13:25:41 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 13:25:41 Deploying app: TestOnDeleteBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/03 13:25:42 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnDeleteBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/03 13:25:42 Starting bucket ops to source bucket
2021/12/03 13:25:53 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 13:25:58 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 13:26:03 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 13:26:03 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/03 13:26:03 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 13:26:03 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 13:26:08 Waiting for app: TestOnDeleteBucketOp to get un-deployed
2021/12/03 13:26:13 Waiting for app: TestOnDeleteBucketOp to get un-deployed
2021/12/03 13:26:18 Waiting for app: TestOnDeleteBucketOp to get un-deployed
2021/12/03 13:26:23 Waiting for app: TestOnDeleteBucketOp to get un-deployed
2021/12/03 13:26:23 App: TestOnDeleteBucketOp got un-deployed
2021/12/03 13:26:23 No eventing-con process running
2021/12/03 13:26:23 Function: TestOnDeleteBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnDeleteBucketOp deleting in the background"
}
2021/12/03 13:26:25 Context: Bucket flush, response code: 200 dump: 
2021/12/03 13:26:30 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 13:26:33 Context: Bucket flush, response code: 200 dump: 
2021/12/03 13:26:43 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/03 13:26:48 Waiting for dst bucket item count to get to: 0 curr count: 2496
2021/12/03 13:26:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnDeleteBucketOp (90.12s)
=== RUN   TestOnDeleteBucketOp5K
2021/12/03 13:26:58 Waiting for app: TestOnDeleteBucketOp5K to get un-deployed
2021/12/03 13:26:58 App: TestOnDeleteBucketOp5K got un-deployed
2021/12/03 13:26:58 No eventing-con process running
2021/12/03 13:26:58 Function: TestOnDeleteBucketOp5K delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnDeleteBucketOp5K not found"
 }
}
2021/12/03 13:27:00 Context: Bucket flush, response code: 200 dump: 
2021/12/03 13:27:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 13:27:06 Context: Bucket flush, response code: 200 dump: 
2021/12/03 13:27:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 13:27:11 Deploying app: TestOnDeleteBucketOp5K. Item count src bucket: 0 dst bucket: 0
2021/12/03 13:27:13 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnDeleteBucketOp5K' in metakv",
  "warnings": null
 }
}
2021/12/03 13:27:13 Starting bucket ops to source bucket
2021/12/03 13:27:24 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 13:27:29 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 13:27:34 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 13:27:34 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/03 13:27:34 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 13:27:34 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 13:27:39 Waiting for app: TestOnDeleteBucketOp5K to get un-deployed
2021/12/03 13:27:44 Waiting for app: TestOnDeleteBucketOp5K to get un-deployed
2021/12/03 13:27:49 Waiting for app: TestOnDeleteBucketOp5K to get un-deployed
2021/12/03 13:27:54 Waiting for app: TestOnDeleteBucketOp5K to get un-deployed
2021/12/03 13:27:54 App: TestOnDeleteBucketOp5K got un-deployed
2021/12/03 13:27:54 No eventing-con process running
2021/12/03 13:27:54 Function: TestOnDeleteBucketOp5K delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnDeleteBucketOp5K deleting in the background"
}
2021/12/03 13:27:55 Context: Bucket flush, response code: 200 dump: 
2021/12/03 13:28:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 13:28:03 Context: Bucket flush, response code: 200 dump: 
2021/12/03 13:28:13 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/03 13:28:18 Waiting for dst bucket item count to get to: 0 curr count: 2496
2021/12/03 13:28:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnDeleteBucketOp5K (90.12s)
=== RUN   TestTimerBucketOp
2021/12/03 13:28:28 Waiting for app: TestTimerBucketOp to get un-deployed
2021/12/03 13:28:28 App: TestTimerBucketOp got un-deployed
2021/12/03 13:28:28 No eventing-con process running
2021/12/03 13:28:28 Function: TestTimerBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestTimerBucketOp not found"
 }
}
2021/12/03 13:28:30 Context: Bucket flush, response code: 200 dump: 
2021/12/03 13:28:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 13:28:36 Context: Bucket flush, response code: 200 dump: 
2021/12/03 13:28:41 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 13:28:41 Deploying app: TestTimerBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/03 13:28:43 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestTimerBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/03 13:28:43 Starting bucket ops to source bucket
2021/12/03 13:28:51 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 13:28:56 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 13:29:01 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 13:29:06 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 13:29:11 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 13:29:16 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 13:29:21 Waiting for dst bucket item count to get to: 5000 curr count: 313
2021/12/03 13:29:26 Waiting for dst bucket item count to get to: 5000 curr count: 313
2021/12/03 13:29:26 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/03 13:29:26 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 13:29:26 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 13:29:31 Waiting for app: TestTimerBucketOp to get un-deployed
2021/12/03 13:29:36 Waiting for app: TestTimerBucketOp to get un-deployed
2021/12/03 13:29:41 Waiting for app: TestTimerBucketOp to get un-deployed
2021/12/03 13:29:46 Waiting for app: TestTimerBucketOp to get un-deployed
2021/12/03 13:29:46 App: TestTimerBucketOp got un-deployed
2021/12/03 13:29:46 No eventing-con process running
2021/12/03 13:29:46 Function: TestTimerBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestTimerBucketOp deleting in the background"
}
2021/12/03 13:29:48 Context: Bucket flush, response code: 200 dump: 
2021/12/03 13:29:58 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/03 13:29:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 13:29:59 Context: Bucket flush, response code: 200 dump: 
2021/12/03 13:30:09 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/03 13:30:14 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/03 13:30:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestTimerBucketOp (116.59s)
=== RUN   TestTimerInPastBucketOp
2021/12/03 13:30:24 Waiting for app: TestTimerInPastBucketOp to get un-deployed
2021/12/03 13:30:24 App: TestTimerInPastBucketOp got un-deployed
2021/12/03 13:30:24 No eventing-con process running
2021/12/03 13:30:24 Function: TestTimerInPastBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestTimerInPastBucketOp not found"
 }
}
2021/12/03 13:30:26 Context: Bucket flush, response code: 200 dump: 
2021/12/03 13:30:31 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 13:30:33 Context: Bucket flush, response code: 200 dump: 
2021/12/03 13:30:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 13:30:38 Deploying app: TestTimerInPastBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/03 13:30:39 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestTimerInPastBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/03 13:30:39 Starting bucket ops to source bucket
2021/12/03 13:30:47 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 13:30:52 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 13:30:57 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 13:31:02 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 13:31:07 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 13:31:12 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 13:31:17 Waiting for dst bucket item count to get to: 5000 curr count: 1512
2021/12/03 13:31:22 Waiting for dst bucket item count to get to: 5000 curr count: 4012
2021/12/03 13:31:22 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/03 13:31:22 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 13:31:22 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 13:31:27 Waiting for app: TestTimerInPastBucketOp to get un-deployed
2021/12/03 13:31:33 Waiting for app: TestTimerInPastBucketOp to get un-deployed
2021/12/03 13:31:38 Waiting for app: TestTimerInPastBucketOp to get un-deployed
2021/12/03 13:31:43 Waiting for app: TestTimerInPastBucketOp to get un-deployed
2021/12/03 13:31:43 App: TestTimerInPastBucketOp got un-deployed
2021/12/03 13:31:43 No eventing-con process running
2021/12/03 13:31:43 Function: TestTimerInPastBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestTimerInPastBucketOp deleting in the background"
}
2021/12/03 13:31:44 Context: Bucket flush, response code: 200 dump: 
2021/12/03 13:31:54 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/03 13:31:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 13:31:56 Context: Bucket flush, response code: 200 dump: 
2021/12/03 13:32:06 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/03 13:32:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestTimerInPastBucketOp (111.66s)
=== RUN   TestDeployUndeployLoopTimer
2021/12/03 13:32:16 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/03 13:32:16 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/03 13:32:16 No eventing-con process running
2021/12/03 13:32:16 Function: TestDeployUndeployLoopTimer delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDeployUndeployLoopTimer not found"
 }
}
2021/12/03 13:32:18 Context: Bucket flush, response code: 200 dump: 
2021/12/03 13:32:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 13:32:25 Context: Bucket flush, response code: 200 dump: 
2021/12/03 13:32:30 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 13:32:30 Deploying app: TestDeployUndeployLoopTimer. Item count src bucket: 0 dst bucket: 0
2021/12/03 13:32:31 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopTimer' in metakv",
  "warnings": null
 }
}
2021/12/03 13:32:31 Starting bucket ops to source bucket
2021/12/03 13:32:39 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 13:32:44 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 13:32:49 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 13:32:54 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 13:32:59 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 13:33:04 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 13:33:09 Waiting for dst bucket item count to get to: 5000 curr count: 2011
2021/12/03 13:33:14 Waiting for dst bucket item count to get to: 5000 curr count: 2011
2021/12/03 13:33:14 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/03 13:33:19 Waiting for app: TestDeployUndeployLoopTimer to get deployed
2021/12/03 13:33:19 App: TestDeployUndeployLoopTimer got deployed
2021/12/03 13:33:19 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 13:33:19 Undeploying app: bucket_op_with_timer
2021/12/03 13:33:19 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 13:33:24 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/03 13:33:29 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/03 13:33:34 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/03 13:33:39 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/03 13:33:39 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/03 13:33:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 13:33:39 No eventing-con process running
2021/12/03 13:33:41 Context: Bucket flush, response code: 200 dump: 
2021/12/03 13:33:48 Context: Bucket flush, response code: 200 dump: 
2021/12/03 13:34:23 Deploying app: TestDeployUndeployLoopTimer. Item count src bucket: 0 dst bucket: 0
2021/12/03 13:34:24 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopTimer' in metakv",
  "warnings": null
 }
}
2021/12/03 13:34:24 Starting bucket ops to source bucket
2021/12/03 13:34:32 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 13:34:37 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 13:34:42 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 13:34:47 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 13:34:52 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 13:34:57 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 13:35:02 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 13:35:07 Waiting for dst bucket item count to get to: 5000 curr count: 2500
2021/12/03 13:35:07 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/03 13:35:12 Waiting for app: TestDeployUndeployLoopTimer to get deployed
2021/12/03 13:35:12 App: TestDeployUndeployLoopTimer got deployed
2021/12/03 13:35:12 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 13:35:12 Undeploying app: bucket_op_with_timer
2021/12/03 13:35:13 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 13:35:18 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/03 13:35:23 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/03 13:35:28 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/03 13:35:33 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/03 13:35:33 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/03 13:35:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 13:35:33 No eventing-con process running
2021/12/03 13:35:34 Context: Bucket flush, response code: 200 dump: 
2021/12/03 13:35:42 Context: Bucket flush, response code: 200 dump: 
2021/12/03 13:36:17 Deploying app: TestDeployUndeployLoopTimer. Item count src bucket: 0 dst bucket: 0
2021/12/03 13:36:18 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopTimer' in metakv",
  "warnings": null
 }
}
2021/12/03 13:36:18 Starting bucket ops to source bucket
2021/12/03 13:36:26 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 13:36:31 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 13:36:36 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 13:36:41 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 13:36:46 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 13:36:51 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 13:36:56 Waiting for dst bucket item count to get to: 5000 curr count: 2500
2021/12/03 13:37:01 Waiting for dst bucket item count to get to: 5000 curr count: 4828
2021/12/03 13:37:06 Waiting for dst bucket item count to get to: 5000 curr count: 4828
2021/12/03 13:37:06 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/03 13:37:11 Waiting for app: TestDeployUndeployLoopTimer to get deployed
2021/12/03 13:37:11 App: TestDeployUndeployLoopTimer got deployed
2021/12/03 13:37:11 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 13:37:11 Undeploying app: bucket_op_with_timer
2021/12/03 13:37:11 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 13:37:16 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/03 13:37:21 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/03 13:37:26 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/03 13:37:31 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/03 13:37:31 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/03 13:37:31 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 13:37:31 No eventing-con process running
2021/12/03 13:37:33 Context: Bucket flush, response code: 200 dump: 
2021/12/03 13:37:40 Context: Bucket flush, response code: 200 dump: 
2021/12/03 13:38:15 Deploying app: TestDeployUndeployLoopTimer. Item count src bucket: 0 dst bucket: 0
2021/12/03 13:38:16 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopTimer' in metakv",
  "warnings": null
 }
}
2021/12/03 13:38:16 Starting bucket ops to source bucket
2021/12/03 13:38:24 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 13:38:29 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 13:38:34 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 13:38:39 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 13:38:44 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 13:38:49 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 13:38:54 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 13:38:54 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/03 13:38:59 Waiting for app: TestDeployUndeployLoopTimer to get deployed
2021/12/03 13:38:59 App: TestDeployUndeployLoopTimer got deployed
2021/12/03 13:39:00 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 13:39:00 Undeploying app: bucket_op_with_timer
2021/12/03 13:39:00 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 13:39:05 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/03 13:39:10 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/03 13:39:15 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/03 13:39:20 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/03 13:39:20 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/03 13:39:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 13:39:20 No eventing-con process running
2021/12/03 13:39:21 Context: Bucket flush, response code: 200 dump: 
2021/12/03 13:39:29 Context: Bucket flush, response code: 200 dump: 
2021/12/03 13:40:04 Deploying app: TestDeployUndeployLoopTimer. Item count src bucket: 0 dst bucket: 0
2021/12/03 13:40:05 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopTimer' in metakv",
  "warnings": null
 }
}
2021/12/03 13:40:05 Starting bucket ops to source bucket
2021/12/03 13:40:13 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 13:40:18 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 13:40:23 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 13:40:28 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 13:40:33 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 13:40:38 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 13:40:43 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 13:40:43 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/03 13:40:48 Waiting for app: TestDeployUndeployLoopTimer to get deployed
2021/12/03 13:40:48 App: TestDeployUndeployLoopTimer got deployed
2021/12/03 13:40:48 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 13:40:48 Undeploying app: bucket_op_with_timer
2021/12/03 13:40:48 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 13:40:53 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/03 13:40:58 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/03 13:41:03 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/03 13:41:08 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/03 13:41:08 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/03 13:41:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 13:41:08 No eventing-con process running
2021/12/03 13:41:10 Context: Bucket flush, response code: 200 dump: 
2021/12/03 13:41:18 Context: Bucket flush, response code: 200 dump: 
2021/12/03 13:41:53 Function: TestDeployUndeployLoopTimer delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDeployUndeployLoopTimer deleting in the background"
}
--- PASS: TestDeployUndeployLoopTimer (586.46s)
=== RUN   TestMultipleHandlers
2021/12/03 13:42:03 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get un-deployed
2021/12/03 13:42:03 App: TestMultipleHandlers_bucket_op_on_update got un-deployed
2021/12/03 13:42:03 No eventing-con process running
2021/12/03 13:42:03 Function: TestMultipleHandlers_bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMultipleHandlers_bucket_op_on_update not found"
 }
}
2021/12/03 13:42:04 Context: Bucket flush, response code: 200 dump: 
2021/12/03 13:42:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 13:42:11 Context: Bucket flush, response code: 200 dump: 
2021/12/03 13:42:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 13:42:21 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get un-deployed
2021/12/03 13:42:21 App: TestMultipleHandlers_bucket_op_on_delete got un-deployed
2021/12/03 13:42:21 No eventing-con process running
2021/12/03 13:42:21 Function: TestMultipleHandlers_bucket_op_on_delete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMultipleHandlers_bucket_op_on_delete not found"
 }
}
2021/12/03 13:42:23 Context: Bucket flush, response code: 200 dump: 
2021/12/03 13:42:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 13:42:30 Context: Bucket flush, response code: 200 dump: 
2021/12/03 13:42:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 13:42:35 Deploying app: TestMultipleHandlers_bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/03 13:42:36 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMultipleHandlers_bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/03 13:42:36 Deploying app: TestMultipleHandlers_bucket_op_on_delete. Item count src bucket: 0 dst bucket: 0
2021/12/03 13:42:37 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMultipleHandlers_bucket_op_on_delete' in metakv",
  "warnings": null
 }
}
2021/12/03 13:42:42 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get deployed
2021/12/03 13:42:47 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get deployed
2021/12/03 13:42:52 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get deployed
2021/12/03 13:42:57 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get deployed
2021/12/03 13:42:57 App: TestMultipleHandlers_bucket_op_on_update got deployed
2021/12/03 13:43:02 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get deployed
2021/12/03 13:43:07 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get deployed
2021/12/03 13:43:12 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get deployed
2021/12/03 13:43:12 App: TestMultipleHandlers_bucket_op_on_delete got deployed
2021/12/03 13:43:12 Starting bucket ops to source bucket
2021/12/03 13:43:22 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 13:43:27 Waiting for dst bucket item count to get to: 5000 curr count: 4887
2021/12/03 13:43:32 Waiting for dst bucket item count to get to: 5000 curr count: 4887
2021/12/03 13:43:32 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/03 13:43:32 Starting bucket ops to source bucket
2021/12/03 13:43:48 Waiting for dst bucket item count to get to: 10000 curr count: 5925
2021/12/03 13:43:53 Waiting for dst bucket item count to get to: 10000 curr count: 5925
2021/12/03 13:43:53 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/03 13:43:53 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 13:43:53 Function: TestMultipleHandlers_bucket_op_on_update update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/03 13:43:53 Function: TestMultipleHandlers_bucket_op_on_delete update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/03 13:43:53 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 13:43:58 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get un-deployed
2021/12/03 13:44:03 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get un-deployed
2021/12/03 13:44:08 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get un-deployed
2021/12/03 13:44:13 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get un-deployed
2021/12/03 13:44:13 App: TestMultipleHandlers_bucket_op_on_update got un-deployed
2021/12/03 13:44:13 No eventing-con process running
2021/12/03 13:44:13 Function: TestMultipleHandlers_bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMultipleHandlers_bucket_op_on_update deleting in the background"
}
2021/12/03 13:44:15 Context: Bucket flush, response code: 200 dump: 
2021/12/03 13:44:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 13:44:22 Context: Bucket flush, response code: 200 dump: 
2021/12/03 13:44:32 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/03 13:44:37 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/03 13:44:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 13:44:37 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 13:44:42 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get un-deployed
2021/12/03 13:44:47 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get un-deployed
2021/12/03 13:44:52 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get un-deployed
2021/12/03 13:44:57 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get un-deployed
2021/12/03 13:44:57 App: TestMultipleHandlers_bucket_op_on_delete got un-deployed
2021/12/03 13:44:57 No eventing-con process running
2021/12/03 13:44:57 Function: TestMultipleHandlers_bucket_op_on_delete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMultipleHandlers_bucket_op_on_delete deleting in the background"
}
2021/12/03 13:44:59 Context: Bucket flush, response code: 200 dump: 
2021/12/03 13:45:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 13:45:06 Context: Bucket flush, response code: 200 dump: 
2021/12/03 13:45:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestMultipleHandlers (198.41s)
=== RUN   TestPauseResumeLoopDefaultSettings
2021/12/03 13:45:21 Waiting for app: TestPauseResumeLoopDefaultSettings to get un-deployed
2021/12/03 13:45:21 App: TestPauseResumeLoopDefaultSettings got un-deployed
2021/12/03 13:45:21 No eventing-con process running
2021/12/03 13:45:21 Function: TestPauseResumeLoopDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestPauseResumeLoopDefaultSettings not found"
 }
}
2021/12/03 13:45:23 Context: Bucket flush, response code: 200 dump: 
2021/12/03 13:45:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 13:45:30 Context: Bucket flush, response code: 200 dump: 
2021/12/03 13:45:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 13:45:35 Deploying app: TestPauseResumeLoopDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/03 13:45:36 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestPauseResumeLoopDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/03 13:45:36 Starting bucket ops to source bucket
2021/12/03 13:45:44 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 13:45:49 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 13:45:54 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 13:45:59 Waiting for dst bucket item count to get to: 5000 curr count: 2314
2021/12/03 13:46:04 Waiting for dst bucket item count to get to: 5000 curr count: 2314
2021/12/03 13:46:04 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/03 13:46:04 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 13:46:09 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to deployed
2021/12/03 13:46:09 Function: TestPauseResumeLoopDefaultSettings status changed to deployed
2021/12/03 13:46:09 Pausing app: bucket_op_on_update
2021/12/03 13:46:09 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/03 13:46:14 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to paused
2021/12/03 13:46:14 Function: TestPauseResumeLoopDefaultSettings status changed to paused
2021/12/03 13:46:14 Resuming app: bucket_op_on_update
2021/12/03 13:46:15 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/03 13:46:15 Starting bucket ops to source bucket
2021/12/03 13:46:23 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/03 13:46:28 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/03 13:46:33 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/03 13:46:33 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/03 13:46:33 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 13:46:38 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to deployed
2021/12/03 13:46:38 Function: TestPauseResumeLoopDefaultSettings status changed to deployed
2021/12/03 13:46:38 Pausing app: bucket_op_on_update
2021/12/03 13:46:38 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/03 13:46:43 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to paused
2021/12/03 13:46:43 Function: TestPauseResumeLoopDefaultSettings status changed to paused
2021/12/03 13:46:43 Resuming app: bucket_op_on_update
2021/12/03 13:46:44 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/03 13:46:44 Starting bucket ops to source bucket
2021/12/03 13:46:53 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/03 13:46:58 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/03 13:47:03 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/03 13:47:03 src & dst bucket item count matched up. src bucket count: 15000 dst bucket count: 15000
2021/12/03 13:47:03 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 13:47:08 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to deployed
2021/12/03 13:47:08 Function: TestPauseResumeLoopDefaultSettings status changed to deployed
2021/12/03 13:47:08 Pausing app: bucket_op_on_update
2021/12/03 13:47:08 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/03 13:47:13 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to paused
2021/12/03 13:47:13 Function: TestPauseResumeLoopDefaultSettings status changed to paused
2021/12/03 13:47:13 Resuming app: bucket_op_on_update
2021/12/03 13:47:14 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/03 13:47:14 Starting bucket ops to source bucket
2021/12/03 13:47:22 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/03 13:47:27 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/03 13:47:32 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/03 13:47:32 src & dst bucket item count matched up. src bucket count: 20000 dst bucket count: 20000
2021/12/03 13:47:32 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 13:47:37 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to deployed
2021/12/03 13:47:37 Function: TestPauseResumeLoopDefaultSettings status changed to deployed
2021/12/03 13:47:37 Pausing app: bucket_op_on_update
2021/12/03 13:47:37 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/03 13:47:42 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to paused
2021/12/03 13:47:42 Function: TestPauseResumeLoopDefaultSettings status changed to paused
2021/12/03 13:47:42 Resuming app: bucket_op_on_update
2021/12/03 13:47:43 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/03 13:47:43 Starting bucket ops to source bucket
2021/12/03 13:47:51 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/03 13:47:56 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/03 13:48:01 Waiting for dst bucket item count to get to: 25000 curr count: 20003
2021/12/03 13:48:06 Waiting for dst bucket item count to get to: 25000 curr count: 20003
2021/12/03 13:48:06 src & dst bucket item count matched up. src bucket count: 25000 dst bucket count: 25000
2021/12/03 13:48:06 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 13:48:11 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to deployed
2021/12/03 13:48:11 Function: TestPauseResumeLoopDefaultSettings status changed to deployed
2021/12/03 13:48:11 Pausing app: bucket_op_on_update
2021/12/03 13:48:11 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/03 13:48:16 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to paused
2021/12/03 13:48:16 Function: TestPauseResumeLoopDefaultSettings status changed to paused
2021/12/03 13:48:16 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 13:48:21 Waiting for app: TestPauseResumeLoopDefaultSettings to get un-deployed
2021/12/03 13:48:26 Waiting for app: TestPauseResumeLoopDefaultSettings to get un-deployed
2021/12/03 13:48:31 Waiting for app: TestPauseResumeLoopDefaultSettings to get un-deployed
2021/12/03 13:48:36 Waiting for app: TestPauseResumeLoopDefaultSettings to get un-deployed
2021/12/03 13:48:36 App: TestPauseResumeLoopDefaultSettings got un-deployed
2021/12/03 13:48:36 No eventing-con process running
2021/12/03 13:48:36 Function: TestPauseResumeLoopDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestPauseResumeLoopDefaultSettings deleting in the background"
}
2021/12/03 13:48:38 Context: Bucket flush, response code: 200 dump: 
2021/12/03 13:48:48 Waiting for dst bucket item count to get to: 0 curr count: 18201
2021/12/03 13:48:53 Waiting for dst bucket item count to get to: 0 curr count: 5701
2021/12/03 13:48:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 13:48:55 Context: Bucket flush, response code: 200 dump: 
2021/12/03 13:49:05 Waiting for dst bucket item count to get to: 0 curr count: 25000
2021/12/03 13:49:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestPauseResumeLoopDefaultSettings (234.15s)
=== RUN   TestPauseResumeLoopNonDefaultSettings
2021/12/03 13:49:15 Waiting for app: TestPauseResumeLoopNonDefaultSettings to get un-deployed
2021/12/03 13:49:15 App: TestPauseResumeLoopNonDefaultSettings got un-deployed
2021/12/03 13:49:15 No eventing-con process running
2021/12/03 13:49:15 Function: TestPauseResumeLoopNonDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestPauseResumeLoopNonDefaultSettings not found"
 }
}
2021/12/03 13:49:17 Context: Bucket flush, response code: 200 dump: 
2021/12/03 13:49:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 13:49:24 Context: Bucket flush, response code: 200 dump: 
2021/12/03 13:49:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 13:49:29 Deploying app: TestPauseResumeLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/03 13:49:30 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestPauseResumeLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/03 13:49:30 Starting bucket ops to source bucket
2021/12/03 13:49:38 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 13:49:43 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 13:49:48 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 13:49:53 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 13:49:58 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 13:50:03 Waiting for dst bucket item count to get to: 5000 curr count: 122
2021/12/03 13:50:08 Waiting for dst bucket item count to get to: 5000 curr count: 2029
2021/12/03 13:50:13 Waiting for dst bucket item count to get to: 5000 curr count: 4407
2021/12/03 13:50:13 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/03 13:50:13 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 13:50:18 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to deployed
2021/12/03 13:50:18 Function: TestPauseResumeLoopNonDefaultSettings status changed to deployed
2021/12/03 13:50:18 Pausing app: bucket_op_with_timer
2021/12/03 13:50:18 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/03 13:50:23 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to paused
2021/12/03 13:50:23 Function: TestPauseResumeLoopNonDefaultSettings status changed to paused
2021/12/03 13:50:23 Resuming app: bucket_op_with_timer
2021/12/03 13:50:24 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/03 13:50:24 Starting bucket ops to source bucket
2021/12/03 13:50:32 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/03 13:50:37 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/03 13:50:42 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/03 13:50:47 Waiting for dst bucket item count to get to: 10000 curr count: 5376
2021/12/03 13:50:52 Waiting for dst bucket item count to get to: 10000 curr count: 5376
2021/12/03 13:50:52 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/03 13:50:52 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 13:50:57 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to deployed
2021/12/03 13:50:57 Function: TestPauseResumeLoopNonDefaultSettings status changed to deployed
2021/12/03 13:50:57 Pausing app: bucket_op_with_timer
2021/12/03 13:50:58 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/03 13:51:03 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to paused
2021/12/03 13:51:03 Function: TestPauseResumeLoopNonDefaultSettings status changed to paused
2021/12/03 13:51:03 Resuming app: bucket_op_with_timer
2021/12/03 13:51:04 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/03 13:51:04 Starting bucket ops to source bucket
2021/12/03 13:51:12 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/03 13:51:17 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/03 13:51:22 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/03 13:51:27 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/03 13:51:32 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/03 13:51:32 src & dst bucket item count matched up. src bucket count: 15000 dst bucket count: 15000
2021/12/03 13:51:32 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 13:51:37 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to deployed
2021/12/03 13:51:37 Function: TestPauseResumeLoopNonDefaultSettings status changed to deployed
2021/12/03 13:51:37 Pausing app: bucket_op_with_timer
2021/12/03 13:51:37 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/03 13:51:42 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to paused
2021/12/03 13:51:42 Function: TestPauseResumeLoopNonDefaultSettings status changed to paused
2021/12/03 13:51:42 Resuming app: bucket_op_with_timer
2021/12/03 13:51:43 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/03 13:51:43 Starting bucket ops to source bucket
2021/12/03 13:51:51 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/03 13:51:56 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/03 13:52:01 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/03 13:52:06 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/03 13:52:11 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/03 13:52:16 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/03 13:52:16 src & dst bucket item count matched up. src bucket count: 20000 dst bucket count: 20000
2021/12/03 13:52:16 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 13:52:21 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to deployed
2021/12/03 13:52:21 Function: TestPauseResumeLoopNonDefaultSettings status changed to deployed
2021/12/03 13:52:21 Pausing app: bucket_op_with_timer
2021/12/03 13:52:21 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/03 13:52:26 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to paused
2021/12/03 13:52:26 Function: TestPauseResumeLoopNonDefaultSettings status changed to paused
2021/12/03 13:52:26 Resuming app: bucket_op_with_timer
2021/12/03 13:52:27 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/03 13:52:27 Starting bucket ops to source bucket
2021/12/03 13:52:35 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/03 13:52:40 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/03 13:52:45 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/03 13:52:50 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/03 13:52:55 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/03 13:53:00 Waiting for dst bucket item count to get to: 25000 curr count: 20956
2021/12/03 13:53:05 Waiting for dst bucket item count to get to: 25000 curr count: 20956
2021/12/03 13:53:05 src & dst bucket item count matched up. src bucket count: 25000 dst bucket count: 25000
2021/12/03 13:53:05 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 13:53:10 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to deployed
2021/12/03 13:53:10 Function: TestPauseResumeLoopNonDefaultSettings status changed to deployed
2021/12/03 13:53:10 Pausing app: bucket_op_with_timer
2021/12/03 13:53:10 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/03 13:53:15 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to paused
2021/12/03 13:53:15 Function: TestPauseResumeLoopNonDefaultSettings status changed to paused
2021/12/03 13:53:16 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 13:53:21 Waiting for app: TestPauseResumeLoopNonDefaultSettings to get un-deployed
2021/12/03 13:53:26 Waiting for app: TestPauseResumeLoopNonDefaultSettings to get un-deployed
2021/12/03 13:53:31 Waiting for app: TestPauseResumeLoopNonDefaultSettings to get un-deployed
2021/12/03 13:53:36 Waiting for app: TestPauseResumeLoopNonDefaultSettings to get un-deployed
2021/12/03 13:53:36 App: TestPauseResumeLoopNonDefaultSettings got un-deployed
2021/12/03 13:53:36 No eventing-con process running
2021/12/03 13:53:36 Function: TestPauseResumeLoopNonDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestPauseResumeLoopNonDefaultSettings deleting in the background"
}
2021/12/03 13:53:37 Context: Bucket flush, response code: 200 dump: 
2021/12/03 13:53:47 Waiting for dst bucket item count to get to: 0 curr count: 12500
2021/12/03 13:53:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 13:53:49 Context: Bucket flush, response code: 200 dump: 
2021/12/03 13:53:59 Waiting for dst bucket item count to get to: 0 curr count: 25000
2021/12/03 13:54:04 Waiting for dst bucket item count to get to: 0 curr count: 12500
2021/12/03 13:54:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestPauseResumeLoopNonDefaultSettings (299.07s)
=== RUN   TestPauseAndResumeWithWorkerCountChange
2021/12/03 13:54:14 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get un-deployed
2021/12/03 13:54:14 App: TestPauseAndResumeWithWorkerCountChange got un-deployed
2021/12/03 13:54:14 No eventing-con process running
2021/12/03 13:54:14 Function: TestPauseAndResumeWithWorkerCountChange delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestPauseAndResumeWithWorkerCountChange not found"
 }
}
2021/12/03 13:54:16 Context: Bucket flush, response code: 200 dump: 
2021/12/03 13:54:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 13:54:23 Context: Bucket flush, response code: 200 dump: 
2021/12/03 13:54:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 13:54:28 Deploying app: TestPauseAndResumeWithWorkerCountChange. Item count src bucket: 0 dst bucket: 0
2021/12/03 13:54:29 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestPauseAndResumeWithWorkerCountChange' in metakv",
  "warnings": null
 }
}
2021/12/03 13:54:34 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get deployed
2021/12/03 13:54:39 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get deployed
2021/12/03 13:54:44 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get deployed
2021/12/03 13:54:49 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get deployed
2021/12/03 13:54:49 App: TestPauseAndResumeWithWorkerCountChange got deployed
2021/12/03 13:54:49 Starting bucket ops to source bucket
2021/12/03 13:54:58 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 13:55:03 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 13:55:08 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 13:55:13 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 13:55:18 Waiting for dst bucket item count to get to: 5000 curr count: 793
2021/12/03 13:55:23 Waiting for dst bucket item count to get to: 5000 curr count: 1590
2021/12/03 13:55:28 Waiting for dst bucket item count to get to: 5000 curr count: 3297
2021/12/03 13:55:28 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/03 13:55:28 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 13:55:28 Pausing app: bucket_op_with_timer
2021/12/03 13:55:29 Function: TestPauseAndResumeWithWorkerCountChange update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/03 13:55:34 Waiting for function: TestPauseAndResumeWithWorkerCountChange status to change to paused
2021/12/03 13:55:34 Function: TestPauseAndResumeWithWorkerCountChange status changed to paused
2021/12/03 13:55:34 Resuming app: bucket_op_with_timer
2021/12/03 13:55:35 Function: TestPauseAndResumeWithWorkerCountChange update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:6] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/03 13:55:35 Starting bucket ops to source bucket
2021/12/03 13:55:46 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/03 13:55:51 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/03 13:55:56 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/03 13:56:01 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/03 13:56:06 Waiting for dst bucket item count to get to: 10000 curr count: 7717
2021/12/03 13:56:11 Waiting for dst bucket item count to get to: 10000 curr count: 7717
2021/12/03 13:56:11 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/03 13:56:12 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 13:56:17 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get un-deployed
2021/12/03 13:56:22 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get un-deployed
2021/12/03 13:56:27 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get un-deployed
2021/12/03 13:56:32 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get un-deployed
2021/12/03 13:56:32 App: TestPauseAndResumeWithWorkerCountChange got un-deployed
2021/12/03 13:56:32 No eventing-con process running
2021/12/03 13:56:32 Function: TestPauseAndResumeWithWorkerCountChange delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestPauseAndResumeWithWorkerCountChange deleting in the background"
}
2021/12/03 13:56:33 Context: Bucket flush, response code: 200 dump: 
2021/12/03 13:56:43 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/03 13:56:48 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/03 13:56:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 13:56:50 Context: Bucket flush, response code: 200 dump: 
2021/12/03 13:57:00 Waiting for dst bucket item count to get to: 0 curr count: 7148
2021/12/03 13:57:05 Waiting for dst bucket item count to get to: 0 curr count: 7148
2021/12/03 13:57:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestPauseAndResumeWithWorkerCountChange (181.00s)
=== RUN   TestPauseResumeWithEventingReb
2021/12/03 13:57:15 Waiting for app: TestPauseResumeWithEventingReb to get un-deployed
2021/12/03 13:57:15 App: TestPauseResumeWithEventingReb got un-deployed
2021/12/03 13:57:15 No eventing-con process running
2021/12/03 13:57:15 Function: TestPauseResumeWithEventingReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestPauseResumeWithEventingReb not found"
 }
}
2021/12/03 13:57:17 Context: Bucket flush, response code: 200 dump: 
2021/12/03 13:57:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 13:57:24 Context: Bucket flush, response code: 200 dump: 
2021/12/03 13:57:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 13:57:29 Deploying app: TestPauseResumeWithEventingReb. Item count src bucket: 0 dst bucket: 0
2021/12/03 13:57:30 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestPauseResumeWithEventingReb' in metakv",
  "warnings": null
 }
}
2021/12/03 13:57:35 Waiting for app: TestPauseResumeWithEventingReb to get deployed
2021/12/03 13:57:40 Waiting for app: TestPauseResumeWithEventingReb to get deployed
2021/12/03 13:57:45 Waiting for app: TestPauseResumeWithEventingReb to get deployed
2021/12/03 13:57:50 Waiting for app: TestPauseResumeWithEventingReb to get deployed
2021/12/03 13:57:50 App: TestPauseResumeWithEventingReb got deployed
2021/12/03 13:57:50 Starting bucket ops to source bucket
2021/12/03 13:57:59 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 13:58:04 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 13:58:09 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 13:58:14 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 13:58:19 Waiting for dst bucket item count to get to: 5000 curr count: 2695
2021/12/03 13:58:24 Waiting for dst bucket item count to get to: 5000 curr count: 2695
2021/12/03 13:58:24 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/03 13:58:24 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 13:58:24 Pausing app: bucket_op_with_timer
2021/12/03 13:58:25 Function: TestPauseResumeWithEventingReb update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/03 13:58:30 Waiting for function: TestPauseResumeWithEventingReb status to change to paused
2021/12/03 13:58:30 Function: TestPauseResumeWithEventingReb status changed to paused
2021/12/03 13:58:30 Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/03 13:58:30 Removing node(s): [] from the cluster
2021/12/03 13:58:35 Rebalance progress: 29.16666666666666
2021/12/03 13:58:40 Rebalance progress: 29.16666666666666
2021/12/03 13:58:45 Rebalance progress: 100
2021/12/03 13:58:45 Resuming app: bucket_op_with_timer
2021/12/03 13:58:47 Function: TestPauseResumeWithEventingReb update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:6] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/03 13:58:47 Starting bucket ops to source bucket
2021/12/03 13:58:59 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/03 13:59:04 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/03 13:59:09 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/03 13:59:14 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/03 13:59:19 Waiting for dst bucket item count to get to: 10000 curr count: 5984
2021/12/03 13:59:24 Waiting for dst bucket item count to get to: 10000 curr count: 5984
2021/12/03 13:59:29 Waiting for dst bucket item count to get to: 10000 curr count: 9688
2021/12/03 13:59:34 Waiting for dst bucket item count to get to: 10000 curr count: 9688
2021/12/03 13:59:34 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/03 13:59:34 Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/03 13:59:39 Rebalance progress: 29.16666666666666
2021/12/03 13:59:44 Rebalance progress: 29.16666666666666
2021/12/03 13:59:49 Rebalance progress: 100

VB distribution across eventing nodes and their workers::

2021/12/03 13:59:55 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:70} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/03 14:00:10 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:70} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/03 14:00:25 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:70} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/03 14:00:40 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:70} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/03 14:00:55 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:70} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/03 14:01:10 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:70} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/03 14:01:25 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:70} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/03 14:01:40 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:70} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/03 14:01:55 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:70} {Name:#primary Bucket:hello-world Completion:100}]

DCP Stream statuses:
	stream status: running
	len: 1024
	vb list dump: [0-1023]

vbucket curr owner:
Producer node: 127.0.0.1:9302	Node UUID: 64f26d3a3eef20ed66f1588e56d18517
	workerID: worker_TestPauseResumeWithEventingReb_4
	len: 57
	v: [228-284]
	workerID: worker_TestPauseResumeWithEventingReb_5
	len: 57
	v: [285-341]
	workerID: worker_TestPauseResumeWithEventingReb_0
	len: 57
	v: [0-56]
	workerID: worker_TestPauseResumeWithEventingReb_1
	len: 57
	v: [57-113]
	workerID: worker_TestPauseResumeWithEventingReb_2
	len: 57
	v: [114-170]
	workerID: worker_TestPauseResumeWithEventingReb_3
	len: 57
	v: [171-227]
Producer node: 172.31.2.134:9300	Node UUID: 64cb1bf93ae4eb8a6053cb5564021e18
	workerID: worker_TestPauseResumeWithEventingReb_5
	len: 56
	v: [968-1023]
	workerID: worker_TestPauseResumeWithEventingReb_0
	len: 57
	v: [683-739]
	workerID: worker_TestPauseResumeWithEventingReb_1
	len: 57
	v: [740-796]
	workerID: worker_TestPauseResumeWithEventingReb_2
	len: 57
	v: [797-853]
	workerID: worker_TestPauseResumeWithEventingReb_3
	len: 57
	v: [854-910]
	workerID: worker_TestPauseResumeWithEventingReb_4
	len: 57
	v: [911-967]
Producer node: 127.0.0.1:9303	Node UUID: c4be832488fb52bd9b1d204987f6b740
	workerID: worker_TestPauseResumeWithEventingReb_0
	len: 57
	v: [342-398]
	workerID: worker_TestPauseResumeWithEventingReb_1
	len: 57
	v: [399-455]
	workerID: worker_TestPauseResumeWithEventingReb_2
	len: 57
	v: [456-512]
	workerID: worker_TestPauseResumeWithEventingReb_3
	len: 57
	v: [513-569]
	workerID: worker_TestPauseResumeWithEventingReb_4
	len: 57
	v: [570-626]
	workerID: worker_TestPauseResumeWithEventingReb_5
	len: 56
	v: [627-682]


2021/12/03 14:02:10 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 14:02:10 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 14:02:15 clusterSetup.go:397: Waiting for app: TestPauseResumeWithEventingReb to get un-deployed
2021/12/03 14:02:20 clusterSetup.go:397: Waiting for app: TestPauseResumeWithEventingReb to get un-deployed
2021/12/03 14:02:25 clusterSetup.go:397: Waiting for app: TestPauseResumeWithEventingReb to get un-deployed
2021/12/03 14:02:30 clusterSetup.go:397: Waiting for app: TestPauseResumeWithEventingReb to get un-deployed
2021/12/03 14:02:30 clusterSetup.go:405: App: TestPauseResumeWithEventingReb got un-deployed
2021/12/03 14:02:30 eventingSetup.go:46: No eventing-con process running
2021/12/03 14:02:30 eventingSetup.go:418: Function: TestPauseResumeWithEventingReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestPauseResumeWithEventingReb deleting in the background"
}
2021/12/03 14:02:32 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/03 14:02:42 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/03 14:02:47 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/03 14:02:47 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 14:02:48 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/03 14:02:59 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 8593
2021/12/03 14:03:04 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3593
2021/12/03 14:03:04 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestPauseResumeWithEventingReb (358.31s)
=== RUN   TestBucketDeleteAfterPause
2021/12/03 14:03:14 clusterSetup.go:397: Waiting for app: TestBucketDeleteAfterPause to get un-deployed
2021/12/03 14:03:14 clusterSetup.go:405: App: TestBucketDeleteAfterPause got un-deployed
2021/12/03 14:03:14 eventingSetup.go:46: No eventing-con process running
2021/12/03 14:03:14 eventingSetup.go:418: Function: TestBucketDeleteAfterPause delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestBucketDeleteAfterPause not found"
 }
}
2021/12/03 14:03:15 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/03 14:03:20 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 14:03:22 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/03 14:03:27 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 14:03:27 eventingSetup.go:125: Deploying app: TestBucketDeleteAfterPause. Item count src bucket: 0 dst bucket: 0
2021/12/03 14:03:28 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestBucketDeleteAfterPause' in metakv",
  "warnings": null
 }
}
2021/12/03 14:03:33 clusterSetup.go:346: Waiting for app: TestBucketDeleteAfterPause to get deployed
2021/12/03 14:03:38 clusterSetup.go:346: Waiting for app: TestBucketDeleteAfterPause to get deployed
2021/12/03 14:03:43 clusterSetup.go:346: Waiting for app: TestBucketDeleteAfterPause to get deployed
2021/12/03 14:03:48 clusterSetup.go:346: Waiting for app: TestBucketDeleteAfterPause to get deployed
2021/12/03 14:03:48 clusterSetup.go:354: App: TestBucketDeleteAfterPause got deployed
2021/12/03 14:03:48 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/03 14:03:58 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 441
2021/12/03 14:04:03 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 2941
2021/12/03 14:04:08 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 2941
2021/12/03 14:04:08 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/03 14:04:08 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 14:04:08 handler_test.go:712: Pausing app: bucket_op_on_update
2021/12/03 14:04:08 eventingSetup.go:270: Function: TestBucketDeleteAfterPause update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/03 14:04:13 eventingSetup.go:719: Waiting for function: TestBucketDeleteAfterPause status to change to paused
2021/12/03 14:04:13 eventingSetup.go:727: Function: TestBucketDeleteAfterPause status changed to paused
2021/12/03 14:04:13 handler_test.go:716: Deleting source bucket: default
2021/12/03 14:04:20 eventingSetup.go:719: Waiting for function: TestBucketDeleteAfterPause status to change to undeployed
2021/12/03 14:04:20 eventingSetup.go:727: Function: TestBucketDeleteAfterPause status changed to undeployed
2021/12/03 14:04:25 clusterSetup.go:397: Waiting for app: TestBucketDeleteAfterPause to get un-deployed
2021/12/03 14:04:25 clusterSetup.go:405: App: TestBucketDeleteAfterPause got un-deployed
2021/12/03 14:04:25 eventingSetup.go:46: No eventing-con process running
2021/12/03 14:04:25 eventingSetup.go:418: Function: TestBucketDeleteAfterPause delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestBucketDeleteAfterPause deleting in the background"
}
2021/12/03 14:04:26 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/03 14:04:36 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/03 14:04:36 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 14:04:38 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/03 14:04:48 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3756
2021/12/03 14:04:53 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1256
2021/12/03 14:04:53 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestBucketDeleteAfterPause (109.57s)
=== RUN   TestChangeFnCodeBetweenPauseResume
2021/12/03 14:05:03 clusterSetup.go:397: Waiting for app: TestChangeFnCodeBetweenPauseResume to get un-deployed
2021/12/03 14:05:03 clusterSetup.go:405: App: TestChangeFnCodeBetweenPauseResume got un-deployed
2021/12/03 14:05:03 eventingSetup.go:46: No eventing-con process running
2021/12/03 14:05:03 eventingSetup.go:418: Function: TestChangeFnCodeBetweenPauseResume delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestChangeFnCodeBetweenPauseResume not found"
 }
}
2021/12/03 14:05:04 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/03 14:05:09 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 14:05:11 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/03 14:05:16 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 14:05:16 eventingSetup.go:125: Deploying app: TestChangeFnCodeBetweenPauseResume. Item count src bucket: 0 dst bucket: 0
2021/12/03 14:05:17 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestChangeFnCodeBetweenPauseResume' in metakv",
  "warnings": null
 }
}
2021/12/03 14:05:22 clusterSetup.go:346: Waiting for app: TestChangeFnCodeBetweenPauseResume to get deployed
2021/12/03 14:05:27 clusterSetup.go:346: Waiting for app: TestChangeFnCodeBetweenPauseResume to get deployed
2021/12/03 14:05:33 clusterSetup.go:346: Waiting for app: TestChangeFnCodeBetweenPauseResume to get deployed
2021/12/03 14:05:38 clusterSetup.go:346: Waiting for app: TestChangeFnCodeBetweenPauseResume to get deployed
2021/12/03 14:05:38 clusterSetup.go:354: App: TestChangeFnCodeBetweenPauseResume got deployed
2021/12/03 14:05:38 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/03 14:06:13 handler_test.go:739: Pausing function: TestChangeFnCodeBetweenPauseResume
2021/12/03 14:06:13 eventingSetup.go:270: Function: TestChangeFnCodeBetweenPauseResume update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/03 14:06:18 eventingSetup.go:719: Waiting for function: TestChangeFnCodeBetweenPauseResume status to change to paused
2021/12/03 14:06:18 eventingSetup.go:727: Function: TestChangeFnCodeBetweenPauseResume status changed to paused
2021/12/03 14:09:18 handler_test.go:748: Resuming function: TestChangeFnCodeBetweenPauseResume
2021/12/03 14:09:18 eventingSetup.go:125: Deploying app: TestChangeFnCodeBetweenPauseResume. Item count src bucket: 5000 dst bucket: 0
2021/12/03 14:09:19 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestChangeFnCodeBetweenPauseResume' in metakv",
  "warnings": null
 }
}
2021/12/03 14:09:24 eventingSetup.go:795: Waiting for function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter to get to 5000
2021/12/03 14:09:24 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 14:09:24 eventingSetup.go:799: Function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter got to 0 expected 5000
2021/12/03 14:09:29 eventingSetup.go:795: Waiting for function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter to get to 5000
2021/12/03 14:09:29 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 14:09:29 eventingSetup.go:799: Function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter got to 2951 expected 5000
2021/12/03 14:09:34 eventingSetup.go:795: Waiting for function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter to get to 5000
2021/12/03 14:09:34 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 14:09:34 eventingSetup.go:799: Function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter got to 3595 expected 5000
2021/12/03 14:09:39 eventingSetup.go:795: Waiting for function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter to get to 5000
2021/12/03 14:09:39 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 14:09:39 eventingSetup.go:804: Function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter got to 5000
2021/12/03 14:09:39 handler_test.go:753: Undeploying function: TestChangeFnCodeBetweenPauseResume
2021/12/03 14:09:39 eventingSetup.go:270: Function: TestChangeFnCodeBetweenPauseResume update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/03 14:09:49 clusterSetup.go:397: Waiting for app: TestChangeFnCodeBetweenPauseResume to get un-deployed
2021/12/03 14:09:54 clusterSetup.go:397: Waiting for app: TestChangeFnCodeBetweenPauseResume to get un-deployed
2021/12/03 14:09:59 clusterSetup.go:397: Waiting for app: TestChangeFnCodeBetweenPauseResume to get un-deployed
2021/12/03 14:09:59 clusterSetup.go:405: App: TestChangeFnCodeBetweenPauseResume got un-deployed
2021/12/03 14:09:59 eventingSetup.go:46: No eventing-con process running
2021/12/03 14:09:59 eventingSetup.go:418: Function: TestChangeFnCodeBetweenPauseResume delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestChangeFnCodeBetweenPauseResume deleting in the background"
}
2021/12/03 14:10:00 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/03 14:10:10 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4943
2021/12/03 14:10:15 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4943
2021/12/03 14:10:15 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 14:10:17 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/03 14:10:22 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestChangeFnCodeBetweenPauseResume (329.24s)
=== RUN   TestCommentUnCommentOnDelete
2021/12/03 14:10:32 clusterSetup.go:397: Waiting for app: TestCommentUnCommentOnDelete to get un-deployed
2021/12/03 14:10:32 clusterSetup.go:405: App: TestCommentUnCommentOnDelete got un-deployed
2021/12/03 14:10:32 eventingSetup.go:46: No eventing-con process running
2021/12/03 14:10:32 eventingSetup.go:418: Function: TestCommentUnCommentOnDelete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestCommentUnCommentOnDelete not found"
 }
}
2021/12/03 14:10:34 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/03 14:10:39 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 14:10:41 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/03 14:10:46 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 14:10:46 eventingSetup.go:125: Deploying app: TestCommentUnCommentOnDelete. Item count src bucket: 0 dst bucket: 0
2021/12/03 14:10:47 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCommentUnCommentOnDelete' in metakv",
  "warnings": null
 }
}
2021/12/03 14:10:52 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/03 14:10:57 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/03 14:11:02 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/03 14:11:07 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/03 14:11:07 clusterSetup.go:354: App: TestCommentUnCommentOnDelete got deployed
2021/12/03 14:11:07 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/03 14:11:16 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 14:11:21 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 1676
2021/12/03 14:11:26 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 1676
2021/12/03 14:11:26 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/03 14:11:26 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 14:11:26 handler_test.go:780: Undeploying app: TestCommentUnCommentOnDelete
2021/12/03 14:11:26 eventingSetup.go:270: Function: TestCommentUnCommentOnDelete update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/03 14:11:56 eventingSetup.go:125: Deploying app: TestCommentUnCommentOnDelete. Item count src bucket: 5000 dst bucket: 5000
2021/12/03 14:11:57 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCommentUnCommentOnDelete' in metakv",
  "warnings": null
 }
}
2021/12/03 14:12:02 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/03 14:12:07 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/03 14:12:12 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/03 14:12:17 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/03 14:12:17 clusterSetup.go:354: App: TestCommentUnCommentOnDelete got deployed
2021/12/03 14:12:17 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/03 14:12:30 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/03 14:12:35 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/03 14:12:35 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 14:12:35 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 14:12:35 handler_test.go:799: Undeploying app: TestCommentUnCommentOnDelete
2021/12/03 14:12:35 eventingSetup.go:270: Function: TestCommentUnCommentOnDelete update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/03 14:12:45 clusterSetup.go:397: Waiting for app: TestCommentUnCommentOnDelete to get un-deployed
2021/12/03 14:12:50 clusterSetup.go:397: Waiting for app: TestCommentUnCommentOnDelete to get un-deployed
2021/12/03 14:12:55 clusterSetup.go:397: Waiting for app: TestCommentUnCommentOnDelete to get un-deployed
2021/12/03 14:12:55 clusterSetup.go:405: App: TestCommentUnCommentOnDelete got un-deployed
2021/12/03 14:12:55 eventingSetup.go:46: No eventing-con process running
2021/12/03 14:12:55 eventingSetup.go:418: Function: TestCommentUnCommentOnDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCommentUnCommentOnDelete deleting in the background"
}
2021/12/03 14:12:56 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/03 14:13:01 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 14:13:03 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/03 14:13:08 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCommentUnCommentOnDelete (166.02s)
=== RUN   TestCPPWorkerCleanup
2021/12/03 14:13:18 clusterSetup.go:397: Waiting for app: TestCPPWorkerCleanup to get un-deployed
2021/12/03 14:13:18 clusterSetup.go:405: App: TestCPPWorkerCleanup got un-deployed
2021/12/03 14:13:18 eventingSetup.go:46: No eventing-con process running
2021/12/03 14:13:18 eventingSetup.go:418: Function: TestCPPWorkerCleanup delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestCPPWorkerCleanup not found"
 }
}
2021/12/03 14:13:20 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/03 14:13:25 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 14:13:27 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/03 14:13:32 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 14:13:32 eventingSetup.go:125: Deploying app: TestCPPWorkerCleanup. Item count src bucket: 0 dst bucket: 0
2021/12/03 14:13:33 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCPPWorkerCleanup' in metakv",
  "warnings": null
 }
}
2021/12/03 14:13:33 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/03 14:13:40 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 14:13:45 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 14:13:50 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 14:13:55 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 14:14:00 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 2680
2021/12/03 14:14:05 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 2680
2021/12/03 14:14:05 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/03 14:14:05 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 14:14:05 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 14:14:10 clusterSetup.go:397: Waiting for app: TestCPPWorkerCleanup to get un-deployed
2021/12/03 14:14:15 clusterSetup.go:397: Waiting for app: TestCPPWorkerCleanup to get un-deployed
2021/12/03 14:14:20 clusterSetup.go:397: Waiting for app: TestCPPWorkerCleanup to get un-deployed
2021/12/03 14:14:25 clusterSetup.go:397: Waiting for app: TestCPPWorkerCleanup to get un-deployed
2021/12/03 14:14:25 clusterSetup.go:405: App: TestCPPWorkerCleanup got un-deployed
2021/12/03 14:14:25 eventingSetup.go:46: No eventing-con process running
2021/12/03 14:14:25 eventingSetup.go:418: Function: TestCPPWorkerCleanup delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCPPWorkerCleanup deleting in the background"
}
2021/12/03 14:14:26 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/03 14:14:36 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/03 14:14:36 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 14:14:38 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/03 14:14:48 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3785
2021/12/03 14:14:53 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1285
2021/12/03 14:14:53 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCPPWorkerCleanup (134.82s)
=== RUN   TestWithUserXattrs
2021/12/03 14:15:33 clusterSetup.go:397: Waiting for app: TestWithUserXattrs to get un-deployed
2021/12/03 14:15:33 clusterSetup.go:405: App: TestWithUserXattrs got un-deployed
2021/12/03 14:15:33 eventingSetup.go:46: No eventing-con process running
2021/12/03 14:15:33 eventingSetup.go:418: Function: TestWithUserXattrs delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestWithUserXattrs not found"
 }
}
2021/12/03 14:15:34 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/03 14:15:39 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 14:15:41 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/03 14:15:46 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 14:15:46 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/03 14:15:48 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/03 14:15:50 eventingSetup.go:125: Deploying app: TestWithUserXattrs. Item count src bucket: 0 dst bucket: 0
2021/12/03 14:15:52 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestWithUserXattrs' in metakv",
  "warnings": null
 }
}
2021/12/03 14:15:57 clusterSetup.go:346: Waiting for app: TestWithUserXattrs to get deployed
2021/12/03 14:16:02 clusterSetup.go:346: Waiting for app: TestWithUserXattrs to get deployed
2021/12/03 14:16:07 clusterSetup.go:346: Waiting for app: TestWithUserXattrs to get deployed
2021/12/03 14:16:12 clusterSetup.go:346: Waiting for app: TestWithUserXattrs to get deployed
2021/12/03 14:16:12 clusterSetup.go:354: App: TestWithUserXattrs got deployed
2021/12/03 14:16:12 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/03 14:16:21 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 14:16:26 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 14:16:26 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/03 14:16:26 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 14:16:26 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 14:16:31 clusterSetup.go:397: Waiting for app: TestWithUserXattrs to get un-deployed
2021/12/03 14:16:36 clusterSetup.go:397: Waiting for app: TestWithUserXattrs to get un-deployed
2021/12/03 14:16:41 clusterSetup.go:397: Waiting for app: TestWithUserXattrs to get un-deployed
2021/12/03 14:16:46 clusterSetup.go:397: Waiting for app: TestWithUserXattrs to get un-deployed
2021/12/03 14:16:46 clusterSetup.go:405: App: TestWithUserXattrs got un-deployed
2021/12/03 14:16:46 eventingSetup.go:46: No eventing-con process running
2021/12/03 14:16:46 eventingSetup.go:418: Function: TestWithUserXattrs delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestWithUserXattrs deleting in the background"
}
2021/12/03 14:16:47 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/03 14:16:58 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/03 14:16:58 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 14:16:59 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/03 14:17:09 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/03 14:17:14 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/03 14:17:14 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestWithUserXattrs (111.20s)
=== RUN   TestEventProcessingPostBucketFlush
2021/12/03 14:17:24 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/03 14:17:24 clusterSetup.go:405: App: TestEventProcessingPostBucketFlush got un-deployed
2021/12/03 14:17:24 eventingSetup.go:46: No eventing-con process running
2021/12/03 14:17:24 eventingSetup.go:418: Function: TestEventProcessingPostBucketFlush delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventProcessingPostBucketFlush not found"
 }
}
2021/12/03 14:17:26 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/03 14:17:31 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 14:17:33 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/03 14:17:38 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 14:17:38 eventingSetup.go:125: Deploying app: TestEventProcessingPostBucketFlush. Item count src bucket: 0 dst bucket: 0
2021/12/03 14:17:39 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventProcessingPostBucketFlush' in metakv",
  "warnings": null
 }
}
2021/12/03 14:17:44 clusterSetup.go:346: Waiting for app: TestEventProcessingPostBucketFlush to get deployed
2021/12/03 14:17:49 clusterSetup.go:346: Waiting for app: TestEventProcessingPostBucketFlush to get deployed
2021/12/03 14:17:54 clusterSetup.go:346: Waiting for app: TestEventProcessingPostBucketFlush to get deployed
2021/12/03 14:17:59 clusterSetup.go:346: Waiting for app: TestEventProcessingPostBucketFlush to get deployed
2021/12/03 14:17:59 clusterSetup.go:354: App: TestEventProcessingPostBucketFlush got deployed
2021/12/03 14:17:59 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/03 14:18:08 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 14:18:13 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 324
2021/12/03 14:18:18 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 2824
2021/12/03 14:18:18 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/03 14:18:18 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 14:18:21 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/03 14:18:31 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3678
2021/12/03 14:18:36 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3678
2021/12/03 14:18:36 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 14:18:38 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/03 14:18:48 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3915
2021/12/03 14:18:53 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1415
2021/12/03 14:18:53 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 14:18:53 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/03 14:19:19 eventingSetup.go:441: Waiting for dst bucket item count to get to: 25000 curr count: 12109
2021/12/03 14:19:24 eventingSetup.go:441: Waiting for dst bucket item count to get to: 25000 curr count: 18365
2021/12/03 14:19:29 eventingSetup.go:441: Waiting for dst bucket item count to get to: 25000 curr count: 22361
2021/12/03 14:19:29 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 25000 dst bucket count: 25000
2021/12/03 14:19:29 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 14:19:29 eventingSetup.go:270: Function: TestEventProcessingPostBucketFlush update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/03 14:19:34 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/03 14:19:39 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/03 14:19:44 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/03 14:19:49 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/03 14:19:49 clusterSetup.go:405: App: TestEventProcessingPostBucketFlush got un-deployed
2021/12/03 14:19:49 eventingSetup.go:46: No eventing-con process running
2021/12/03 14:19:54 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/03 14:19:54 clusterSetup.go:405: App: TestEventProcessingPostBucketFlush got un-deployed
2021/12/03 14:19:54 eventingSetup.go:46: No eventing-con process running
2021/12/03 14:19:54 eventingSetup.go:418: Function: TestEventProcessingPostBucketFlush delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventProcessingPostBucketFlush deleting in the background"
}
2021/12/03 14:19:55 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/03 14:20:05 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 25000
2021/12/03 14:20:05 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 14:20:07 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/03 14:20:17 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 25000
2021/12/03 14:20:17 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventProcessingPostBucketFlush (182.55s)
=== RUN   TestMetaBucketDelete
2021/12/03 14:20:27 clusterSetup.go:397: Waiting for app: TestMetaBucketDelete to get un-deployed
2021/12/03 14:20:27 clusterSetup.go:405: App: TestMetaBucketDelete got un-deployed
2021/12/03 14:20:27 eventingSetup.go:46: No eventing-con process running
2021/12/03 14:20:27 eventingSetup.go:418: Function: TestMetaBucketDelete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMetaBucketDelete not found"
 }
}
2021/12/03 14:20:28 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/03 14:20:33 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 14:20:35 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/03 14:20:40 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 14:20:40 eventingSetup.go:125: Deploying app: TestMetaBucketDelete. Item count src bucket: 0 dst bucket: 0
2021/12/03 14:20:41 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMetaBucketDelete' in metakv",
  "warnings": null
 }
}
2021/12/03 14:20:46 clusterSetup.go:346: Waiting for app: TestMetaBucketDelete to get deployed
2021/12/03 14:20:51 clusterSetup.go:346: Waiting for app: TestMetaBucketDelete to get deployed
2021/12/03 14:20:56 clusterSetup.go:346: Waiting for app: TestMetaBucketDelete to get deployed
2021/12/03 14:21:01 clusterSetup.go:346: Waiting for app: TestMetaBucketDelete to get deployed
2021/12/03 14:21:01 clusterSetup.go:354: App: TestMetaBucketDelete got deployed
2021/12/03 14:21:01 handler_test.go:902: Deleting metadata bucket: eventing
2021/12/03 14:21:02 handler_test.go:904: Deleted metadata bucket: eventing
2021/12/03 14:21:07 clusterSetup.go:397: Waiting for app: TestMetaBucketDelete to get un-deployed
2021/12/03 14:21:07 clusterSetup.go:405: App: TestMetaBucketDelete got un-deployed
2021/12/03 14:21:23 clusterSetup.go:397: Waiting for app: TestMetaBucketDelete to get un-deployed
2021/12/03 14:21:23 clusterSetup.go:405: App: TestMetaBucketDelete got un-deployed
2021/12/03 14:21:23 eventingSetup.go:46: No eventing-con process running
2021/12/03 14:21:23 eventingSetup.go:418: Function: TestMetaBucketDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMetaBucketDelete deleting in the background"
}
2021/12/03 14:21:24 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/03 14:21:29 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 14:21:31 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/03 14:21:36 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestMetaBucketDelete (78.76s)
=== RUN   TestMetaBucketDeleteWithBootstrap
2021/12/03 14:21:46 clusterSetup.go:397: Waiting for app: TestMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/03 14:21:46 clusterSetup.go:405: App: TestMetaBucketDeleteWithBootstrap got un-deployed
2021/12/03 14:21:46 eventingSetup.go:46: No eventing-con process running
2021/12/03 14:21:46 eventingSetup.go:418: Function: TestMetaBucketDeleteWithBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMetaBucketDeleteWithBootstrap not found"
 }
}
2021/12/03 14:21:47 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/03 14:21:52 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 14:21:54 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/03 14:21:59 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 14:21:59 eventingSetup.go:125: Deploying app: TestMetaBucketDeleteWithBootstrap. Item count src bucket: 0 dst bucket: 0
2021/12/03 14:22:00 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMetaBucketDeleteWithBootstrap' in metakv",
  "warnings": null
 }
}
2021/12/03 14:22:19 handler_test.go:923: Deleting metadata bucket: eventing
2021/12/03 14:22:19 handler_test.go:925: Deleted metadata bucket: eventing
2021/12/03 14:22:34 clusterSetup.go:397: Waiting for app: TestMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/03 14:22:34 clusterSetup.go:405: App: TestMetaBucketDeleteWithBootstrap got un-deployed
2021/12/03 14:22:49 clusterSetup.go:397: Waiting for app: TestMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/03 14:22:49 clusterSetup.go:405: App: TestMetaBucketDeleteWithBootstrap got un-deployed
2021/12/03 14:22:49 eventingSetup.go:46: No eventing-con process running
2021/12/03 14:22:50 eventingSetup.go:418: Function: TestMetaBucketDeleteWithBootstrap delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMetaBucketDeleteWithBootstrap deleting in the background"
}
2021/12/03 14:22:51 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/03 14:22:56 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 14:22:58 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/03 14:23:03 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestMetaBucketDeleteWithBootstrap (86.90s)
=== RUN   TestSourceBucketDelete
2021/12/03 14:23:13 clusterSetup.go:397: Waiting for app: TestSourceBucketDelete to get un-deployed
2021/12/03 14:23:13 clusterSetup.go:405: App: TestSourceBucketDelete got un-deployed
2021/12/03 14:23:13 eventingSetup.go:46: No eventing-con process running
2021/12/03 14:23:13 eventingSetup.go:418: Function: TestSourceBucketDelete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestSourceBucketDelete not found"
 }
}
2021/12/03 14:23:14 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/03 14:23:19 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 14:23:21 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/03 14:23:26 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 14:23:26 eventingSetup.go:125: Deploying app: TestSourceBucketDelete. Item count src bucket: 0 dst bucket: 0
2021/12/03 14:23:27 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestSourceBucketDelete' in metakv",
  "warnings": null
 }
}
2021/12/03 14:23:32 clusterSetup.go:346: Waiting for app: TestSourceBucketDelete to get deployed
2021/12/03 14:23:37 clusterSetup.go:346: Waiting for app: TestSourceBucketDelete to get deployed
2021/12/03 14:23:42 clusterSetup.go:346: Waiting for app: TestSourceBucketDelete to get deployed
2021/12/03 14:23:47 clusterSetup.go:346: Waiting for app: TestSourceBucketDelete to get deployed
2021/12/03 14:23:47 clusterSetup.go:354: App: TestSourceBucketDelete got deployed
2021/12/03 14:23:47 handler_test.go:945: Deleting source bucket: default
2021/12/03 14:23:49 handler_test.go:947: Deleted source bucket: default
2021/12/03 14:23:54 clusterSetup.go:397: Waiting for app: TestSourceBucketDelete to get un-deployed
2021/12/03 14:23:54 clusterSetup.go:405: App: TestSourceBucketDelete got un-deployed
2021/12/03 14:24:09 clusterSetup.go:397: Waiting for app: TestSourceBucketDelete to get un-deployed
2021/12/03 14:24:09 clusterSetup.go:405: App: TestSourceBucketDelete got un-deployed
2021/12/03 14:24:09 eventingSetup.go:46: No eventing-con process running
2021/12/03 14:24:09 eventingSetup.go:418: Function: TestSourceBucketDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestSourceBucketDelete deleting in the background"
}
2021/12/03 14:24:10 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/03 14:24:15 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 14:24:17 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/03 14:24:22 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestSourceBucketDelete (79.12s)
=== RUN   TestSourceBucketDeleteWithBootstrap
2021/12/03 14:24:32 clusterSetup.go:397: Waiting for app: TestSourceBucketDeleteWithBootstrap to get un-deployed
2021/12/03 14:24:32 clusterSetup.go:405: App: TestSourceBucketDeleteWithBootstrap got un-deployed
2021/12/03 14:24:32 eventingSetup.go:46: No eventing-con process running
2021/12/03 14:24:32 eventingSetup.go:418: Function: TestSourceBucketDeleteWithBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestSourceBucketDeleteWithBootstrap not found"
 }
}
2021/12/03 14:24:33 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/03 14:24:38 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 14:24:40 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/03 14:24:45 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 14:24:45 eventingSetup.go:125: Deploying app: TestSourceBucketDeleteWithBootstrap. Item count src bucket: 0 dst bucket: 0
2021/12/03 14:24:46 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestSourceBucketDeleteWithBootstrap' in metakv",
  "warnings": null
 }
}
2021/12/03 14:25:05 handler_test.go:966: Deleting source bucket: default
2021/12/03 14:25:06 handler_test.go:968: Deleted source bucket: default
2021/12/03 14:25:21 clusterSetup.go:397: Waiting for app: TestSourceBucketDeleteWithBootstrap to get un-deployed
2021/12/03 14:25:21 clusterSetup.go:405: App: TestSourceBucketDeleteWithBootstrap got un-deployed
2021/12/03 14:25:36 clusterSetup.go:397: Waiting for app: TestSourceBucketDeleteWithBootstrap to get un-deployed
2021/12/03 14:25:36 clusterSetup.go:405: App: TestSourceBucketDeleteWithBootstrap got un-deployed
2021/12/03 14:25:36 eventingSetup.go:46: No eventing-con process running
2021/12/03 14:25:36 eventingSetup.go:418: Function: TestSourceBucketDeleteWithBootstrap delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestSourceBucketDeleteWithBootstrap deleting in the background"
}
2021/12/03 14:25:38 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/03 14:25:43 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 14:25:44 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/03 14:25:49 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestSourceBucketDeleteWithBootstrap (87.73s)
=== RUN   TestSourceAndMetaBucketDeleteWithBootstrap
2021/12/03 14:25:59 clusterSetup.go:397: Waiting for app: TestSourceAndMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/03 14:25:59 clusterSetup.go:405: App: TestSourceAndMetaBucketDeleteWithBootstrap got un-deployed
2021/12/03 14:25:59 eventingSetup.go:46: No eventing-con process running
2021/12/03 14:25:59 eventingSetup.go:418: Function: TestSourceAndMetaBucketDeleteWithBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestSourceAndMetaBucketDeleteWithBootstrap not found"
 }
}
2021/12/03 14:26:01 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/03 14:26:06 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 14:26:08 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/03 14:26:13 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 14:26:13 eventingSetup.go:125: Deploying app: TestSourceAndMetaBucketDeleteWithBootstrap. Item count src bucket: 0 dst bucket: 0
2021/12/03 14:26:14 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestSourceAndMetaBucketDeleteWithBootstrap' in metakv",
  "warnings": null
 }
}
2021/12/03 14:26:33 handler_test.go:988: Deleting source bucket: default
2021/12/03 14:26:34 handler_test.go:990: Deleted source bucket: default
2021/12/03 14:26:34 handler_test.go:992: Deleting metadata bucket: eventing
2021/12/03 14:26:34 handler_test.go:994: Deleted metadata bucket: eventing
2021/12/03 14:26:49 clusterSetup.go:397: Waiting for app: TestSourceAndMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/03 14:26:49 clusterSetup.go:405: App: TestSourceAndMetaBucketDeleteWithBootstrap got un-deployed
2021/12/03 14:27:05 clusterSetup.go:397: Waiting for app: TestSourceAndMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/03 14:27:05 clusterSetup.go:405: App: TestSourceAndMetaBucketDeleteWithBootstrap got un-deployed
2021/12/03 14:27:05 eventingSetup.go:46: No eventing-con process running
2021/12/03 14:27:05 eventingSetup.go:418: Function: TestSourceAndMetaBucketDeleteWithBootstrap delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestSourceAndMetaBucketDeleteWithBootstrap deleting in the background"
}
2021/12/03 14:27:06 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/03 14:27:11 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 14:27:13 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/03 14:27:18 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestSourceAndMetaBucketDeleteWithBootstrap (88.19s)
=== RUN   TestUndeployDuringBootstrap
2021/12/03 14:27:28 clusterSetup.go:397: Waiting for app: TestUndeployDuringBootstrap to get un-deployed
2021/12/03 14:27:28 clusterSetup.go:405: App: TestUndeployDuringBootstrap got un-deployed
2021/12/03 14:27:28 eventingSetup.go:46: No eventing-con process running
2021/12/03 14:27:28 eventingSetup.go:418: Function: TestUndeployDuringBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestUndeployDuringBootstrap not found"
 }
}
2021/12/03 14:27:29 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/03 14:27:34 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 14:27:36 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/03 14:27:41 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 14:27:41 eventingSetup.go:125: Deploying app: TestUndeployDuringBootstrap. Item count src bucket: 0 dst bucket: 0
2021/12/03 14:27:42 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestUndeployDuringBootstrap' in metakv",
  "warnings": null
 }
}
2021/12/03 14:27:42 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/03 14:27:54 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 14:27:54 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 14:27:54 clusterSetup.go:375: Apps undergoing bootstrap: true
2021/12/03 14:27:54 clusterSetup.go:378: App: TestUndeployDuringBootstrap started bootstrap
2021/12/03 14:27:54 clusterSetup.go:375: Apps undergoing bootstrap: true
2021/12/03 14:27:59 clusterSetup.go:375: Apps undergoing bootstrap: true
2021/12/03 14:28:04 clusterSetup.go:367: No apps undergoing bootstrap
2021/12/03 14:28:09 clusterSetup.go:397: Waiting for app: TestUndeployDuringBootstrap to get un-deployed
2021/12/03 14:28:14 clusterSetup.go:397: Waiting for app: TestUndeployDuringBootstrap to get un-deployed
2021/12/03 14:28:19 clusterSetup.go:397: Waiting for app: TestUndeployDuringBootstrap to get un-deployed
2021/12/03 14:28:19 clusterSetup.go:405: App: TestUndeployDuringBootstrap got un-deployed
2021/12/03 14:28:19 eventingSetup.go:46: No eventing-con process running
2021/12/03 14:28:44 clusterSetup.go:397: Waiting for app: TestUndeployDuringBootstrap to get un-deployed
2021/12/03 14:28:44 clusterSetup.go:405: App: TestUndeployDuringBootstrap got un-deployed
2021/12/03 14:28:44 eventingSetup.go:46: No eventing-con process running
2021/12/03 14:28:44 eventingSetup.go:418: Function: TestUndeployDuringBootstrap delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestUndeployDuringBootstrap deleting in the background"
}
2021/12/03 14:28:45 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/03 14:28:55 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/03 14:28:55 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 14:28:57 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/03 14:29:07 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/03 14:29:07 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestUndeployDuringBootstrap (109.48s)
=== RUN   TestDeleteBeforeUndeploy
2021/12/03 14:29:12 eventingSetup.go:125: Deploying app: TestDeleteBeforeUndeploy. Item count src bucket: 0 dst bucket: 0
2021/12/03 14:29:13 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeleteBeforeUndeploy' in metakv",
  "warnings": null
 }
}
2021/12/03 14:29:18 clusterSetup.go:346: Waiting for app: TestDeleteBeforeUndeploy to get deployed
2021/12/03 14:29:23 clusterSetup.go:346: Waiting for app: TestDeleteBeforeUndeploy to get deployed
2021/12/03 14:29:28 clusterSetup.go:346: Waiting for app: TestDeleteBeforeUndeploy to get deployed
2021/12/03 14:29:33 clusterSetup.go:346: Waiting for app: TestDeleteBeforeUndeploy to get deployed
2021/12/03 14:29:33 clusterSetup.go:354: App: TestDeleteBeforeUndeploy got deployed
2021/12/03 14:29:33 eventingSetup.go:418: Function: TestDeleteBeforeUndeploy delete from main store request response code: 422 dump: {
 "name": "ERR_APP_NOT_UNDEPLOYED",
 "code": 22,
 "description": "Function hasn't undeployed",
 "attributes": null,
 "runtime_info": {
  "code": 22,
  "info": "Function: TestDeleteBeforeUndeploy skipping delete request from primary store, as it hasn't been undeployed"
 }
}
2021/12/03 14:29:33 eventingSetup.go:270: Function: TestDeleteBeforeUndeploy update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/03 14:29:33 eventingSetup.go:418: Function: TestDeleteBeforeUndeploy delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDeleteBeforeUndeploy deleting in the background"
}
2021/12/03 14:29:38 clusterSetup.go:397: Waiting for app: TestDeleteBeforeUndeploy to get un-deployed
2021/12/03 14:29:39 clusterSetup.go:405: App: TestDeleteBeforeUndeploy got un-deployed
2021/12/03 14:29:39 eventingSetup.go:46: No eventing-con process running
2021/12/03 14:29:39 eventingSetup.go:418: Function: TestDeleteBeforeUndeploy delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDeleteBeforeUndeploy not found"
 }
}
2021/12/03 14:29:40 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/03 14:29:45 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 14:29:47 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/03 14:29:52 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDeleteBeforeUndeploy (44.71s)
=== RUN   TestUndeployWhenTimersAreFired
2021/12/03 14:30:02 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/03 14:30:02 clusterSetup.go:405: App: TestUndeployWhenTimersAreFired got un-deployed
2021/12/03 14:30:02 eventingSetup.go:46: No eventing-con process running
2021/12/03 14:30:02 eventingSetup.go:418: Function: TestUndeployWhenTimersAreFired delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestUndeployWhenTimersAreFired not found"
 }
}
2021/12/03 14:30:03 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/03 14:30:08 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 14:30:10 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/03 14:30:15 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 14:30:15 eventingSetup.go:125: Deploying app: TestUndeployWhenTimersAreFired. Item count src bucket: 0 dst bucket: 0
2021/12/03 14:30:17 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestUndeployWhenTimersAreFired' in metakv",
  "warnings": null
 }
}
2021/12/03 14:30:22 clusterSetup.go:346: Waiting for app: TestUndeployWhenTimersAreFired to get deployed
2021/12/03 14:30:27 clusterSetup.go:346: Waiting for app: TestUndeployWhenTimersAreFired to get deployed
2021/12/03 14:30:32 clusterSetup.go:346: Waiting for app: TestUndeployWhenTimersAreFired to get deployed
2021/12/03 14:30:37 clusterSetup.go:346: Waiting for app: TestUndeployWhenTimersAreFired to get deployed
2021/12/03 14:30:37 clusterSetup.go:354: App: TestUndeployWhenTimersAreFired got deployed
2021/12/03 14:30:37 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/03 14:31:07 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 14:31:12 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/03 14:31:17 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/03 14:31:22 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/03 14:31:27 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/03 14:31:27 clusterSetup.go:405: App: TestUndeployWhenTimersAreFired got un-deployed
2021/12/03 14:31:27 eventingSetup.go:46: No eventing-con process running
2021/12/03 14:33:12 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/03 14:33:12 clusterSetup.go:405: App: TestUndeployWhenTimersAreFired got un-deployed
2021/12/03 14:33:12 eventingSetup.go:46: No eventing-con process running
2021/12/03 14:33:12 eventingSetup.go:418: Function: TestUndeployWhenTimersAreFired delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestUndeployWhenTimersAreFired deleting in the background"
}
2021/12/03 14:33:13 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/03 14:33:23 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/03 14:33:28 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/03 14:33:28 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 14:33:30 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/03 14:33:40 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3055
2021/12/03 14:33:45 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3055
2021/12/03 14:33:45 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestUndeployWhenTimersAreFired (233.20s)
=== RUN   TestUndeployWithKVFailover
2021/12/03 14:33:55 clusterSetup.go:397: Waiting for app: TestUndeployWithKVFailover to get un-deployed
2021/12/03 14:33:55 clusterSetup.go:405: App: TestUndeployWithKVFailover got un-deployed
2021/12/03 14:33:55 eventingSetup.go:46: No eventing-con process running
2021/12/03 14:33:55 eventingSetup.go:418: Function: TestUndeployWithKVFailover delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestUndeployWithKVFailover not found"
 }
}
2021/12/03 14:33:56 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/03 14:34:01 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 14:34:03 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/03 14:34:08 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 14:34:08 eventingSetup.go:125: Deploying app: TestUndeployWithKVFailover. Item count src bucket: 0 dst bucket: 0
2021/12/03 14:34:09 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestUndeployWithKVFailover' in metakv",
  "warnings": null
 }
}
2021/12/03 14:34:09 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/03 14:34:16 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 14:34:21 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 14:34:26 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 14:34:31 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 14:34:36 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 14:34:41 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 14:34:47 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 1328
2021/12/03 14:34:52 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 3828
2021/12/03 14:34:52 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/03 14:34:52 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: kv to the cluster
2021/12/03 14:34:52 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/03 14:34:57 Rebalance progress: 29.16666666666666
2021/12/03 14:35:02 Rebalance progress: 29.16666666666666
2021/12/03 14:35:07 Rebalance progress: 100
2021/12/03 14:35:07 Failing over: [127.0.0.1:19003]
2021/12/03 14:35:17 Function: TestUndeployWithKVFailover update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/03 14:36:17 Removing node(s): [] from the cluster
2021/12/03 14:36:22 Rebalance progress: 29.16666666666666
2021/12/03 14:36:27 Rebalance progress: 29.16666666666666
2021/12/03 14:36:32 Rebalance progress: 100
2021/12/03 14:36:37 Waiting for app: TestUndeployWithKVFailover to get un-deployed
2021/12/03 14:36:37 App: TestUndeployWithKVFailover got un-deployed
2021/12/03 14:36:37 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 14:36:42 Waiting for app: TestUndeployWithKVFailover to get un-deployed
2021/12/03 14:36:42 App: TestUndeployWithKVFailover got un-deployed
2021/12/03 14:36:42 No eventing-con process running
2021/12/03 14:36:42 Function: TestUndeployWithKVFailover delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestUndeployWithKVFailover deleting in the background"
}
2021/12/03 14:36:43 Context: Bucket flush, response code: 200 dump: 
2021/12/03 14:36:53 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/03 14:36:58 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/03 14:36:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 14:37:00 Context: Bucket flush, response code: 200 dump: 
2021/12/03 14:37:10 Waiting for dst bucket item count to get to: 0 curr count: 3578
2021/12/03 14:37:15 Waiting for dst bucket item count to get to: 0 curr count: 3578
2021/12/03 14:37:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestUndeployWithKVFailover (210.24s)
=== RUN   TestBucketFlushWhileFnDeployed
2021/12/03 14:37:25 Waiting for app: TestBucketFlushWhileFnDeployed to get un-deployed
2021/12/03 14:37:25 App: TestBucketFlushWhileFnDeployed got un-deployed
2021/12/03 14:37:25 No eventing-con process running
2021/12/03 14:37:25 Function: TestBucketFlushWhileFnDeployed delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestBucketFlushWhileFnDeployed not found"
 }
}
2021/12/03 14:37:27 Context: Bucket flush, response code: 200 dump: 
2021/12/03 14:37:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 14:37:33 Context: Bucket flush, response code: 200 dump: 
2021/12/03 14:37:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 14:37:38 Deploying app: TestBucketFlushWhileFnDeployed. Item count src bucket: 0 dst bucket: 0
2021/12/03 14:37:40 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestBucketFlushWhileFnDeployed' in metakv",
  "warnings": null
 }
}
2021/12/03 14:37:45 Waiting for app: TestBucketFlushWhileFnDeployed to get deployed
2021/12/03 14:37:50 Waiting for app: TestBucketFlushWhileFnDeployed to get deployed
2021/12/03 14:37:55 Waiting for app: TestBucketFlushWhileFnDeployed to get deployed
2021/12/03 14:38:00 Waiting for app: TestBucketFlushWhileFnDeployed to get deployed
2021/12/03 14:38:00 App: TestBucketFlushWhileFnDeployed got deployed
2021/12/03 14:38:00 Starting bucket ops to source bucket
2021/12/03 14:38:20 Waiting for dst bucket item count to get to: 20000 curr count: 10760
2021/12/03 14:38:25 Waiting for dst bucket item count to get to: 20000 curr count: 16034
2021/12/03 14:38:25 src & dst bucket item count matched up. src bucket count: 20000 dst bucket count: 20000
2021/12/03 14:38:25 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 14:38:28 Context: Bucket flush, response code: 200 dump: 
2021/12/03 14:38:38 Waiting for dst bucket item count to get to: 0 curr count: 11143
2021/12/03 14:38:43 Waiting for dst bucket item count to get to: 0 curr count: 1143
2021/12/03 14:38:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 14:38:45 Context: Bucket flush, response code: 200 dump: 
2021/12/03 14:38:55 Waiting for dst bucket item count to get to: 0 curr count: 20000
2021/12/03 14:38:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 14:38:55 Starting bucket ops to source bucket
2021/12/03 14:39:08 Waiting for dst bucket item count to get to: 10000 curr count: 4080
2021/12/03 14:39:13 Waiting for dst bucket item count to get to: 10000 curr count: 4080
2021/12/03 14:39:13 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/03 14:39:13 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 14:39:13 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 14:39:18 Waiting for app: TestBucketFlushWhileFnDeployed to get un-deployed
2021/12/03 14:39:23 Waiting for app: TestBucketFlushWhileFnDeployed to get un-deployed
2021/12/03 14:39:28 Waiting for app: TestBucketFlushWhileFnDeployed to get un-deployed
2021/12/03 14:39:33 Waiting for app: TestBucketFlushWhileFnDeployed to get un-deployed
2021/12/03 14:39:33 App: TestBucketFlushWhileFnDeployed got un-deployed
2021/12/03 14:39:33 No eventing-con process running
2021/12/03 14:39:33 Function: TestBucketFlushWhileFnDeployed delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestBucketFlushWhileFnDeployed deleting in the background"
}
2021/12/03 14:39:34 Context: Bucket flush, response code: 200 dump: 
2021/12/03 14:39:44 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/03 14:39:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 14:39:46 Context: Bucket flush, response code: 200 dump: 
2021/12/03 14:39:56 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/03 14:39:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestBucketFlushWhileFnDeployed (160.93s)
=== RUN   TestTimerOverwrite
2021/12/03 14:40:06 Waiting for app: TestTimerOverwrite to get un-deployed
2021/12/03 14:40:06 App: TestTimerOverwrite got un-deployed
2021/12/03 14:40:06 No eventing-con process running
2021/12/03 14:40:06 Function: TestTimerOverwrite delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestTimerOverwrite not found"
 }
}
2021/12/03 14:40:08 Context: Bucket flush, response code: 200 dump: 
2021/12/03 14:40:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 14:40:14 Context: Bucket flush, response code: 200 dump: 
2021/12/03 14:40:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 14:40:19 Deploying app: TestTimerOverwrite. Item count src bucket: 0 dst bucket: 0
2021/12/03 14:40:21 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestTimerOverwrite' in metakv",
  "warnings": null
 }
}
2021/12/03 14:40:26 Waiting for app: TestTimerOverwrite to get deployed
2021/12/03 14:40:31 Waiting for app: TestTimerOverwrite to get deployed
2021/12/03 14:40:36 Waiting for app: TestTimerOverwrite to get deployed
2021/12/03 14:40:41 Waiting for app: TestTimerOverwrite to get deployed
2021/12/03 14:40:41 App: TestTimerOverwrite got deployed
2021/12/03 14:40:51 Starting bucket ops to source bucket
2021/12/03 14:41:00 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 14:41:05 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 14:41:10 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 14:41:15 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 14:41:20 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 14:41:25 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 14:41:30 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 14:41:35 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 14:41:40 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 14:41:45 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 14:41:50 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 14:41:55 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 14:42:00 Waiting for dst bucket item count to get to: 5000 curr count: 2083
2021/12/03 14:42:05 Waiting for dst bucket item count to get to: 5000 curr count: 2083
2021/12/03 14:42:05 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/03 14:42:15 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 14:42:15 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 14:42:20 Waiting for app: TestTimerOverwrite to get un-deployed
2021/12/03 14:42:25 Waiting for app: TestTimerOverwrite to get un-deployed
2021/12/03 14:42:30 Waiting for app: TestTimerOverwrite to get un-deployed
2021/12/03 14:42:35 Waiting for app: TestTimerOverwrite to get un-deployed
2021/12/03 14:42:35 App: TestTimerOverwrite got un-deployed
2021/12/03 14:42:35 No eventing-con process running
2021/12/03 14:42:35 Function: TestTimerOverwrite delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestTimerOverwrite deleting in the background"
}
2021/12/03 14:42:36 Context: Bucket flush, response code: 200 dump: 
2021/12/03 14:42:46 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/03 14:42:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 14:42:48 Context: Bucket flush, response code: 200 dump: 
2021/12/03 14:42:58 Waiting for dst bucket item count to get to: 0 curr count: 3901
2021/12/03 14:43:03 Waiting for dst bucket item count to get to: 0 curr count: 1402
2021/12/03 14:43:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestTimerOverwrite (186.98s)
=== RUN   TestUndeployBackdoorDuringBootstrap
2021/12/03 14:43:08 Adding node: https://127.0.0.1:19003 with role: kv to the cluster
2021/12/03 14:43:08 Removing node(s): [] from the cluster
2021/12/03 14:43:13 Rebalance progress: 100
2021/12/03 14:43:13 Failing over: [127.0.0.1:19003]
2021/12/03 14:43:18 Waiting for app: TestUndeployBackdoorDuringBootstrap to get un-deployed
2021/12/03 14:43:18 App: TestUndeployBackdoorDuringBootstrap got un-deployed
2021/12/03 14:43:18 No eventing-con process running
2021/12/03 14:43:18 Function: TestUndeployBackdoorDuringBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestUndeployBackdoorDuringBootstrap not found"
 }
}
2021/12/03 14:43:20 Context: Bucket flush, response code: 200 dump: 
2021/12/03 14:43:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 14:43:27 Context: Bucket flush, response code: 200 dump: 
2021/12/03 14:43:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 14:43:32 Deploying app: TestUndeployBackdoorDuringBootstrap. Item count src bucket: 0 dst bucket: 0
2021/12/03 14:43:33 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestUndeployBackdoorDuringBootstrap' in metakv",
  "warnings": null
 }
}
2021/12/03 14:43:33 Starting bucket ops to source bucket
2021/12/03 14:43:43 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 14:43:43 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 14:43:43 Context: Post to set retry counter, response code: 200 dump: 
2021/12/03 14:43:43 set retry counter, response &{[] }
2021/12/03 14:44:48 Waiting for app: TestUndeployBackdoorDuringBootstrap to get un-deployed
2021/12/03 14:44:48 App: TestUndeployBackdoorDuringBootstrap got un-deployed
2021/12/03 14:44:48 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 14:44:48 Function: TestUndeployBackdoorDuringBootstrap delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestUndeployBackdoorDuringBootstrap deleting in the background"
}
2021/12/03 14:44:53 Waiting for app: TestUndeployBackdoorDuringBootstrap to get un-deployed
2021/12/03 14:44:53 App: TestUndeployBackdoorDuringBootstrap got un-deployed
2021/12/03 14:44:53 No eventing-con process running
2021/12/03 14:44:53 Function: TestUndeployBackdoorDuringBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestUndeployBackdoorDuringBootstrap not found"
 }
}
2021/12/03 14:44:54 Context: Bucket flush, response code: 200 dump: 
2021/12/03 14:45:04 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/03 14:45:09 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/03 14:45:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 14:45:11 Context: Bucket flush, response code: 200 dump: 
2021/12/03 14:45:21 Waiting for dst bucket item count to get to: 0 curr count: 4979
2021/12/03 14:45:26 Waiting for dst bucket item count to get to: 0 curr count: 4979
2021/12/03 14:45:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 14:45:26 Removing node(s): [] from the cluster
2021/12/03 14:45:31 Rebalance progress: 100
--- PASS: TestUndeployBackdoorDuringBootstrap (148.02s)
=== RUN   TestOnUpdateSrcMutation
2021/12/03 14:45:41 Waiting for app: TestOnUpdateSrcMutation to get un-deployed
2021/12/03 14:45:41 App: TestOnUpdateSrcMutation got un-deployed
2021/12/03 14:45:41 No eventing-con process running
2021/12/03 14:45:41 Function: TestOnUpdateSrcMutation delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateSrcMutation not found"
 }
}
2021/12/03 14:45:43 Context: Bucket flush, response code: 200 dump: 
2021/12/03 14:45:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 14:45:49 Context: Bucket flush, response code: 200 dump: 
2021/12/03 14:45:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 14:45:54 Deploying app: TestOnUpdateSrcMutation. Item count src bucket: 0 dst bucket: 0
2021/12/03 14:45:56 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateSrcMutation' in metakv",
  "warnings": null
 }
}
2021/12/03 14:45:56 Starting bucket ops to source bucket
2021/12/03 14:46:23 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/03 14:46:23 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 14:46:23 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 14:46:28 Waiting for app: TestOnUpdateSrcMutation to get un-deployed
2021/12/03 14:46:33 Waiting for app: TestOnUpdateSrcMutation to get un-deployed
2021/12/03 14:46:38 Waiting for app: TestOnUpdateSrcMutation to get un-deployed
2021/12/03 14:46:43 Waiting for app: TestOnUpdateSrcMutation to get un-deployed
2021/12/03 14:46:43 App: TestOnUpdateSrcMutation got un-deployed
2021/12/03 14:46:43 No eventing-con process running
2021/12/03 14:46:43 Function: TestOnUpdateSrcMutation delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateSrcMutation deleting in the background"
}
2021/12/03 14:46:44 Context: Bucket flush, response code: 200 dump: 
2021/12/03 14:46:54 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/03 14:46:59 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/03 14:46:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 14:47:01 Context: Bucket flush, response code: 200 dump: 
2021/12/03 14:47:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateSrcMutation (94.72s)
=== RUN   TestOnDeleteSrcMutation
2021/12/03 14:47:16 Waiting for app: TestOnDeleteSrcMutation to get un-deployed
2021/12/03 14:47:16 App: TestOnDeleteSrcMutation got un-deployed
2021/12/03 14:47:16 No eventing-con process running
2021/12/03 14:47:16 Function: TestOnDeleteSrcMutation delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnDeleteSrcMutation not found"
 }
}
2021/12/03 14:47:17 Context: Bucket flush, response code: 200 dump: 
2021/12/03 14:47:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 14:47:24 Context: Bucket flush, response code: 200 dump: 
2021/12/03 14:47:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 14:47:29 Deploying app: TestOnDeleteSrcMutation. Item count src bucket: 0 dst bucket: 0
2021/12/03 14:47:30 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnDeleteSrcMutation' in metakv",
  "warnings": null
 }
}
2021/12/03 14:47:30 Starting bucket ops to source bucket
2021/12/03 14:47:54 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/03 14:47:54 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 14:47:54 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 14:47:59 Waiting for app: TestOnDeleteSrcMutation to get un-deployed
2021/12/03 14:48:04 Waiting for app: TestOnDeleteSrcMutation to get un-deployed
2021/12/03 14:48:09 Waiting for app: TestOnDeleteSrcMutation to get un-deployed
2021/12/03 14:48:14 Waiting for app: TestOnDeleteSrcMutation to get un-deployed
2021/12/03 14:48:14 App: TestOnDeleteSrcMutation got un-deployed
2021/12/03 14:48:14 No eventing-con process running
2021/12/03 14:48:14 Function: TestOnDeleteSrcMutation delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnDeleteSrcMutation deleting in the background"
}
2021/12/03 14:48:16 Context: Bucket flush, response code: 200 dump: 
2021/12/03 14:48:26 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/03 14:48:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 14:48:27 Context: Bucket flush, response code: 200 dump: 
2021/12/03 14:48:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnDeleteSrcMutation (86.51s)
=== RUN   TestOnUpdateSrcMutationWithTimer
2021/12/03 14:48:42 Waiting for app: TestOnUpdateSrcMutationWithTimer to get un-deployed
2021/12/03 14:48:42 App: TestOnUpdateSrcMutationWithTimer got un-deployed
2021/12/03 14:48:42 No eventing-con process running
2021/12/03 14:48:42 Function: TestOnUpdateSrcMutationWithTimer delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateSrcMutationWithTimer not found"
 }
}
2021/12/03 14:48:44 Context: Bucket flush, response code: 200 dump: 
2021/12/03 14:48:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 14:48:51 Context: Bucket flush, response code: 200 dump: 
2021/12/03 14:48:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 14:48:56 Deploying app: TestOnUpdateSrcMutationWithTimer. Item count src bucket: 0 dst bucket: 0
2021/12/03 14:48:57 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateSrcMutationWithTimer' in metakv",
  "warnings": null
 }
}
2021/12/03 14:48:57 Starting bucket ops to source bucket
2021/12/03 14:49:34 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/03 14:49:34 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 14:49:34 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 14:49:39 Waiting for app: TestOnUpdateSrcMutationWithTimer to get un-deployed
2021/12/03 14:49:44 Waiting for app: TestOnUpdateSrcMutationWithTimer to get un-deployed
2021/12/03 14:49:49 Waiting for app: TestOnUpdateSrcMutationWithTimer to get un-deployed
2021/12/03 14:49:54 Waiting for app: TestOnUpdateSrcMutationWithTimer to get un-deployed
2021/12/03 14:49:54 App: TestOnUpdateSrcMutationWithTimer got un-deployed
2021/12/03 14:49:54 No eventing-con process running
2021/12/03 14:49:54 Function: TestOnUpdateSrcMutationWithTimer delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateSrcMutationWithTimer deleting in the background"
}
2021/12/03 14:49:55 Context: Bucket flush, response code: 200 dump: 
2021/12/03 14:50:05 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/03 14:50:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 14:50:07 Context: Bucket flush, response code: 200 dump: 
2021/12/03 14:50:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateSrcMutationWithTimer (99.75s)
=== RUN   TestOnDeleteSrcMutationsWithTimer
2021/12/03 14:50:22 Waiting for app: TestOnDeleteSrcMutationsWithTimer to get un-deployed
2021/12/03 14:50:22 App: TestOnDeleteSrcMutationsWithTimer got un-deployed
2021/12/03 14:50:22 No eventing-con process running
2021/12/03 14:50:22 Function: TestOnDeleteSrcMutationsWithTimer delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnDeleteSrcMutationsWithTimer not found"
 }
}
2021/12/03 14:50:24 Context: Bucket flush, response code: 200 dump: 
2021/12/03 14:50:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 14:50:30 Context: Bucket flush, response code: 200 dump: 
2021/12/03 14:50:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 14:50:35 Deploying app: TestOnDeleteSrcMutationsWithTimer. Item count src bucket: 0 dst bucket: 0
2021/12/03 14:50:37 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnDeleteSrcMutationsWithTimer' in metakv",
  "warnings": null
 }
}
2021/12/03 14:50:37 Starting bucket ops to source bucket
2021/12/03 14:51:25 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/03 14:51:25 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 14:51:25 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 14:51:30 Waiting for app: TestOnDeleteSrcMutationsWithTimer to get un-deployed
2021/12/03 14:51:35 Waiting for app: TestOnDeleteSrcMutationsWithTimer to get un-deployed
2021/12/03 14:51:40 Waiting for app: TestOnDeleteSrcMutationsWithTimer to get un-deployed
2021/12/03 14:51:45 Waiting for app: TestOnDeleteSrcMutationsWithTimer to get un-deployed
2021/12/03 14:51:45 App: TestOnDeleteSrcMutationsWithTimer got un-deployed
2021/12/03 14:51:45 No eventing-con process running
2021/12/03 14:51:45 Function: TestOnDeleteSrcMutationsWithTimer delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnDeleteSrcMutationsWithTimer deleting in the background"
}
2021/12/03 14:51:47 Context: Bucket flush, response code: 200 dump: 
2021/12/03 14:51:57 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/03 14:51:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 14:51:59 Context: Bucket flush, response code: 200 dump: 
2021/12/03 14:52:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnDeleteSrcMutationsWithTimer (111.39s)
=== RUN   TestInterHandlerRecursion
2021/12/03 14:52:14 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/03 14:52:14 App: TestInterHandlerRecursion_source_bucket_update_op got un-deployed
2021/12/03 14:52:14 No eventing-con process running
2021/12/03 14:52:14 Function: TestInterHandlerRecursion_source_bucket_update_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestInterHandlerRecursion_source_bucket_update_op not found"
 }
}
2021/12/03 14:52:15 Context: Bucket flush, response code: 200 dump: 
2021/12/03 14:52:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 14:52:22 Context: Bucket flush, response code: 200 dump: 
2021/12/03 14:52:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 14:52:32 Waiting for app: TestInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/03 14:52:32 App: TestInterHandlerRecursion_src_bucket_op_on_delete got un-deployed
2021/12/03 14:52:32 No eventing-con process running
2021/12/03 14:52:32 Function: TestInterHandlerRecursion_src_bucket_op_on_delete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestInterHandlerRecursion_src_bucket_op_on_delete not found"
 }
}
2021/12/03 14:52:33 Context: Bucket flush, response code: 200 dump: 
2021/12/03 14:52:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 14:52:40 Context: Bucket flush, response code: 200 dump: 
2021/12/03 14:52:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 14:52:45 Deploying app: TestInterHandlerRecursion_source_bucket_update_op. Item count src bucket: 0 dst bucket: 0
2021/12/03 14:52:46 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterHandlerRecursion_source_bucket_update_op' in metakv",
  "warnings": null
 }
}
2021/12/03 14:52:46 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterHandlerRecursion_source_bucket_update_op' in metakv",
  "warnings": null
 }
} err 
2021/12/03 14:52:51 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/03 14:52:56 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/03 14:53:01 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/03 14:53:06 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/03 14:53:06 App: TestInterHandlerRecursion_source_bucket_update_op got deployed
2021/12/03 14:53:06 Deploying app: TestInterHandlerRecursion_src_bucket_op_on_delete. Item count src bucket: 0 dst bucket: 0
2021/12/03 14:53:06 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_INTER_FUNCTION_RECURSION",
 "code": 50,
 "description": "Inter function recursion error, only one function is allowed to do source bucket mutation/delete on a bucket",
 "attributes": null,
 "runtime_info": {
  "code": 50,
  "info": "Inter handler recursion error"
 }
}
2021/12/03 14:53:06 response body {
 "name": "ERR_INTER_FUNCTION_RECURSION",
 "code": 50,
 "description": "Inter function recursion error, only one function is allowed to do source bucket mutation/delete on a bucket",
 "attributes": null,
 "runtime_info": {
  "code": 50,
  "info": "Inter handler recursion error"
 }
} err 
2021/12/03 14:53:06 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 14:53:11 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/03 14:53:16 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/03 14:53:21 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/03 14:53:26 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/03 14:53:26 App: TestInterHandlerRecursion_source_bucket_update_op got un-deployed
2021/12/03 14:53:26 No eventing-con process running
2021/12/03 14:53:26 Function: TestInterHandlerRecursion_source_bucket_update_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestInterHandlerRecursion_source_bucket_update_op deleting in the background"
}
2021/12/03 14:53:28 Context: Bucket flush, response code: 200 dump: 
2021/12/03 14:53:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 14:53:34 Context: Bucket flush, response code: 200 dump: 
2021/12/03 14:53:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 14:53:44 Waiting for app: TestInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/03 14:53:44 App: TestInterHandlerRecursion_src_bucket_op_on_delete got un-deployed
2021/12/03 14:53:44 No eventing-con process running
2021/12/03 14:53:44 Function: TestInterHandlerRecursion_src_bucket_op_on_delete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestInterHandlerRecursion_src_bucket_op_on_delete not found"
 }
}
2021/12/03 14:53:46 Context: Bucket flush, response code: 200 dump: 
2021/12/03 14:53:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 14:53:53 Context: Bucket flush, response code: 200 dump: 
2021/12/03 14:53:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestInterHandlerRecursion (114.09s)
=== RUN   TestInterBucketRecursion
2021/12/03 14:54:03 Deploying app: TestInterBucketRecursion_function1. Item count src bucket: 0 dst bucket: 0
2021/12/03 14:54:04 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterBucketRecursion_function1' in metakv",
  "warnings": null
 }
}
2021/12/03 14:54:04 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterBucketRecursion_function1' in metakv",
  "warnings": null
 }
} err 
2021/12/03 14:54:04 Deploying app: TestInterBucketRecursion_function2. Item count src bucket: 0 dst bucket: 0
2021/12/03 14:54:04 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_INTER_BUCKET_RECURSION",
 "code": 51,
 "description": "Inter bucket recursion error, deployment of current handler will cause inter bucket recursion",
 "attributes": null,
 "runtime_info": {
  "code": 51,
  "info": "Inter bucket recursion error; function: TestInterBucketRecursion_function2 causes a cycle involving functions: [TestInterBucketRecursion_function1], hence deployment is disallowed"
 }
}
2021/12/03 14:54:09 Waiting for app: TestInterBucketRecursion_function1 to get deployed
2021/12/03 14:54:14 Waiting for app: TestInterBucketRecursion_function1 to get deployed
2021/12/03 14:54:19 Waiting for app: TestInterBucketRecursion_function1 to get deployed
2021/12/03 14:54:24 Waiting for app: TestInterBucketRecursion_function1 to get deployed
2021/12/03 14:54:24 App: TestInterBucketRecursion_function1 got deployed
2021/12/03 14:54:24 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 14:54:29 Waiting for app: TestInterBucketRecursion_function1 to get un-deployed
2021/12/03 14:54:34 Waiting for app: TestInterBucketRecursion_function1 to get un-deployed
2021/12/03 14:54:39 Waiting for app: TestInterBucketRecursion_function1 to get un-deployed
2021/12/03 14:54:44 Waiting for app: TestInterBucketRecursion_function1 to get un-deployed
2021/12/03 14:54:44 App: TestInterBucketRecursion_function1 got un-deployed
2021/12/03 14:54:44 No eventing-con process running
2021/12/03 14:54:44 Function: TestInterBucketRecursion_function1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestInterBucketRecursion_function1 deleting in the background"
}
2021/12/03 14:54:45 Context: Bucket flush, response code: 200 dump: 
2021/12/03 14:54:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 14:54:52 Context: Bucket flush, response code: 200 dump: 
2021/12/03 14:54:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 14:55:02 Waiting for app: TestInterBucketRecursion_function2 to get un-deployed
2021/12/03 14:55:02 App: TestInterBucketRecursion_function2 got un-deployed
2021/12/03 14:55:02 No eventing-con process running
2021/12/03 14:55:02 Function: TestInterBucketRecursion_function2 delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestInterBucketRecursion_function2 not found"
 }
}
2021/12/03 14:55:03 Context: Bucket flush, response code: 200 dump: 
2021/12/03 14:55:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 14:55:10 Context: Bucket flush, response code: 200 dump: 
2021/12/03 14:55:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestInterBucketRecursion (77.63s)
=== RUN   TestLargeHandler
2021/12/03 14:55:15 Changed force_compress value to false
2021/12/03 14:55:15 Deploying app: TestLargeHandler_function. Item count src bucket: 0 dst bucket: 0
2021/12/03 14:55:15 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_APPCODE_SIZE",
 "code": 39,
 "description": "Function Code size is more than the configured limit",
 "attributes": null,
 "runtime_info": {
  "code": 39,
  "info": "Function: TestLargeHandler_function handler Code size is more than 131072. Code Size: 131768"
 }
}
2021/12/03 14:55:15 Changed force_compress value to true
2021/12/03 14:55:15 Deploying app: TestLargeHandler_function. Item count src bucket: 0 dst bucket: 0
2021/12/03 14:55:17 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestLargeHandler_function' in metakv",
  "warnings": null
 }
}
2021/12/03 14:55:22 Waiting for app: TestLargeHandler_function to get deployed
2021/12/03 14:55:27 Waiting for app: TestLargeHandler_function to get deployed
2021/12/03 14:55:32 Waiting for app: TestLargeHandler_function to get deployed
2021/12/03 14:55:37 Waiting for app: TestLargeHandler_function to get deployed
2021/12/03 14:55:37 App: TestLargeHandler_function got deployed
2021/12/03 14:55:38 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 14:55:43 Waiting for app: TestLargeHandler_function to get un-deployed
2021/12/03 14:55:48 Waiting for app: TestLargeHandler_function to get un-deployed
2021/12/03 14:55:53 Waiting for app: TestLargeHandler_function to get un-deployed
2021/12/03 14:55:58 Waiting for app: TestLargeHandler_function to get un-deployed
2021/12/03 14:55:58 App: TestLargeHandler_function got un-deployed
2021/12/03 14:55:58 No eventing-con process running
2021/12/03 14:55:58 Function: TestLargeHandler_function delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestLargeHandler_function deleting in the background"
}
2021/12/03 14:55:59 Context: Bucket flush, response code: 200 dump: 
2021/12/03 14:56:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 14:56:06 Context: Bucket flush, response code: 200 dump: 
2021/12/03 14:56:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestLargeHandler (55.47s)
=== RUN   TestAllowInterHandlerRecursion
2021/12/03 14:56:21 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/03 14:56:21 App: TestAllowInterHandlerRecursion_source_bucket_update_op got un-deployed
2021/12/03 14:56:21 No eventing-con process running
2021/12/03 14:56:21 Function: TestAllowInterHandlerRecursion_source_bucket_update_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestAllowInterHandlerRecursion_source_bucket_update_op not found"
 }
}
2021/12/03 14:56:22 Context: Bucket flush, response code: 200 dump: 
2021/12/03 14:56:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 14:56:29 Context: Bucket flush, response code: 200 dump: 
2021/12/03 14:56:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 14:56:39 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/03 14:56:39 App: TestAllowInterHandlerRecursion_src_bucket_op_on_delete got un-deployed
2021/12/03 14:56:39 No eventing-con process running
2021/12/03 14:56:39 Function: TestAllowInterHandlerRecursion_src_bucket_op_on_delete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestAllowInterHandlerRecursion_src_bucket_op_on_delete not found"
 }
}
2021/12/03 14:56:40 Context: Bucket flush, response code: 200 dump: 
2021/12/03 14:56:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 14:56:47 Context: Bucket flush, response code: 200 dump: 
2021/12/03 14:56:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 14:56:52 Deploying app: TestAllowInterHandlerRecursion_source_bucket_update_op. Item count src bucket: 0 dst bucket: 0
2021/12/03 14:56:53 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterHandlerRecursion_source_bucket_update_op' in metakv",
  "warnings": null
 }
}
2021/12/03 14:56:53 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterHandlerRecursion_source_bucket_update_op' in metakv",
  "warnings": null
 }
} err 
2021/12/03 14:56:58 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/03 14:57:03 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/03 14:57:08 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/03 14:57:13 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/03 14:57:13 App: TestAllowInterHandlerRecursion_source_bucket_update_op got deployed
2021/12/03 14:57:13 Deploying app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete. Item count src bucket: 0 dst bucket: 0
2021/12/03 14:57:15 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterHandlerRecursion_src_bucket_op_on_delete' in metakv",
  "warnings": null
 }
}
2021/12/03 14:57:15 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterHandlerRecursion_src_bucket_op_on_delete' in metakv",
  "warnings": null
 }
} err 
2021/12/03 14:57:20 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get deployed
2021/12/03 14:57:25 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get deployed
2021/12/03 14:57:30 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get deployed
2021/12/03 14:57:35 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get deployed
2021/12/03 14:57:35 App: TestAllowInterHandlerRecursion_src_bucket_op_on_delete got deployed
2021/12/03 14:57:35 TestAllowInterHandlerRecursion_src_bucket_op_on_delete is deployed
2021/12/03 14:57:35 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 14:57:40 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/03 14:57:45 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/03 14:57:50 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/03 14:57:55 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/03 14:57:55 App: TestAllowInterHandlerRecursion_source_bucket_update_op got un-deployed
2021/12/03 14:57:55 No eventing-con process running
2021/12/03 14:57:55 Function: TestAllowInterHandlerRecursion_source_bucket_update_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAllowInterHandlerRecursion_source_bucket_update_op deleting in the background"
}
2021/12/03 14:57:57 Context: Bucket flush, response code: 200 dump: 
2021/12/03 14:58:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 14:58:04 Context: Bucket flush, response code: 200 dump: 
2021/12/03 14:58:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 14:58:09 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 14:58:14 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/03 14:58:19 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/03 14:58:24 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/03 14:58:29 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/03 14:58:29 App: TestAllowInterHandlerRecursion_src_bucket_op_on_delete got un-deployed
2021/12/03 14:58:29 No eventing-con process running
2021/12/03 14:58:29 Function: TestAllowInterHandlerRecursion_src_bucket_op_on_delete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAllowInterHandlerRecursion_src_bucket_op_on_delete deleting in the background"
}
2021/12/03 14:58:31 Context: Bucket flush, response code: 200 dump: 
2021/12/03 14:58:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 14:58:37 Context: Bucket flush, response code: 200 dump: 
2021/12/03 14:58:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAllowInterHandlerRecursion (151.65s)
=== RUN   TestAllowInterBucketRecursion
2021/12/03 14:58:47 Deploying app: TestAllowInterBucketRecursion_function1. Item count src bucket: 0 dst bucket: 0
2021/12/03 14:58:49 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterBucketRecursion_function1' in metakv",
  "warnings": null
 }
}
2021/12/03 14:58:49 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterBucketRecursion_function1' in metakv",
  "warnings": null
 }
} err 
2021/12/03 14:58:54 Waiting for app: TestAllowInterBucketRecursion_function1 to get deployed
2021/12/03 14:58:59 Waiting for app: TestAllowInterBucketRecursion_function1 to get deployed
2021/12/03 14:59:04 Waiting for app: TestAllowInterBucketRecursion_function1 to get deployed
2021/12/03 14:59:09 Waiting for app: TestAllowInterBucketRecursion_function1 to get deployed
2021/12/03 14:59:09 App: TestAllowInterBucketRecursion_function1 got deployed
2021/12/03 14:59:09 Deploying app: TestAllowInterBucketRecursion_function2. Item count src bucket: 0 dst bucket: 0
2021/12/03 14:59:10 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterBucketRecursion_function2' in metakv",
  "warnings": null
 }
}
2021/12/03 14:59:10 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterBucketRecursion_function2' in metakv",
  "warnings": null
 }
} err 
2021/12/03 14:59:15 Waiting for app: TestAllowInterBucketRecursion_function2 to get deployed
2021/12/03 14:59:20 Waiting for app: TestAllowInterBucketRecursion_function2 to get deployed
2021/12/03 14:59:25 Waiting for app: TestAllowInterBucketRecursion_function2 to get deployed
2021/12/03 14:59:30 Waiting for app: TestAllowInterBucketRecursion_function2 to get deployed
2021/12/03 14:59:30 App: TestAllowInterBucketRecursion_function2 got deployed
2021/12/03 14:59:30 TestAllowInterBucketRecursion_function2 is deployed
2021/12/03 14:59:30 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 14:59:35 Waiting for app: TestAllowInterBucketRecursion_function1 to get un-deployed
2021/12/03 14:59:40 Waiting for app: TestAllowInterBucketRecursion_function1 to get un-deployed
2021/12/03 14:59:45 Waiting for app: TestAllowInterBucketRecursion_function1 to get un-deployed
2021/12/03 14:59:50 Waiting for app: TestAllowInterBucketRecursion_function1 to get un-deployed
2021/12/03 14:59:50 App: TestAllowInterBucketRecursion_function1 got un-deployed
2021/12/03 14:59:50 No eventing-con process running
2021/12/03 14:59:50 Function: TestAllowInterBucketRecursion_function1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAllowInterBucketRecursion_function1 deleting in the background"
}
2021/12/03 14:59:51 Context: Bucket flush, response code: 200 dump: 
2021/12/03 14:59:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 14:59:59 Context: Bucket flush, response code: 200 dump: 
2021/12/03 15:00:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 15:00:05 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 15:00:10 Waiting for app: TestAllowInterBucketRecursion_function2 to get un-deployed
2021/12/03 15:00:15 Waiting for app: TestAllowInterBucketRecursion_function2 to get un-deployed
2021/12/03 15:00:20 Waiting for app: TestAllowInterBucketRecursion_function2 to get un-deployed
2021/12/03 15:00:25 Waiting for app: TestAllowInterBucketRecursion_function2 to get un-deployed
2021/12/03 15:00:25 App: TestAllowInterBucketRecursion_function2 got un-deployed
2021/12/03 15:00:25 No eventing-con process running
2021/12/03 15:00:25 Function: TestAllowInterBucketRecursion_function2 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAllowInterBucketRecursion_function2 deleting in the background"
}
2021/12/03 15:00:26 Context: Bucket flush, response code: 200 dump: 
2021/12/03 15:00:31 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 15:00:33 Context: Bucket flush, response code: 200 dump: 
2021/12/03 15:00:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAllowInterBucketRecursion (115.45s)
=== RUN   TestInterBucketRecursion2
2021/12/03 15:00:44 Context: Bucket flush, response code: 200 dump: 
2021/12/03 15:00:50 Context: Bucket flush, response code: 200 dump: 
2021/12/03 15:00:57 Context: Bucket flush, response code: 200 dump: 
2021/12/03 15:01:03 Context: Bucket flush, response code: 200 dump: 
2021/12/03 15:01:08 Deploying app: TestInterBucketRecursion2_function1. Item count src bucket: 0 dst bucket: 0
2021/12/03 15:01:09 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterBucketRecursion2_function1' in metakv",
  "warnings": null
 }
}
2021/12/03 15:01:14 Waiting for app: TestInterBucketRecursion2_function1 to get deployed
2021/12/03 15:01:19 Waiting for app: TestInterBucketRecursion2_function1 to get deployed
2021/12/03 15:01:24 Waiting for app: TestInterBucketRecursion2_function1 to get deployed
2021/12/03 15:01:29 Waiting for app: TestInterBucketRecursion2_function1 to get deployed
2021/12/03 15:01:29 App: TestInterBucketRecursion2_function1 got deployed
2021/12/03 15:01:29 Deploying app: TestInterBucketRecursion2_function2. Item count src bucket: 0 dst bucket: 0
2021/12/03 15:01:30 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterBucketRecursion2_function2' in metakv",
  "warnings": null
 }
}
2021/12/03 15:01:35 Waiting for app: TestInterBucketRecursion2_function2 to get deployed
2021/12/03 15:01:40 Waiting for app: TestInterBucketRecursion2_function2 to get deployed
2021/12/03 15:01:45 Waiting for app: TestInterBucketRecursion2_function2 to get deployed
2021/12/03 15:01:50 Waiting for app: TestInterBucketRecursion2_function2 to get deployed
2021/12/03 15:01:50 App: TestInterBucketRecursion2_function2 got deployed
2021/12/03 15:01:50 Deploying app: TestInterBucketRecursion2_function3. Item count src bucket: 0 dst bucket: 0
2021/12/03 15:01:51 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterBucketRecursion2_function3' in metakv",
  "warnings": null
 }
}
2021/12/03 15:01:56 Waiting for app: TestInterBucketRecursion2_function3 to get deployed
2021/12/03 15:02:01 Waiting for app: TestInterBucketRecursion2_function3 to get deployed
2021/12/03 15:02:06 Waiting for app: TestInterBucketRecursion2_function3 to get deployed
2021/12/03 15:02:11 Waiting for app: TestInterBucketRecursion2_function3 to get deployed
2021/12/03 15:02:11 App: TestInterBucketRecursion2_function3 got deployed
2021/12/03 15:02:11 Deploying app: TestInterBucketRecursion2_function4. Item count src bucket: 0 dst bucket: 0
2021/12/03 15:02:11 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_INTER_BUCKET_RECURSION",
 "code": 51,
 "description": "Inter bucket recursion error, deployment of current handler will cause inter bucket recursion",
 "attributes": null,
 "runtime_info": {
  "code": 51,
  "info": "Inter bucket recursion error; function: TestInterBucketRecursion2_function4 causes a cycle involving functions: [TestInterBucketRecursion2_function1 TestInterBucketRecursion2_function2 TestInterBucketRecursion2_function3], hence deployment is disallowed"
 }
}
2021/12/03 15:02:11 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 15:02:16 Waiting for app: TestInterBucketRecursion2_function1 to get un-deployed
2021/12/03 15:02:21 Waiting for app: TestInterBucketRecursion2_function1 to get un-deployed
2021/12/03 15:02:26 Waiting for app: TestInterBucketRecursion2_function1 to get un-deployed
2021/12/03 15:02:31 Waiting for app: TestInterBucketRecursion2_function1 to get un-deployed
2021/12/03 15:02:31 App: TestInterBucketRecursion2_function1 got un-deployed
2021/12/03 15:02:31 No eventing-con process running
2021/12/03 15:02:31 Function: TestInterBucketRecursion2_function1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestInterBucketRecursion2_function1 deleting in the background"
}
2021/12/03 15:02:33 Context: Bucket flush, response code: 200 dump: 
2021/12/03 15:02:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 15:02:40 Context: Bucket flush, response code: 200 dump: 
2021/12/03 15:02:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 15:02:45 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 15:02:50 Waiting for app: TestInterBucketRecursion2_function2 to get un-deployed
2021/12/03 15:02:55 Waiting for app: TestInterBucketRecursion2_function2 to get un-deployed
2021/12/03 15:03:00 Waiting for app: TestInterBucketRecursion2_function2 to get un-deployed
2021/12/03 15:03:05 Waiting for app: TestInterBucketRecursion2_function2 to get un-deployed
2021/12/03 15:03:05 App: TestInterBucketRecursion2_function2 got un-deployed
2021/12/03 15:03:05 No eventing-con process running
2021/12/03 15:03:05 Function: TestInterBucketRecursion2_function2 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestInterBucketRecursion2_function2 deleting in the background"
}
2021/12/03 15:03:06 Context: Bucket flush, response code: 200 dump: 
2021/12/03 15:03:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 15:03:13 Context: Bucket flush, response code: 200 dump: 
2021/12/03 15:03:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 15:03:18 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 15:03:23 Waiting for app: TestInterBucketRecursion2_function3 to get un-deployed
2021/12/03 15:03:28 Waiting for app: TestInterBucketRecursion2_function3 to get un-deployed
2021/12/03 15:03:33 Waiting for app: TestInterBucketRecursion2_function3 to get un-deployed
2021/12/03 15:03:38 Waiting for app: TestInterBucketRecursion2_function3 to get un-deployed
2021/12/03 15:03:38 App: TestInterBucketRecursion2_function3 got un-deployed
2021/12/03 15:03:38 No eventing-con process running
2021/12/03 15:03:38 Function: TestInterBucketRecursion2_function3 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestInterBucketRecursion2_function3 deleting in the background"
}
2021/12/03 15:03:40 Context: Bucket flush, response code: 200 dump: 
2021/12/03 15:03:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 15:03:47 Context: Bucket flush, response code: 200 dump: 
2021/12/03 15:03:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 15:03:57 Waiting for app: TestInterBucketRecursion2_function4 to get un-deployed
2021/12/03 15:03:57 App: TestInterBucketRecursion2_function4 got un-deployed
2021/12/03 15:03:57 No eventing-con process running
2021/12/03 15:03:57 Function: TestInterBucketRecursion2_function4 delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestInterBucketRecursion2_function4 not found"
 }
}
2021/12/03 15:03:58 Context: Bucket flush, response code: 200 dump: 
2021/12/03 15:04:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 15:04:05 Context: Bucket flush, response code: 200 dump: 
2021/12/03 15:04:10 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestInterBucketRecursion2 (214.13s)
=== RUN   TestBucketDeleteWithRebOut
2021/12/03 15:04:17 Removing node(s): [127.0.0.1:19001 127.0.0.1:19002 127.0.0.1:19003] from the cluster
2021/12/03 15:04:22 Rebalance progress: 100

VB distribution across eventing nodes and their workers::

2021/12/03 15:04:25 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/03 15:04:40 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/03 15:04:55 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/03 15:05:10 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/03 15:05:25 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/03 15:05:40 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/03 15:05:55 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/03 15:06:10 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/03 15:06:25 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/03 15:06:40 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/03 15:06:55 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/03 15:07:10 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/03 15:07:25 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/03 15:07:40 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/03 15:07:55 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/03 15:08:10 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/03 15:08:25 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/03 15:08:40 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/03 15:08:55 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/03 15:09:10 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/03 15:09:25 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/03 15:09:40 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/03 15:09:55 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]

DCP Stream statuses:

vbucket curr owner:


2021/12/03 15:10:10 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 15:10:10 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/03 15:10:10 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/03 15:10:15 Rebalance progress: 100

VB distribution across eventing nodes and their workers::


DCP Stream statuses:

vbucket curr owner:


2021/12/03 15:10:15 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 15:10:15 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/03 15:10:16 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/03 15:10:21 Rebalance progress: 100

VB distribution across eventing nodes and their workers::


DCP Stream statuses:

vbucket curr owner:


2021/12/03 15:10:21 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 15:10:26 clusterSetup.go:397: Waiting for app: TestBucketDeleteWithRebOut to get un-deployed
2021/12/03 15:10:26 clusterSetup.go:405: App: TestBucketDeleteWithRebOut got un-deployed
2021/12/03 15:10:26 eventingSetup.go:46: No eventing-con process running
2021/12/03 15:10:26 eventingSetup.go:418: Function: TestBucketDeleteWithRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestBucketDeleteWithRebOut not found"
 }
}
2021/12/03 15:10:27 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/03 15:10:32 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 15:10:34 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/03 15:10:39 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 15:10:44 eventingSetup.go:125: Deploying app: TestBucketDeleteWithRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/03 15:10:45 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestBucketDeleteWithRebOut' in metakv",
  "warnings": null
 }
}
2021/12/03 15:10:50 clusterSetup.go:346: Waiting for app: TestBucketDeleteWithRebOut to get deployed
2021/12/03 15:10:55 clusterSetup.go:346: Waiting for app: TestBucketDeleteWithRebOut to get deployed
2021/12/03 15:11:00 clusterSetup.go:346: Waiting for app: TestBucketDeleteWithRebOut to get deployed
2021/12/03 15:11:05 clusterSetup.go:346: Waiting for app: TestBucketDeleteWithRebOut to get deployed
2021/12/03 15:11:05 clusterSetup.go:354: App: TestBucketDeleteWithRebOut got deployed

VB distribution across eventing nodes and their workers::


DCP Stream statuses:
	stream status: running
	len: 1024
	vb list dump: [0-1023]

vbucket curr owner:
Producer node: 127.0.0.1:9302	Node UUID: 64f26d3a3eef20ed66f1588e56d18517
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [0-113]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [114-227]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 114
	v: [228-341]
Producer node: 172.31.2.134:9300	Node UUID: 64cb1bf93ae4eb8a6053cb5564021e18
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 113
	v: [911-1023]
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [683-796]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [797-910]
Producer node: 127.0.0.1:9303	Node UUID: c4be832488fb52bd9b1d204987f6b740
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [342-455]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [456-569]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 113
	v: [570-682]


2021/12/03 15:11:05 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 15:11:05 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/03 15:11:10 Rebalance progress: 29.16666666666666
2021/12/03 15:11:15 Rebalance progress: 29.16666666666666
2021/12/03 15:11:20 Rebalance progress: 100

VB distribution across eventing nodes and their workers::


DCP Stream statuses:
	stream status: running
	len: 1024
	vb list dump: [0-1023]

vbucket curr owner:
Producer node: 127.0.0.1:9302	Node UUID: 64f26d3a3eef20ed66f1588e56d18517
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [0-113]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [114-227]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 114
	v: [228-341]
Producer node: 172.31.2.134:9300	Node UUID: 64cb1bf93ae4eb8a6053cb5564021e18
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 113
	v: [911-1023]
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [683-796]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [797-910]
Producer node: 127.0.0.1:9303	Node UUID: c4be832488fb52bd9b1d204987f6b740
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [456-569]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 113
	v: [570-682]
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [342-455]


2021/12/03 15:11:20 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 15:11:20 handler_test.go:1674: Deleting source bucket: default
2021/12/03 15:11:22 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/03 15:11:23 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/03 15:11:28 Rebalance progress: 29.16666666666666
2021/12/03 15:11:33 Rebalance progress: 29.16666666666666
2021/12/03 15:11:38 Rebalance progress: 100

VB distribution across eventing nodes and their workers::


DCP Stream statuses:
	stream status: stopped
	len: 135
	vb list dump: [19, 25, 27-28, 55, 60-61, 77-78, 84, 89, 91-92, 101, 114, 118-119, 123, 126, 132-133, 136, 139, 152, 175, 187, 201-202, 207, 209, 220, 223, 225, 229, 235, 244, 254, 274, 280, 282, 285, 299, 301, 303, 307, 311, 343, 350, 363, 368-369, 371, 386, 402, 412, 436, 439, 445-446, 448, 450, 457-458, 461-462, 464, 468, 470-471, 474, 482, 489-490, 497, 499, 503-505, 510, 573-574, 588-589, 592, 594, 602, 604, 608, 617, 627, 629, 632, 636, 663, 681, 686, 691-694, 710, 726, 729, 740, 763, 769, 779, 793, 811, 820, 823, 843, 847, 870, 892-893, 897, 899, 903, 905, 909, 913, 918, 920, 922, 929, 957, 971, 992, 1003, 1012, 1018, 1020-1022]
	stream status: running
	len: 889
	vb list dump: [0-18, 20-24, 26, 29-54, 56-59, 62-76, 79-83, 85-88, 90, 93-100, 102-113, 115-117, 120-122, 124-125, 127-131, 134-135, 137-138, 140-151, 153-174, 176-186, 188-200, 203-206, 208, 210-219, 221-222, 224, 226-228, 230-234, 236-243, 245-253, 255-273, 275-279, 281, 283-284, 286-298, 300, 302, 304-306, 308-310, 312-342, 344-349, 351-362, 364-367, 370, 372-385, 387-401, 403-411, 413-435, 437-438, 440-444, 447, 449, 451-456, 459-460, 463, 465-467, 469, 472-473, 475-481, 483-488, 491-496, 498, 500-502, 506-509, 511-572, 575-587, 590-591, 593, 595-601, 603, 605-607, 609-616, 618-626, 628, 630-631, 633-635, 637-662, 664-680, 682-685, 687-690, 695-709, 711-725, 727-728, 730-739, 741-762, 764-768, 770-778, 780-792, 794-810, 812-819, 821-822, 824-842, 844-846, 848-869, 871-891, 894-896, 898, 900-902, 904, 906-908, 910-912, 914-917, 919, 921, 923-928, 930-956, 958-970, 972-991, 993-1002, 1004-1011, 1013-1017, 1019, 1023]

vbucket curr owner:
Producer node: 172.31.2.134:9300	Node UUID: 64cb1bf93ae4eb8a6053cb5564021e18
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 99
	v: [911-912, 914-917, 919, 921, 923-928, 930-956, 958-970, 972-991, 993-1002, 1004-1011, 1013-1017, 1019, 1023]
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 101
	v: [683-685, 687-690, 695-709, 711-725, 727-728, 730-739, 741-762, 764-768, 770-778, 780-792, 794-796]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 101
	v: [797-810, 812-819, 821-822, 824-842, 844-846, 848-869, 871-891, 894-896, 898, 900-902, 904, 906-908, 910]
Producer node: 127.0.0.1:9303	Node UUID: c4be832488fb52bd9b1d204987f6b740
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 99
	v: [342, 344-349, 351-362, 364-367, 370, 372-385, 387-401, 403-411, 413-435, 437-438, 440-444, 447, 449, 451-455]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 96
	v: [456, 459-460, 463, 465-467, 469, 472-473, 475-481, 483-488, 491-496, 498, 500-502, 506-509, 511-569]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 97
	v: [570-572, 575-587, 590-591, 593, 595-601, 603, 605-607, 609-616, 618-626, 628, 630-631, 633-635, 637-662, 664-680, 682]
Producer node: 127.0.0.1:9302	Node UUID: 64f26d3a3eef20ed66f1588e56d18517
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 100
	v: [0-18, 20-24, 26, 29-54, 56-59, 62-76, 79-83, 85-88, 90, 93-100, 102-113]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 95
	v: [115-117, 120-122, 124-125, 127-131, 134-135, 137-138, 140-151, 153-174, 176-186, 188-200, 203-206, 208, 210-219, 221-222, 224, 226-227]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 101
	v: [228, 230-234, 236-243, 245-253, 255-273, 275-279, 281, 283-284, 286-298, 300, 302, 304-306, 308-310, 312-341]


2021/12/03 15:11:38 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 15:11:43 eventingSetup.go:719: Waiting for function: TestBucketDeleteWithRebOut status to change to undeployed
2021/12/03 15:11:43 eventingSetup.go:727: Function: TestBucketDeleteWithRebOut status changed to undeployed
2021/12/03 15:11:48 clusterSetup.go:397: Waiting for app: TestBucketDeleteWithRebOut to get un-deployed
2021/12/03 15:11:48 clusterSetup.go:405: App: TestBucketDeleteWithRebOut got un-deployed
2021/12/03 15:11:48 eventingSetup.go:46: No eventing-con process running
2021/12/03 15:11:48 eventingSetup.go:418: Function: TestBucketDeleteWithRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestBucketDeleteWithRebOut deleting in the background"
}
2021/12/03 15:11:50 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/03 15:11:55 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 15:11:57 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/03 15:12:02 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 15:12:02 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv,index,n1ql to the cluster
2021/12/03 15:12:02 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/03 15:12:07 Rebalance progress: 100
--- PASS: TestBucketDeleteWithRebOut (477.95s)
=== RUN   TestN1QLRecursion
2021/12/03 15:12:10 Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/03 15:12:11 Error in adding nodes : , response: ["Prepare join failed. Node is already part of cluster."]
2021/12/03 15:12:11 Removing node(s): [] from the cluster
2021/12/03 15:12:16 Rebalance progress: 100
2021/12/03 15:12:22 Context: Bucket flush, response code: 200 dump: 
2021/12/03 15:12:28 Context: Bucket flush, response code: 200 dump: 
2021/12/03 15:12:34 Context: Bucket flush, response code: 200 dump: 
2021/12/03 15:12:39 Deploying app: TestN1QLRecursion_function1. Item count src bucket: 0 dst bucket: 0
2021/12/03 15:12:40 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLRecursion_function1' in metakv",
  "warnings": null
 }
}
2021/12/03 15:12:40 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLRecursion_function1' in metakv",
  "warnings": null
 }
} err 
2021/12/03 15:12:45 Waiting for app: TestN1QLRecursion_function1 to get deployed
2021/12/03 15:12:50 Waiting for app: TestN1QLRecursion_function1 to get deployed
2021/12/03 15:12:55 Waiting for app: TestN1QLRecursion_function1 to get deployed
2021/12/03 15:13:00 Waiting for app: TestN1QLRecursion_function1 to get deployed
2021/12/03 15:13:00 App: TestN1QLRecursion_function1 got deployed
2021/12/03 15:13:00 TestN1QLRecursion_function1 is deployed
2021/12/03 15:13:00 Deploying app: TestN1QLRecursion_function3. Item count src bucket: 0 dst bucket: 0
2021/12/03 15:13:01 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLRecursion_function3' in metakv",
  "warnings": null
 }
}
2021/12/03 15:13:01 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLRecursion_function3' in metakv",
  "warnings": null
 }
} err 
2021/12/03 15:13:06 Waiting for app: TestN1QLRecursion_function3 to get deployed
2021/12/03 15:13:11 Waiting for app: TestN1QLRecursion_function3 to get deployed
2021/12/03 15:13:16 Waiting for app: TestN1QLRecursion_function3 to get deployed
2021/12/03 15:13:21 Waiting for app: TestN1QLRecursion_function3 to get deployed
2021/12/03 15:13:21 App: TestN1QLRecursion_function3 got deployed
2021/12/03 15:13:21 TestN1QLRecursion_function3 is deployed
2021/12/03 15:13:21 Deploying app: TestN1QLRecursion_function2. Item count src bucket: 0 dst bucket: 0
2021/12/03 15:13:21 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_INTER_BUCKET_RECURSION",
 "code": 51,
 "description": "Inter bucket recursion error, deployment of current handler will cause inter bucket recursion",
 "attributes": null,
 "runtime_info": {
  "code": 51,
  "info": "Inter bucket recursion error; function: TestN1QLRecursion_function2 causes a cycle involving functions: [TestN1QLRecursion_function3 TestN1QLRecursion_function1], hence deployment is disallowed"
 }
}
2021/12/03 15:13:21 response body {
 "name": "ERR_INTER_BUCKET_RECURSION",
 "code": 51,
 "description": "Inter bucket recursion error, deployment of current handler will cause inter bucket recursion",
 "attributes": null,
 "runtime_info": {
  "code": 51,
  "info": "Inter bucket recursion error; function: TestN1QLRecursion_function2 causes a cycle involving functions: [TestN1QLRecursion_function3 TestN1QLRecursion_function1], hence deployment is disallowed"
 }
} err 
2021/12/03 15:13:21 Success: TestN1QLRecursion_function2 is not deployed
2021/12/03 15:13:21 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 15:13:26 Waiting for app: TestN1QLRecursion_function1 to get un-deployed
2021/12/03 15:13:31 Waiting for app: TestN1QLRecursion_function1 to get un-deployed
2021/12/03 15:13:36 Waiting for app: TestN1QLRecursion_function1 to get un-deployed
2021/12/03 15:13:41 Waiting for app: TestN1QLRecursion_function1 to get un-deployed
2021/12/03 15:13:41 App: TestN1QLRecursion_function1 got un-deployed
2021/12/03 15:13:41 No eventing-con process running
2021/12/03 15:13:41 Function: TestN1QLRecursion_function1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLRecursion_function1 deleting in the background"
}
2021/12/03 15:13:43 Context: Bucket flush, response code: 200 dump: 
2021/12/03 15:13:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 15:13:50 Context: Bucket flush, response code: 200 dump: 
2021/12/03 15:13:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 15:13:55 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 15:14:00 Waiting for app: TestN1QLRecursion_function3 to get un-deployed
2021/12/03 15:14:05 Waiting for app: TestN1QLRecursion_function3 to get un-deployed
2021/12/03 15:14:10 Waiting for app: TestN1QLRecursion_function3 to get un-deployed
2021/12/03 15:14:15 Waiting for app: TestN1QLRecursion_function3 to get un-deployed
2021/12/03 15:14:15 App: TestN1QLRecursion_function3 got un-deployed
2021/12/03 15:14:15 No eventing-con process running
2021/12/03 15:14:15 Function: TestN1QLRecursion_function3 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLRecursion_function3 deleting in the background"
}
2021/12/03 15:14:17 Context: Bucket flush, response code: 200 dump: 
2021/12/03 15:14:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 15:14:24 Context: Bucket flush, response code: 200 dump: 
2021/12/03 15:14:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 15:14:31 Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/03 15:14:36 Rebalance progress: 100
--- PASS: TestN1QLRecursion (145.77s)
=== RUN   TestN1QLAllowRecursion
2021/12/03 15:14:36 Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/03 15:14:36 Error in adding nodes : , response: ["Prepare join failed. Node is already part of cluster."]
2021/12/03 15:14:36 Removing node(s): [] from the cluster
2021/12/03 15:14:41 Rebalance progress: 100
2021/12/03 15:14:47 Context: Bucket flush, response code: 200 dump: 
2021/12/03 15:14:53 Context: Bucket flush, response code: 200 dump: 
2021/12/03 15:14:59 Context: Bucket flush, response code: 200 dump: 
2021/12/03 15:15:04 Deploying app: TestN1QLAllowRecursion_function1. Item count src bucket: 0 dst bucket: 0
2021/12/03 15:15:05 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function1' in metakv",
  "warnings": null
 }
}
2021/12/03 15:15:05 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function1' in metakv",
  "warnings": null
 }
} err 
2021/12/03 15:15:10 Waiting for app: TestN1QLAllowRecursion_function1 to get deployed
2021/12/03 15:15:15 Waiting for app: TestN1QLAllowRecursion_function1 to get deployed
2021/12/03 15:15:20 Waiting for app: TestN1QLAllowRecursion_function1 to get deployed
2021/12/03 15:15:25 Waiting for app: TestN1QLAllowRecursion_function1 to get deployed
2021/12/03 15:15:25 App: TestN1QLAllowRecursion_function1 got deployed
2021/12/03 15:15:25 TestN1QLAllowRecursion_function1 is deployed
2021/12/03 15:15:25 Deploying app: TestN1QLAllowRecursion_function3. Item count src bucket: 0 dst bucket: 0
2021/12/03 15:15:26 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function3' in metakv",
  "warnings": null
 }
}
2021/12/03 15:15:26 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function3' in metakv",
  "warnings": null
 }
} err 
2021/12/03 15:15:31 Waiting for app: TestN1QLAllowRecursion_function3 to get deployed
2021/12/03 15:15:36 Waiting for app: TestN1QLAllowRecursion_function3 to get deployed
2021/12/03 15:15:41 Waiting for app: TestN1QLAllowRecursion_function3 to get deployed
2021/12/03 15:15:46 Waiting for app: TestN1QLAllowRecursion_function3 to get deployed
2021/12/03 15:15:46 App: TestN1QLAllowRecursion_function3 got deployed
2021/12/03 15:15:46 TestN1QLAllowRecursion_function3 is deployed
2021/12/03 15:15:46 Deploying app: TestN1QLAllowRecursion_function2. Item count src bucket: 0 dst bucket: 0
2021/12/03 15:15:48 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function2' in metakv",
  "warnings": null
 }
}
2021/12/03 15:15:48 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function2' in metakv",
  "warnings": null
 }
} err 
2021/12/03 15:15:53 Waiting for app: TestN1QLAllowRecursion_function2 to get deployed
2021/12/03 15:15:58 Waiting for app: TestN1QLAllowRecursion_function2 to get deployed
2021/12/03 15:16:03 Waiting for app: TestN1QLAllowRecursion_function2 to get deployed
2021/12/03 15:16:08 Waiting for app: TestN1QLAllowRecursion_function2 to get deployed
2021/12/03 15:16:08 App: TestN1QLAllowRecursion_function2 got deployed
2021/12/03 15:16:08 Success: TestN1QLAllowRecursion_function2 is deployed after allow_interbucket_recursion
2021/12/03 15:16:08 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 15:16:13 Waiting for app: TestN1QLAllowRecursion_function1 to get un-deployed
2021/12/03 15:16:18 Waiting for app: TestN1QLAllowRecursion_function1 to get un-deployed
2021/12/03 15:16:23 Waiting for app: TestN1QLAllowRecursion_function1 to get un-deployed
2021/12/03 15:16:28 Waiting for app: TestN1QLAllowRecursion_function1 to get un-deployed
2021/12/03 15:16:28 App: TestN1QLAllowRecursion_function1 got un-deployed
2021/12/03 15:16:28 No eventing-con process running
2021/12/03 15:16:28 Function: TestN1QLAllowRecursion_function1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLAllowRecursion_function1 deleting in the background"
}
2021/12/03 15:16:30 Context: Bucket flush, response code: 200 dump: 
2021/12/03 15:16:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 15:16:37 Context: Bucket flush, response code: 200 dump: 
2021/12/03 15:16:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 15:16:42 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 15:16:47 Waiting for app: TestN1QLAllowRecursion_function2 to get un-deployed
2021/12/03 15:16:52 Waiting for app: TestN1QLAllowRecursion_function2 to get un-deployed
2021/12/03 15:16:57 Waiting for app: TestN1QLAllowRecursion_function2 to get un-deployed
2021/12/03 15:17:02 Waiting for app: TestN1QLAllowRecursion_function2 to get un-deployed
2021/12/03 15:17:02 App: TestN1QLAllowRecursion_function2 got un-deployed
2021/12/03 15:17:02 No eventing-con process running
2021/12/03 15:17:02 Function: TestN1QLAllowRecursion_function2 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLAllowRecursion_function2 deleting in the background"
}
2021/12/03 15:17:04 Context: Bucket flush, response code: 200 dump: 
2021/12/03 15:17:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 15:17:11 Context: Bucket flush, response code: 200 dump: 
2021/12/03 15:17:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 15:17:16 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 15:17:21 Waiting for app: TestN1QLAllowRecursion_function3 to get un-deployed
2021/12/03 15:17:26 Waiting for app: TestN1QLAllowRecursion_function3 to get un-deployed
2021/12/03 15:17:31 Waiting for app: TestN1QLAllowRecursion_function3 to get un-deployed
2021/12/03 15:17:36 Waiting for app: TestN1QLAllowRecursion_function3 to get un-deployed
2021/12/03 15:17:36 App: TestN1QLAllowRecursion_function3 got un-deployed
2021/12/03 15:17:36 No eventing-con process running
2021/12/03 15:17:36 Function: TestN1QLAllowRecursion_function3 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLAllowRecursion_function3 deleting in the background"
}
2021/12/03 15:17:38 Context: Bucket flush, response code: 200 dump: 
2021/12/03 15:17:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 15:17:45 Context: Bucket flush, response code: 200 dump: 
2021/12/03 15:17:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 15:17:51 Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/03 15:17:56 Rebalance progress: 100
--- PASS: TestN1QLAllowRecursion (200.71s)
=== RUN   TestOnDeleteExpiryBucketOp
2021/12/03 15:17:56 Sleeping for some time
2021/12/03 15:18:06 Waiting for app: TestOnDeleteExpiryBucketOp to get un-deployed
2021/12/03 15:18:06 App: TestOnDeleteExpiryBucketOp got un-deployed
2021/12/03 15:18:06 No eventing-con process running
2021/12/03 15:18:06 Function: TestOnDeleteExpiryBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnDeleteExpiryBucketOp not found"
 }
}
2021/12/03 15:18:08 Context: Bucket flush, response code: 200 dump: 
2021/12/03 15:18:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 15:18:15 Context: Bucket flush, response code: 200 dump: 
2021/12/03 15:18:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 15:18:20 Deploying app: TestOnDeleteExpiryBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/03 15:18:21 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnDeleteExpiryBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/03 15:18:21 Starting bucket ops to source bucket
2021/12/03 15:19:10 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 15:19:15 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 15:19:15 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/03 15:19:15 Starting bucket ops to source bucket
2021/12/03 15:19:16 Starting bucket ops to source bucket
2021/12/03 15:20:22 Waiting for dst bucket item count to get to: 7000 curr count: 5000
2021/12/03 15:20:27 Waiting for dst bucket item count to get to: 7000 curr count: 6875
2021/12/03 15:20:32 Waiting for dst bucket item count to get to: 7000 curr count: 6875
2021/12/03 15:20:37 Waiting for dst bucket item count to get to: 7000 curr count: 6875
2021/12/03 15:20:37 src & dst bucket item count matched up. src bucket count: 7000 dst bucket count: 7000
2021/12/03 15:20:37 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 15:20:37 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 15:20:42 Waiting for app: TestOnDeleteExpiryBucketOp to get un-deployed
2021/12/03 15:20:47 Waiting for app: TestOnDeleteExpiryBucketOp to get un-deployed
2021/12/03 15:20:52 Waiting for app: TestOnDeleteExpiryBucketOp to get un-deployed
2021/12/03 15:20:57 Waiting for app: TestOnDeleteExpiryBucketOp to get un-deployed
2021/12/03 15:20:57 App: TestOnDeleteExpiryBucketOp got un-deployed
2021/12/03 15:20:57 No eventing-con process running
2021/12/03 15:20:57 Function: TestOnDeleteExpiryBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnDeleteExpiryBucketOp deleting in the background"
}
2021/12/03 15:20:59 Context: Bucket flush, response code: 200 dump: 
2021/12/03 15:21:09 Waiting for dst bucket item count to get to: 0 curr count: 499
2021/12/03 15:21:14 Waiting for dst bucket item count to get to: 0 curr count: 499
2021/12/03 15:21:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 15:21:16 Context: Bucket flush, response code: 200 dump: 
2021/12/03 15:21:26 Waiting for dst bucket item count to get to: 0 curr count: 7000
2021/12/03 15:21:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnDeleteExpiryBucketOp (209.84s)
=== RUN   TestTimerOverWriteSameReference
2021/12/03 15:21:31 Waiting for app: TestTimerOverWriteSameReference to get un-deployed
2021/12/03 15:21:31 App: TestTimerOverWriteSameReference got un-deployed
2021/12/03 15:21:31 No eventing-con process running
2021/12/03 15:21:31 Function: TestTimerOverWriteSameReference delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestTimerOverWriteSameReference not found"
 }
}
2021/12/03 15:21:33 Context: Bucket flush, response code: 200 dump: 
2021/12/03 15:21:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 15:21:40 Context: Bucket flush, response code: 200 dump: 
2021/12/03 15:21:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 15:21:45 Deploying app: TestTimerOverWriteSameReference. Item count src bucket: 0 dst bucket: 0
2021/12/03 15:21:46 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestTimerOverWriteSameReference' in metakv",
  "warnings": null
 }
}
2021/12/03 15:21:51 Waiting for app: TestTimerOverWriteSameReference to get deployed
2021/12/03 15:21:56 Waiting for app: TestTimerOverWriteSameReference to get deployed
2021/12/03 15:22:01 Waiting for app: TestTimerOverWriteSameReference to get deployed
2021/12/03 15:22:06 Waiting for app: TestTimerOverWriteSameReference to get deployed
2021/12/03 15:22:06 App: TestTimerOverWriteSameReference got deployed
2021/12/03 15:22:16 Starting bucket ops to source bucket
2021/12/03 15:23:17 src & dst bucket item count matched up. src bucket count: 1 dst bucket count: 1
2021/12/03 15:23:27 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 15:23:28 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 15:23:33 Waiting for app: TestTimerOverWriteSameReference to get un-deployed
2021/12/03 15:23:38 Waiting for app: TestTimerOverWriteSameReference to get un-deployed
2021/12/03 15:23:43 Waiting for app: TestTimerOverWriteSameReference to get un-deployed
2021/12/03 15:23:48 Waiting for app: TestTimerOverWriteSameReference to get un-deployed
2021/12/03 15:23:48 App: TestTimerOverWriteSameReference got un-deployed
2021/12/03 15:23:48 No eventing-con process running
2021/12/03 15:23:48 Function: TestTimerOverWriteSameReference delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestTimerOverWriteSameReference deleting in the background"
}
2021/12/03 15:23:49 Context: Bucket flush, response code: 200 dump: 
2021/12/03 15:23:59 Waiting for dst bucket item count to get to: 0 curr count: 500
2021/12/03 15:24:04 Waiting for dst bucket item count to get to: 0 curr count: 500
2021/12/03 15:24:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 15:24:06 Context: Bucket flush, response code: 200 dump: 
2021/12/03 15:24:16 Waiting for dst bucket item count to get to: 0 curr count: 1
2021/12/03 15:24:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestTimerOverWriteSameReference (170.08s)
=== RUN   TestCancelTimerBucketop
2021/12/03 15:24:21 Waiting for app: TestCancelTimerBucketop to get un-deployed
2021/12/03 15:24:21 App: TestCancelTimerBucketop got un-deployed
2021/12/03 15:24:21 No eventing-con process running
2021/12/03 15:24:21 Function: TestCancelTimerBucketop delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestCancelTimerBucketop not found"
 }
}
2021/12/03 15:24:23 Context: Bucket flush, response code: 200 dump: 
2021/12/03 15:24:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 15:24:30 Context: Bucket flush, response code: 200 dump: 
2021/12/03 15:24:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 15:24:35 Deploying app: TestCancelTimerBucketop. Item count src bucket: 0 dst bucket: 0
2021/12/03 15:24:36 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCancelTimerBucketop' in metakv",
  "warnings": null
 }
}
2021/12/03 15:24:41 Waiting for app: TestCancelTimerBucketop to get deployed
2021/12/03 15:24:46 Waiting for app: TestCancelTimerBucketop to get deployed
2021/12/03 15:24:51 Waiting for app: TestCancelTimerBucketop to get deployed
2021/12/03 15:24:56 Waiting for app: TestCancelTimerBucketop to get deployed
2021/12/03 15:24:56 App: TestCancelTimerBucketop got deployed
2021/12/03 15:24:56 Starting bucket ops to source bucket
2021/12/03 15:25:03 Starting bucket ops to source bucket
2021/12/03 15:26:06 src & dst bucket item count matched up. src bucket count: 1000 dst bucket count: 1000
2021/12/03 15:26:06 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 15:26:06 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 15:26:11 Waiting for app: TestCancelTimerBucketop to get un-deployed
2021/12/03 15:26:16 Waiting for app: TestCancelTimerBucketop to get un-deployed
2021/12/03 15:26:21 Waiting for app: TestCancelTimerBucketop to get un-deployed
2021/12/03 15:26:26 Waiting for app: TestCancelTimerBucketop to get un-deployed
2021/12/03 15:26:26 App: TestCancelTimerBucketop got un-deployed
2021/12/03 15:26:26 No eventing-con process running
2021/12/03 15:26:26 Function: TestCancelTimerBucketop delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCancelTimerBucketop deleting in the background"
}
2021/12/03 15:26:28 Context: Bucket flush, response code: 200 dump: 
2021/12/03 15:26:38 Waiting for dst bucket item count to get to: 0 curr count: 509
2021/12/03 15:26:43 Waiting for dst bucket item count to get to: 0 curr count: 9
2021/12/03 15:26:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 15:26:45 Context: Bucket flush, response code: 200 dump: 
2021/12/03 15:26:55 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/03 15:26:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCancelTimerBucketop (158.24s)
=== RUN   TestJSExpiryDate
2021/12/03 15:27:00 Starting bucket ops to source bucket
2021/12/03 15:27:13 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/03 15:27:13 Deploying app: TestJSExpiryDate. Item count src bucket: 5000 dst bucket: 0
2021/12/03 15:27:14 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestJSExpiryDate' in metakv",
  "warnings": null
 }
}
2021/12/03 15:27:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 15:27:44 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 15:27:44 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 15:27:49 Waiting for app: TestJSExpiryDate to get un-deployed
2021/12/03 15:27:54 Waiting for app: TestJSExpiryDate to get un-deployed
2021/12/03 15:27:59 Waiting for app: TestJSExpiryDate to get un-deployed
2021/12/03 15:28:04 Waiting for app: TestJSExpiryDate to get un-deployed
2021/12/03 15:28:04 App: TestJSExpiryDate got un-deployed
2021/12/03 15:28:04 No eventing-con process running
2021/12/03 15:28:04 Function: TestJSExpiryDate delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestJSExpiryDate deleting in the background"
}
2021/12/03 15:28:06 Context: Bucket flush, response code: 200 dump: 
2021/12/03 15:28:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 15:28:13 Context: Bucket flush, response code: 200 dump: 
2021/12/03 15:28:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestJSExpiryDate (83.89s)
=== RUN   TestBinaryDoc
2021/12/03 15:28:23 Starting bucket ops to source bucket
2021/12/03 15:28:24 Deploying app: TestBinaryDoc. Item count src bucket: 0 dst bucket: 0
2021/12/03 15:28:25 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestBinaryDoc' in metakv",
  "warnings": null
 }
}
2021/12/03 15:28:25 Starting bucket ops to source bucket
2021/12/03 15:28:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 15:28:25 Deleting source bucket docs now
2021/12/03 15:28:25 Starting bucket ops to source bucket
2021/12/03 15:29:02 Waiting for dst bucket item count to get to: 32 curr count: 23
2021/12/03 15:29:07 Waiting for dst bucket item count to get to: 32 curr count: 28
2021/12/03 15:29:07 src & dst bucket item count matched up. src bucket count: 32 dst bucket count: 32
2021/12/03 15:29:07 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 15:29:07 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 15:29:12 Waiting for app: TestBinaryDoc to get un-deployed
2021/12/03 15:29:17 Waiting for app: TestBinaryDoc to get un-deployed
2021/12/03 15:29:22 Waiting for app: TestBinaryDoc to get un-deployed
2021/12/03 15:29:27 Waiting for app: TestBinaryDoc to get un-deployed
2021/12/03 15:29:27 App: TestBinaryDoc got un-deployed
2021/12/03 15:29:27 No eventing-con process running
2021/12/03 15:29:27 Function: TestBinaryDoc delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestBinaryDoc deleting in the background"
}
2021/12/03 15:29:29 Context: Bucket flush, response code: 200 dump: 
2021/12/03 15:29:39 Waiting for dst bucket item count to get to: 0 curr count: 32
2021/12/03 15:29:44 Waiting for dst bucket item count to get to: 0 curr count: 14
2021/12/03 15:29:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 15:29:46 Context: Bucket flush, response code: 200 dump: 
2021/12/03 15:29:56 Waiting for dst bucket item count to get to: 0 curr count: 32
2021/12/03 15:29:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestBinaryDoc (97.36s)
=== RUN   TestConstantBindings
2021/12/03 15:29:56 Deploying app: TestConstantBindings. Item count src bucket: 0 dst bucket: 0
2021/12/03 15:29:57 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestConstantBindings' in metakv",
  "warnings": null
 }
}
2021/12/03 15:29:57 Starting bucket ops to source bucket
2021/12/03 15:30:03 Waiting for dst bucket item count to get to: 6000 curr count: 0
2021/12/03 15:30:08 Waiting for dst bucket item count to get to: 6000 curr count: 0
2021/12/03 15:30:13 Waiting for dst bucket item count to get to: 6000 curr count: 0
2021/12/03 15:30:18 Waiting for dst bucket item count to get to: 6000 curr count: 720
2021/12/03 15:30:23 Waiting for dst bucket item count to get to: 6000 curr count: 720
2021/12/03 15:30:23 src & dst bucket item count matched up. src bucket count: 6000 dst bucket count: 6000
2021/12/03 15:30:23 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 15:30:23 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 15:30:28 Waiting for app: TestConstantBindings to get un-deployed
2021/12/03 15:30:33 Waiting for app: TestConstantBindings to get un-deployed
2021/12/03 15:30:38 Waiting for app: TestConstantBindings to get un-deployed
2021/12/03 15:30:43 Waiting for app: TestConstantBindings to get un-deployed
2021/12/03 15:30:43 App: TestConstantBindings got un-deployed
2021/12/03 15:30:43 No eventing-con process running
2021/12/03 15:30:43 Function: TestConstantBindings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestConstantBindings deleting in the background"
}
2021/12/03 15:30:45 Context: Bucket flush, response code: 200 dump: 
2021/12/03 15:30:55 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/03 15:30:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 15:30:57 Context: Bucket flush, response code: 200 dump: 
2021/12/03 15:31:07 Waiting for dst bucket item count to get to: 0 curr count: 6000
2021/12/03 15:31:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestConstantBindings (70.70s)
=== RUN   TestCodeInsight
2021/12/03 15:31:12 Waiting for app: TestCodeInsight to get un-deployed
2021/12/03 15:31:12 App: TestCodeInsight got un-deployed
2021/12/03 15:31:12 No eventing-con process running
2021/12/03 15:31:12 Function: TestCodeInsight delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestCodeInsight not found"
 }
}
2021/12/03 15:31:13 Context: Bucket flush, response code: 200 dump: 
2021/12/03 15:31:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 15:31:20 Context: Bucket flush, response code: 200 dump: 
2021/12/03 15:31:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 15:31:25 Starting bucket ops to source bucket
2021/12/03 15:31:26 Deploying app: TestCodeInsight. Item count src bucket: 0 dst bucket: 0
2021/12/03 15:31:27 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCodeInsight' in metakv",
  "warnings": null
 }
}
2021/12/03 15:31:32 Waiting for app: TestCodeInsight to get deployed
2021/12/03 15:31:37 Waiting for app: TestCodeInsight to get deployed
2021/12/03 15:31:42 Waiting for app: TestCodeInsight to get deployed
2021/12/03 15:31:47 Waiting for app: TestCodeInsight to get deployed
2021/12/03 15:31:47 App: TestCodeInsight got deployed
2021/12/03 15:31:52 Waiting for dst bucket item count to get to: 1000 curr count: 229
2021/12/03 15:31:52 src & dst bucket item count matched up. src bucket count: 1000 dst bucket count: 1000
2021/12/03 15:31:52 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 15:31:57 Waiting for app: TestCodeInsight to get un-deployed
2021/12/03 15:32:02 Waiting for app: TestCodeInsight to get un-deployed
2021/12/03 15:32:07 Waiting for app: TestCodeInsight to get un-deployed
2021/12/03 15:32:12 Waiting for app: TestCodeInsight to get un-deployed
2021/12/03 15:32:12 App: TestCodeInsight got un-deployed
2021/12/03 15:32:12 No eventing-con process running
2021/12/03 15:32:12 Function: TestCodeInsight delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCodeInsight deleting in the background"
}
2021/12/03 15:32:14 Context: Bucket flush, response code: 200 dump: 
2021/12/03 15:32:24 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/03 15:32:29 Waiting for dst bucket item count to get to: 0 curr count: 500
2021/12/03 15:32:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 15:32:31 Context: Bucket flush, response code: 200 dump: 
2021/12/03 15:32:41 Waiting for dst bucket item count to get to: 0 curr count: 969
2021/12/03 15:32:46 Waiting for dst bucket item count to get to: 0 curr count: 969
2021/12/03 15:32:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCodeInsight (99.46s)
=== RUN   TestParserTransform
--- PASS: TestParserTransform (0.07s)
=== RUN   TestParserDetect
--- PASS: TestParserDetect (0.00s)
PASS
ok  	github.com/couchbase/eventing/tests/functional_tests	12932.501s

Functional tests took 3 hours, 35 minutes and 38 seconds.

N1ql tests

2021/12/03 15:32:54 Started n1ql
2021/12/03 15:33:04 Adding node: https://127.0.0.1:19001 with role: kv,index,n1ql to the cluster
2021/12/03 15:33:04 Error in adding nodes : , response: ["Prepare join failed. Node is already part of cluster."]
2021/12/03 15:33:04 Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/03 15:33:04 Removing node(s): [] from the cluster
2021/12/03 15:33:09 Rebalance progress: 100
=== RUN   TestRecursiveMutationN1QL
2021/12/03 15:33:14 Waiting for app: TestRecursiveMutationN1QL to get un-deployed
2021/12/03 15:33:15 App: TestRecursiveMutationN1QL got un-deployed
2021/12/03 15:33:15 No eventing-con process running
2021/12/03 15:33:15 Function: TestRecursiveMutationN1QL delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestRecursiveMutationN1QL not found"
 }
}
2021/12/03 15:33:16 Context: Bucket flush, response code: 200 dump: 
2021/12/03 15:33:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 15:33:23 Context: Bucket flush, response code: 200 dump: 
2021/12/03 15:33:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 15:33:28 Deploying app: TestRecursiveMutationN1QL. Item count src bucket: 0 dst bucket: 0
2021/12/03 15:33:28 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_INTER_BUCKET_RECURSION",
 "code": 51,
 "description": "Inter bucket recursion error, deployment of current handler will cause inter bucket recursion",
 "attributes": null,
 "runtime_info": {
  "code": 51,
  "info": "Function: TestRecursiveMutationN1QL N1QL DML to source keyspace default:default"
 }
}
--- PASS: TestRecursiveMutationN1QL (18.64s)
=== RUN   TestFlexReset1
2021/12/03 15:33:33 Waiting for app: TestFlexReset1 to get un-deployed
2021/12/03 15:33:33 App: TestFlexReset1 got un-deployed
2021/12/03 15:33:33 No eventing-con process running
2021/12/03 15:33:33 Function: TestFlexReset1 delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestFlexReset1 not found"
 }
}
2021/12/03 15:33:35 Context: Bucket flush, response code: 200 dump: 
2021/12/03 15:33:40 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 15:33:42 Context: Bucket flush, response code: 200 dump: 
2021/12/03 15:33:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 15:33:47 Starting bucket ops to source bucket
2021/12/03 15:33:48 Deploying app: TestFlexReset1. Item count src bucket: 0 dst bucket: 0
2021/12/03 15:33:49 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestFlexReset1' in metakv",
  "warnings": null
 }
}
2021/12/03 15:33:54 Waiting for app: TestFlexReset1 to get deployed
2021/12/03 15:33:59 Waiting for app: TestFlexReset1 to get deployed
2021/12/03 15:34:04 Waiting for app: TestFlexReset1 to get deployed
2021/12/03 15:34:09 Waiting for app: TestFlexReset1 to get deployed
2021/12/03 15:34:09 App: TestFlexReset1 got deployed
2021/12/03 15:34:14 Waiting for dst bucket item count to get to: 1000 curr count: 0
2021/12/03 15:34:14 src & dst bucket item count matched up. src bucket count: 1000 dst bucket count: 1000
2021/12/03 15:34:14 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 15:34:14 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 15:34:19 Waiting for app: TestFlexReset1 to get un-deployed
2021/12/03 15:34:24 Waiting for app: TestFlexReset1 to get un-deployed
2021/12/03 15:34:29 Waiting for app: TestFlexReset1 to get un-deployed
2021/12/03 15:34:34 Waiting for app: TestFlexReset1 to get un-deployed
2021/12/03 15:34:34 App: TestFlexReset1 got un-deployed
2021/12/03 15:34:34 No eventing-con process running
2021/12/03 15:34:34 Function: TestFlexReset1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestFlexReset1 deleting in the background"
}
2021/12/03 15:34:36 Context: Bucket flush, response code: 200 dump: 
2021/12/03 15:34:46 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/03 15:34:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 15:34:48 Context: Bucket flush, response code: 200 dump: 
2021/12/03 15:34:58 Waiting for dst bucket item count to get to: 0 curr count: 500
2021/12/03 15:34:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestFlexReset1 (89.43s)
=== RUN   TestFlexReset2
2021/12/03 15:35:03 Waiting for app: TestFlexReset2 to get un-deployed
2021/12/03 15:35:03 App: TestFlexReset2 got un-deployed
2021/12/03 15:35:03 No eventing-con process running
2021/12/03 15:35:03 Function: TestFlexReset2 delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestFlexReset2 not found"
 }
}
2021/12/03 15:35:04 Context: Bucket flush, response code: 200 dump: 
2021/12/03 15:35:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 15:35:11 Context: Bucket flush, response code: 200 dump: 
2021/12/03 15:35:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 15:35:16 Starting bucket ops to source bucket
2021/12/03 15:35:17 Deploying app: TestFlexReset2. Item count src bucket: 0 dst bucket: 0
2021/12/03 15:35:18 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestFlexReset2' in metakv",
  "warnings": null
 }
}
2021/12/03 15:35:23 Waiting for app: TestFlexReset2 to get deployed
2021/12/03 15:35:28 Waiting for app: TestFlexReset2 to get deployed
2021/12/03 15:35:33 Waiting for app: TestFlexReset2 to get deployed
2021/12/03 15:35:38 Waiting for app: TestFlexReset2 to get deployed
2021/12/03 15:35:38 App: TestFlexReset2 got deployed
2021/12/03 15:35:43 Waiting for dst bucket item count to get to: 1000 curr count: 2
2021/12/03 15:35:43 src & dst bucket item count matched up. src bucket count: 1000 dst bucket count: 1000
2021/12/03 15:35:43 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 15:35:43 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 15:35:48 Waiting for app: TestFlexReset2 to get un-deployed
2021/12/03 15:35:53 Waiting for app: TestFlexReset2 to get un-deployed
2021/12/03 15:35:58 Waiting for app: TestFlexReset2 to get un-deployed
2021/12/03 15:36:03 Waiting for app: TestFlexReset2 to get un-deployed
2021/12/03 15:36:03 App: TestFlexReset2 got un-deployed
2021/12/03 15:36:03 No eventing-con process running
2021/12/03 15:36:03 Function: TestFlexReset2 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestFlexReset2 deleting in the background"
}
2021/12/03 15:36:05 Context: Bucket flush, response code: 200 dump: 
2021/12/03 15:36:15 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/03 15:36:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 15:36:17 Context: Bucket flush, response code: 200 dump: 
2021/12/03 15:36:27 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/03 15:36:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestFlexReset2 (89.26s)
=== RUN   TestN1QLLabelledBreak
2021/12/03 15:36:32 Waiting for app: TestN1QLLabelledBreak to get un-deployed
2021/12/03 15:36:32 App: TestN1QLLabelledBreak got un-deployed
2021/12/03 15:36:32 No eventing-con process running
2021/12/03 15:36:32 Function: TestN1QLLabelledBreak delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLLabelledBreak not found"
 }
}
2021/12/03 15:36:34 Context: Bucket flush, response code: 200 dump: 
2021/12/03 15:36:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 15:36:41 Context: Bucket flush, response code: 200 dump: 
2021/12/03 15:36:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 15:36:46 Deploying app: TestN1QLLabelledBreak. Item count src bucket: 0 dst bucket: 0
2021/12/03 15:36:47 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLLabelledBreak' in metakv",
  "warnings": null
 }
}
2021/12/03 15:36:52 Waiting for app: TestN1QLLabelledBreak to get deployed
2021/12/03 15:36:57 Waiting for app: TestN1QLLabelledBreak to get deployed
2021/12/03 15:37:02 Waiting for app: TestN1QLLabelledBreak to get deployed
2021/12/03 15:37:07 Waiting for app: TestN1QLLabelledBreak to get deployed
2021/12/03 15:37:07 App: TestN1QLLabelledBreak got deployed
2021/12/03 15:37:07 Starting bucket ops to source bucket
2021/12/03 15:37:16 Waiting for dst bucket item count to get to: 10000 curr count: 0
2021/12/03 15:37:21 Waiting for dst bucket item count to get to: 10000 curr count: 187
2021/12/03 15:37:26 Waiting for dst bucket item count to get to: 10000 curr count: 187
2021/12/03 15:37:31 Waiting for dst bucket item count to get to: 10000 curr count: 3520
2021/12/03 15:37:36 Waiting for dst bucket item count to get to: 10000 curr count: 3520
2021/12/03 15:37:41 Waiting for dst bucket item count to get to: 10000 curr count: 7576
2021/12/03 15:37:46 Waiting for dst bucket item count to get to: 10000 curr count: 7576
2021/12/03 15:37:46 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/03 15:37:46 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 15:37:46 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 15:37:51 Waiting for app: TestN1QLLabelledBreak to get un-deployed
2021/12/03 15:37:56 Waiting for app: TestN1QLLabelledBreak to get un-deployed
2021/12/03 15:38:01 Waiting for app: TestN1QLLabelledBreak to get un-deployed
2021/12/03 15:38:06 Waiting for app: TestN1QLLabelledBreak to get un-deployed
2021/12/03 15:38:06 App: TestN1QLLabelledBreak got un-deployed
2021/12/03 15:38:06 No eventing-con process running
2021/12/03 15:38:06 Function: TestN1QLLabelledBreak delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLLabelledBreak deleting in the background"
}
2021/12/03 15:38:08 Context: Bucket flush, response code: 200 dump: 
2021/12/03 15:38:18 Waiting for dst bucket item count to get to: 0 curr count: 3436
2021/12/03 15:38:23 Waiting for dst bucket item count to get to: 0 curr count: 936
2021/12/03 15:38:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 15:38:25 Context: Bucket flush, response code: 200 dump: 
2021/12/03 15:38:35 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/03 15:38:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLLabelledBreak (128.37s)
=== RUN   TestN1QLUnlabelledBreak
2021/12/03 15:38:40 Waiting for app: TestN1QLUnlabelledBreak to get un-deployed
2021/12/03 15:38:40 App: TestN1QLUnlabelledBreak got un-deployed
2021/12/03 15:38:40 No eventing-con process running
2021/12/03 15:38:40 Function: TestN1QLUnlabelledBreak delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLUnlabelledBreak not found"
 }
}
2021/12/03 15:38:42 Context: Bucket flush, response code: 200 dump: 
2021/12/03 15:38:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 15:38:49 Context: Bucket flush, response code: 200 dump: 
2021/12/03 15:38:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 15:38:54 Deploying app: TestN1QLUnlabelledBreak. Item count src bucket: 0 dst bucket: 0
2021/12/03 15:38:55 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLUnlabelledBreak' in metakv",
  "warnings": null
 }
}
2021/12/03 15:39:00 Waiting for app: TestN1QLUnlabelledBreak to get deployed
2021/12/03 15:39:05 Waiting for app: TestN1QLUnlabelledBreak to get deployed
2021/12/03 15:39:10 Waiting for app: TestN1QLUnlabelledBreak to get deployed
2021/12/03 15:39:15 Waiting for app: TestN1QLUnlabelledBreak to get deployed
2021/12/03 15:39:15 App: TestN1QLUnlabelledBreak got deployed
2021/12/03 15:39:15 Starting bucket ops to source bucket
2021/12/03 15:39:25 Waiting for dst bucket item count to get to: 10000 curr count: 221
2021/12/03 15:39:30 Waiting for dst bucket item count to get to: 10000 curr count: 221
2021/12/03 15:39:35 Waiting for dst bucket item count to get to: 10000 curr count: 684
2021/12/03 15:39:40 Waiting for dst bucket item count to get to: 10000 curr count: 5974
2021/12/03 15:39:45 Waiting for dst bucket item count to get to: 10000 curr count: 5974
2021/12/03 15:39:45 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/03 15:39:45 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 15:39:45 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 15:39:50 Waiting for app: TestN1QLUnlabelledBreak to get un-deployed
2021/12/03 15:39:55 Waiting for app: TestN1QLUnlabelledBreak to get un-deployed
2021/12/03 15:40:00 Waiting for app: TestN1QLUnlabelledBreak to get un-deployed
2021/12/03 15:40:05 Waiting for app: TestN1QLUnlabelledBreak to get un-deployed
2021/12/03 15:40:05 App: TestN1QLUnlabelledBreak got un-deployed
2021/12/03 15:40:05 No eventing-con process running
2021/12/03 15:40:05 Function: TestN1QLUnlabelledBreak delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLUnlabelledBreak deleting in the background"
}
2021/12/03 15:40:07 Context: Bucket flush, response code: 200 dump: 
2021/12/03 15:40:17 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/03 15:40:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 15:40:19 Context: Bucket flush, response code: 200 dump: 
2021/12/03 15:40:29 Waiting for dst bucket item count to get to: 0 curr count: 8788
2021/12/03 15:40:34 Waiting for dst bucket item count to get to: 0 curr count: 3788
2021/12/03 15:40:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLUnlabelledBreak (118.35s)
=== RUN   TestN1QLThrowStatement
2021/12/03 15:40:39 Waiting for app: TestN1QLThrowStatement to get un-deployed
2021/12/03 15:40:39 App: TestN1QLThrowStatement got un-deployed
2021/12/03 15:40:39 No eventing-con process running
2021/12/03 15:40:39 Function: TestN1QLThrowStatement delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLThrowStatement not found"
 }
}
2021/12/03 15:40:40 Context: Bucket flush, response code: 200 dump: 
2021/12/03 15:40:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 15:40:47 Context: Bucket flush, response code: 200 dump: 
2021/12/03 15:40:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 15:40:52 Deploying app: TestN1QLThrowStatement. Item count src bucket: 0 dst bucket: 0
2021/12/03 15:40:53 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLThrowStatement' in metakv",
  "warnings": null
 }
}
2021/12/03 15:40:58 Waiting for app: TestN1QLThrowStatement to get deployed
2021/12/03 15:41:03 Waiting for app: TestN1QLThrowStatement to get deployed
2021/12/03 15:41:08 Waiting for app: TestN1QLThrowStatement to get deployed
2021/12/03 15:41:13 Waiting for app: TestN1QLThrowStatement to get deployed
2021/12/03 15:41:13 App: TestN1QLThrowStatement got deployed
2021/12/03 15:41:13 Starting bucket ops to source bucket
2021/12/03 15:41:23 Waiting for dst bucket item count to get to: 10000 curr count: 0
2021/12/03 15:41:28 Waiting for dst bucket item count to get to: 10000 curr count: 270
2021/12/03 15:41:33 Waiting for dst bucket item count to get to: 10000 curr count: 845
2021/12/03 15:41:38 Waiting for dst bucket item count to get to: 10000 curr count: 2783
2021/12/03 15:41:43 Waiting for dst bucket item count to get to: 10000 curr count: 4858
2021/12/03 15:41:48 Waiting for dst bucket item count to get to: 10000 curr count: 6888
2021/12/03 15:41:53 Waiting for dst bucket item count to get to: 10000 curr count: 8874
2021/12/03 15:41:58 Waiting for dst bucket item count to get to: 10000 curr count: 9636
2021/12/03 15:41:58 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/03 15:41:58 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 15:41:58 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 15:42:03 Waiting for app: TestN1QLThrowStatement to get un-deployed
2021/12/03 15:42:08 Waiting for app: TestN1QLThrowStatement to get un-deployed
2021/12/03 15:42:13 Waiting for app: TestN1QLThrowStatement to get un-deployed
2021/12/03 15:42:18 Waiting for app: TestN1QLThrowStatement to get un-deployed
2021/12/03 15:42:18 App: TestN1QLThrowStatement got un-deployed
2021/12/03 15:42:18 No eventing-con process running
2021/12/03 15:42:18 Function: TestN1QLThrowStatement delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLThrowStatement deleting in the background"
}
2021/12/03 15:42:20 Context: Bucket flush, response code: 200 dump: 
2021/12/03 15:42:30 Waiting for dst bucket item count to get to: 0 curr count: 3038
2021/12/03 15:42:35 Waiting for dst bucket item count to get to: 0 curr count: 3038
2021/12/03 15:42:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 15:42:37 Context: Bucket flush, response code: 200 dump: 
2021/12/03 15:42:47 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/03 15:42:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLThrowStatement (133.28s)
=== RUN   TestN1QLNestedForLoop
2021/12/03 15:42:52 Waiting for app: TestN1QLNestedForLoop to get un-deployed
2021/12/03 15:42:52 App: TestN1QLNestedForLoop got un-deployed
2021/12/03 15:42:52 No eventing-con process running
2021/12/03 15:42:52 Function: TestN1QLNestedForLoop delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLNestedForLoop not found"
 }
}
2021/12/03 15:42:54 Context: Bucket flush, response code: 200 dump: 
2021/12/03 15:42:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 15:43:00 Context: Bucket flush, response code: 200 dump: 
2021/12/03 15:43:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 15:43:05 Deploying app: TestN1QLNestedForLoop. Item count src bucket: 0 dst bucket: 0
2021/12/03 15:43:07 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLNestedForLoop' in metakv",
  "warnings": null
 }
}
2021/12/03 15:43:12 Waiting for app: TestN1QLNestedForLoop to get deployed
2021/12/03 15:43:17 Waiting for app: TestN1QLNestedForLoop to get deployed
2021/12/03 15:43:22 Waiting for app: TestN1QLNestedForLoop to get deployed
2021/12/03 15:43:27 Waiting for app: TestN1QLNestedForLoop to get deployed
2021/12/03 15:43:27 App: TestN1QLNestedForLoop got deployed
2021/12/03 15:43:27 Starting bucket ops to source bucket
2021/12/03 15:43:36 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 15:43:41 Waiting for dst bucket item count to get to: 5000 curr count: 99
2021/12/03 15:43:46 Waiting for dst bucket item count to get to: 5000 curr count: 99
2021/12/03 15:43:51 Waiting for dst bucket item count to get to: 5000 curr count: 1964
2021/12/03 15:43:56 Waiting for dst bucket item count to get to: 5000 curr count: 1964
2021/12/03 15:44:01 Waiting for dst bucket item count to get to: 5000 curr count: 4283
2021/12/03 15:44:06 Waiting for dst bucket item count to get to: 5000 curr count: 4283
2021/12/03 15:44:06 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/03 15:44:06 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 15:44:06 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 15:44:11 Waiting for app: TestN1QLNestedForLoop to get un-deployed
2021/12/03 15:44:16 Waiting for app: TestN1QLNestedForLoop to get un-deployed
2021/12/03 15:44:21 Waiting for app: TestN1QLNestedForLoop to get un-deployed
2021/12/03 15:44:26 Waiting for app: TestN1QLNestedForLoop to get un-deployed
2021/12/03 15:44:26 App: TestN1QLNestedForLoop got un-deployed
2021/12/03 15:44:26 No eventing-con process running
2021/12/03 15:44:26 Function: TestN1QLNestedForLoop delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLNestedForLoop deleting in the background"
}
2021/12/03 15:44:28 Context: Bucket flush, response code: 200 dump: 
2021/12/03 15:44:38 Waiting for dst bucket item count to get to: 0 curr count: 3586
2021/12/03 15:44:43 Waiting for dst bucket item count to get to: 0 curr count: 1086
2021/12/03 15:44:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 15:44:45 Context: Bucket flush, response code: 200 dump: 
2021/12/03 15:44:55 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/03 15:44:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLNestedForLoop (128.49s)
=== RUN   TestN1QLAutoClose
2021/12/03 15:45:00 Waiting for app: TestN1QLAutoClose to get un-deployed
2021/12/03 15:45:00 App: TestN1QLAutoClose got un-deployed
2021/12/03 15:45:00 No eventing-con process running
2021/12/03 15:45:00 Function: TestN1QLAutoClose delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLAutoClose not found"
 }
}
2021/12/03 15:45:02 Context: Bucket flush, response code: 200 dump: 
2021/12/03 15:45:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 15:45:09 Context: Bucket flush, response code: 200 dump: 
2021/12/03 15:45:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 15:45:14 Deploying app: TestN1QLAutoClose. Item count src bucket: 0 dst bucket: 0
2021/12/03 15:45:15 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAutoClose' in metakv",
  "warnings": null
 }
}
2021/12/03 15:45:20 Waiting for app: TestN1QLAutoClose to get deployed
2021/12/03 15:45:25 Waiting for app: TestN1QLAutoClose to get deployed
2021/12/03 15:45:30 Waiting for app: TestN1QLAutoClose to get deployed
2021/12/03 15:45:35 Waiting for app: TestN1QLAutoClose to get deployed
2021/12/03 15:45:35 App: TestN1QLAutoClose got deployed
2021/12/03 15:45:35 Starting bucket ops to source bucket
2021/12/03 15:45:41 Waiting for dst bucket item count to get to: 500 curr count: 0
2021/12/03 15:45:46 Waiting for dst bucket item count to get to: 500 curr count: 4
2021/12/03 15:45:51 Waiting for dst bucket item count to get to: 500 curr count: 4
2021/12/03 15:45:56 Waiting for dst bucket item count to get to: 500 curr count: 28
2021/12/03 15:46:01 Waiting for dst bucket item count to get to: 500 curr count: 28
2021/12/03 15:46:06 Waiting for dst bucket item count to get to: 500 curr count: 48
2021/12/03 15:46:11 Waiting for dst bucket item count to get to: 500 curr count: 48
2021/12/03 15:46:16 Waiting for dst bucket item count to get to: 500 curr count: 70
2021/12/03 15:46:21 Waiting for dst bucket item count to get to: 500 curr count: 70
2021/12/03 15:46:26 Waiting for dst bucket item count to get to: 500 curr count: 93
2021/12/03 15:46:31 Waiting for dst bucket item count to get to: 500 curr count: 93
2021/12/03 15:46:36 Waiting for dst bucket item count to get to: 500 curr count: 113
2021/12/03 15:46:41 Waiting for dst bucket item count to get to: 500 curr count: 113
2021/12/03 15:46:46 Waiting for dst bucket item count to get to: 500 curr count: 113
2021/12/03 15:46:51 Waiting for dst bucket item count to get to: 500 curr count: 133
2021/12/03 15:46:56 Waiting for dst bucket item count to get to: 500 curr count: 133
2021/12/03 15:47:01 Waiting for dst bucket item count to get to: 500 curr count: 154
2021/12/03 15:47:06 Waiting for dst bucket item count to get to: 500 curr count: 154
2021/12/03 15:47:11 Waiting for dst bucket item count to get to: 500 curr count: 177
2021/12/03 15:47:16 Waiting for dst bucket item count to get to: 500 curr count: 177
2021/12/03 15:47:21 Waiting for dst bucket item count to get to: 500 curr count: 198
2021/12/03 15:47:26 Waiting for dst bucket item count to get to: 500 curr count: 198
2021/12/03 15:47:31 Waiting for dst bucket item count to get to: 500 curr count: 221
2021/12/03 15:47:36 Waiting for dst bucket item count to get to: 500 curr count: 221
2021/12/03 15:47:41 Waiting for dst bucket item count to get to: 500 curr count: 242
2021/12/03 15:47:46 Waiting for dst bucket item count to get to: 500 curr count: 242
2021/12/03 15:47:51 Waiting for dst bucket item count to get to: 500 curr count: 263
2021/12/03 15:47:56 Waiting for dst bucket item count to get to: 500 curr count: 263
2021/12/03 15:48:01 Waiting for dst bucket item count to get to: 500 curr count: 284
2021/12/03 15:48:06 Waiting for dst bucket item count to get to: 500 curr count: 296
2021/12/03 15:48:11 Waiting for dst bucket item count to get to: 500 curr count: 305
2021/12/03 15:48:16 Waiting for dst bucket item count to get to: 500 curr count: 317
2021/12/03 15:48:21 Waiting for dst bucket item count to get to: 500 curr count: 327
2021/12/03 15:48:26 Waiting for dst bucket item count to get to: 500 curr count: 339
2021/12/03 15:48:31 Waiting for dst bucket item count to get to: 500 curr count: 347
2021/12/03 15:48:36 Waiting for dst bucket item count to get to: 500 curr count: 358
2021/12/03 15:48:41 Waiting for dst bucket item count to get to: 500 curr count: 367
2021/12/03 15:48:46 Waiting for dst bucket item count to get to: 500 curr count: 377
2021/12/03 15:48:51 Waiting for dst bucket item count to get to: 500 curr count: 389
2021/12/03 15:48:56 Waiting for dst bucket item count to get to: 500 curr count: 399
2021/12/03 15:49:01 Waiting for dst bucket item count to get to: 500 curr count: 407
2021/12/03 15:49:06 Waiting for dst bucket item count to get to: 500 curr count: 419
2021/12/03 15:49:11 Waiting for dst bucket item count to get to: 500 curr count: 429
2021/12/03 15:49:16 Waiting for dst bucket item count to get to: 500 curr count: 441
2021/12/03 15:49:21 Waiting for dst bucket item count to get to: 500 curr count: 451
2021/12/03 15:49:26 Waiting for dst bucket item count to get to: 500 curr count: 463
2021/12/03 15:49:31 Waiting for dst bucket item count to get to: 500 curr count: 472
2021/12/03 15:49:36 Waiting for dst bucket item count to get to: 500 curr count: 481
2021/12/03 15:49:41 Waiting for dst bucket item count to get to: 500 curr count: 493
2021/12/03 15:49:41 src & dst bucket item count matched up. src bucket count: 500 dst bucket count: 500
2021/12/03 15:49:41 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 15:49:41 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 15:49:46 Waiting for app: TestN1QLAutoClose to get un-deployed
2021/12/03 15:49:51 Waiting for app: TestN1QLAutoClose to get un-deployed
2021/12/03 15:49:56 Waiting for app: TestN1QLAutoClose to get un-deployed
2021/12/03 15:50:01 Waiting for app: TestN1QLAutoClose to get un-deployed
2021/12/03 15:50:01 App: TestN1QLAutoClose got un-deployed
2021/12/03 15:50:01 No eventing-con process running
2021/12/03 15:50:02 Function: TestN1QLAutoClose delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLAutoClose deleting in the background"
}
2021/12/03 15:50:03 Context: Bucket flush, response code: 200 dump: 
2021/12/03 15:50:13 Waiting for dst bucket item count to get to: 0 curr count: 500
2021/12/03 15:50:18 Waiting for dst bucket item count to get to: 0 curr count: 250
2021/12/03 15:50:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 15:50:20 Context: Bucket flush, response code: 200 dump: 
2021/12/03 15:50:30 Waiting for dst bucket item count to get to: 0 curr count: 324
2021/12/03 15:50:35 Waiting for dst bucket item count to get to: 0 curr count: 324
2021/12/03 15:50:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLAutoClose (339.79s)
=== RUN   TestN1QLPosParams
2021/12/03 15:50:40 Waiting for app: TestN1QLPosParams to get un-deployed
2021/12/03 15:50:40 App: TestN1QLPosParams got un-deployed
2021/12/03 15:50:40 No eventing-con process running
2021/12/03 15:50:40 Function: TestN1QLPosParams delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLPosParams not found"
 }
}
2021/12/03 15:50:42 Context: Bucket flush, response code: 200 dump: 
2021/12/03 15:50:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 15:50:49 Context: Bucket flush, response code: 200 dump: 
2021/12/03 15:50:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 15:50:54 Deploying app: TestN1QLPosParams. Item count src bucket: 0 dst bucket: 0
2021/12/03 15:50:55 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLPosParams' in metakv",
  "warnings": null
 }
}
2021/12/03 15:51:00 Waiting for app: TestN1QLPosParams to get deployed
2021/12/03 15:51:05 Waiting for app: TestN1QLPosParams to get deployed
2021/12/03 15:51:10 Waiting for app: TestN1QLPosParams to get deployed
2021/12/03 15:51:15 Waiting for app: TestN1QLPosParams to get deployed
2021/12/03 15:51:15 App: TestN1QLPosParams got deployed
2021/12/03 15:51:15 Starting bucket ops to source bucket
2021/12/03 15:51:25 Waiting for dst bucket item count to get to: 5000 curr count: 344
2021/12/03 15:51:30 Waiting for dst bucket item count to get to: 5000 curr count: 1062
2021/12/03 15:51:30 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/03 15:51:30 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 15:51:30 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 15:51:35 Waiting for app: TestN1QLPosParams to get un-deployed
2021/12/03 15:51:40 Waiting for app: TestN1QLPosParams to get un-deployed
2021/12/03 15:51:45 Waiting for app: TestN1QLPosParams to get un-deployed
2021/12/03 15:51:50 Waiting for app: TestN1QLPosParams to get un-deployed
2021/12/03 15:51:50 App: TestN1QLPosParams got un-deployed
2021/12/03 15:51:50 No eventing-con process running
2021/12/03 15:51:50 Function: TestN1QLPosParams delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLPosParams deleting in the background"
}
2021/12/03 15:51:52 Context: Bucket flush, response code: 200 dump: 
2021/12/03 15:52:02 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/03 15:52:07 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/03 15:52:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 15:52:09 Context: Bucket flush, response code: 200 dump: 
2021/12/03 15:52:19 Waiting for dst bucket item count to get to: 0 curr count: 4330
2021/12/03 15:52:24 Waiting for dst bucket item count to get to: 0 curr count: 1830
2021/12/03 15:52:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLPosParams (108.43s)
=== RUN   TestN1QLExhaustConnPool
2021/12/03 15:52:29 Waiting for app: TestN1QLExhaustConnPool to get un-deployed
2021/12/03 15:52:29 App: TestN1QLExhaustConnPool got un-deployed
2021/12/03 15:52:29 No eventing-con process running
2021/12/03 15:52:29 Function: TestN1QLExhaustConnPool delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLExhaustConnPool not found"
 }
}
2021/12/03 15:52:30 Context: Bucket flush, response code: 200 dump: 
2021/12/03 15:52:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 15:52:37 Context: Bucket flush, response code: 200 dump: 
2021/12/03 15:52:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 15:52:42 Deploying app: TestN1QLExhaustConnPool. Item count src bucket: 0 dst bucket: 0
2021/12/03 15:52:43 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLExhaustConnPool' in metakv",
  "warnings": null
 }
}
2021/12/03 15:52:48 Waiting for app: TestN1QLExhaustConnPool to get deployed
2021/12/03 15:52:53 Waiting for app: TestN1QLExhaustConnPool to get deployed
2021/12/03 15:52:58 Waiting for app: TestN1QLExhaustConnPool to get deployed
2021/12/03 15:53:03 Waiting for app: TestN1QLExhaustConnPool to get deployed
2021/12/03 15:53:03 App: TestN1QLExhaustConnPool got deployed
2021/12/03 15:53:03 Starting bucket ops to source bucket
2021/12/03 15:53:09 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/03 15:53:14 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/03 15:53:14 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/03 15:53:14 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 15:53:14 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 15:53:19 Waiting for app: TestN1QLExhaustConnPool to get un-deployed
2021/12/03 15:53:24 Waiting for app: TestN1QLExhaustConnPool to get un-deployed
2021/12/03 15:53:29 Waiting for app: TestN1QLExhaustConnPool to get un-deployed
2021/12/03 15:53:34 Waiting for app: TestN1QLExhaustConnPool to get un-deployed
2021/12/03 15:53:34 App: TestN1QLExhaustConnPool got un-deployed
2021/12/03 15:53:34 No eventing-con process running
2021/12/03 15:53:34 Function: TestN1QLExhaustConnPool delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLExhaustConnPool deleting in the background"
}
2021/12/03 15:53:36 Context: Bucket flush, response code: 200 dump: 
2021/12/03 15:53:46 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/03 15:53:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 15:53:47 Context: Bucket flush, response code: 200 dump: 
2021/12/03 15:53:58 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/03 15:53:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLExhaustConnPool (93.98s)
=== RUN   TestN1QLGraceSufficientTimeOut
2021/12/03 15:54:03 Waiting for app: TestN1QLGraceSufficientTimeOut to get un-deployed
2021/12/03 15:54:03 App: TestN1QLGraceSufficientTimeOut got un-deployed
2021/12/03 15:54:03 No eventing-con process running
2021/12/03 15:54:03 Function: TestN1QLGraceSufficientTimeOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLGraceSufficientTimeOut not found"
 }
}
2021/12/03 15:54:04 Context: Bucket flush, response code: 200 dump: 
2021/12/03 15:54:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 15:54:11 Context: Bucket flush, response code: 200 dump: 
2021/12/03 15:54:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 15:54:16 Starting bucket ops to source bucket
2021/12/03 15:54:34 Starting bucket ops to source bucket
2021/12/03 15:54:52 Deploying app: TestN1QLGraceSufficientTimeOut. Item count src bucket: 30000 dst bucket: 7302
2021/12/03 15:54:53 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLGraceSufficientTimeOut' in metakv",
  "warnings": null
 }
}
2021/12/03 15:54:58 Waiting for app: TestN1QLGraceSufficientTimeOut to get deployed
2021/12/03 15:55:03 Waiting for app: TestN1QLGraceSufficientTimeOut to get deployed
2021/12/03 15:55:08 Waiting for app: TestN1QLGraceSufficientTimeOut to get deployed
2021/12/03 15:55:13 Waiting for app: TestN1QLGraceSufficientTimeOut to get deployed
2021/12/03 15:55:13 App: TestN1QLGraceSufficientTimeOut got deployed
2021/12/03 15:55:13 Starting bucket ops to source bucket
2021/12/03 15:55:18 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/03 15:55:23 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/03 15:55:28 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/03 15:55:33 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/03 15:55:38 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/03 15:55:43 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/03 15:55:48 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/03 15:55:53 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/03 15:55:58 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/03 15:56:03 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/03 15:56:03 src & dst bucket item count matched up. src bucket count: 30001 dst bucket count: 30001
2021/12/03 15:56:03 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 15:56:03 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 15:56:08 Waiting for app: TestN1QLGraceSufficientTimeOut to get un-deployed
2021/12/03 15:56:13 Waiting for app: TestN1QLGraceSufficientTimeOut to get un-deployed
2021/12/03 15:56:18 Waiting for app: TestN1QLGraceSufficientTimeOut to get un-deployed
2021/12/03 15:56:23 Waiting for app: TestN1QLGraceSufficientTimeOut to get un-deployed
2021/12/03 15:56:23 App: TestN1QLGraceSufficientTimeOut got un-deployed
2021/12/03 15:56:23 No eventing-con process running
2021/12/03 15:56:23 Function: TestN1QLGraceSufficientTimeOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLGraceSufficientTimeOut deleting in the background"
}
2021/12/03 15:56:25 Context: Bucket flush, response code: 200 dump: 
2021/12/03 15:56:35 Waiting for dst bucket item count to get to: 0 curr count: 30000
2021/12/03 15:56:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 15:56:37 Context: Bucket flush, response code: 200 dump: 
2021/12/03 15:56:47 Waiting for dst bucket item count to get to: 0 curr count: 30001
2021/12/03 15:56:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLGraceSufficientTimeOut (169.37s)
=== RUN   TestN1QLQueryConsistency
2021/12/03 15:56:52 Waiting for app: n1qlquery_consistency to get un-deployed
2021/12/03 15:56:52 App: n1qlquery_consistency got un-deployed
2021/12/03 15:56:52 No eventing-con process running
2021/12/03 15:56:52 Function: n1qlquery_consistency delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: n1qlquery_consistency not found"
 }
}
2021/12/03 15:56:54 Context: Bucket flush, response code: 200 dump: 
2021/12/03 15:56:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 15:57:01 Context: Bucket flush, response code: 200 dump: 
2021/12/03 15:57:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 15:57:06 Starting bucket ops to source bucket
2021/12/03 15:57:09 Deploying app: n1qlquery_consistency. Item count src bucket: 0 dst bucket: 0
2021/12/03 15:57:10 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'n1qlquery_consistency' in metakv",
  "warnings": null
 }
}
2021/12/03 15:57:15 Waiting for app: n1qlquery_consistency to get deployed
2021/12/03 15:57:20 Waiting for app: n1qlquery_consistency to get deployed
2021/12/03 15:57:25 Waiting for app: n1qlquery_consistency to get deployed
2021/12/03 15:57:30 Waiting for app: n1qlquery_consistency to get deployed
2021/12/03 15:57:30 App: n1qlquery_consistency got deployed
2021/12/03 15:57:35 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/03 15:57:40 Waiting for dst bucket item count to get to: 5000 curr count: 3936
2021/12/03 15:57:45 Waiting for dst bucket item count to get to: 5000 curr count: 3936
2021/12/03 15:57:45 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/03 15:57:45 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/03 15:57:45 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 15:57:50 Waiting for app: n1qlquery_consistency to get un-deployed
2021/12/03 15:57:55 Waiting for app: n1qlquery_consistency to get un-deployed
2021/12/03 15:58:00 Waiting for app: n1qlquery_consistency to get un-deployed
2021/12/03 15:58:05 Waiting for app: n1qlquery_consistency to get un-deployed
2021/12/03 15:58:05 App: n1qlquery_consistency got un-deployed
2021/12/03 15:58:05 No eventing-con process running
2021/12/03 15:58:05 Function: n1qlquery_consistency delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: n1qlquery_consistency deleting in the background"
}
2021/12/03 15:58:07 Context: Bucket flush, response code: 200 dump: 
2021/12/03 15:58:17 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/03 15:58:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 15:58:19 Context: Bucket flush, response code: 200 dump: 
2021/12/03 15:58:29 Waiting for dst bucket item count to get to: 0 curr count: 4525
2021/12/03 15:58:34 Waiting for dst bucket item count to get to: 0 curr count: 2025
2021/12/03 15:58:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLQueryConsistency (106.73s)
PASS
ok  	github.com/couchbase/eventing/tests/functional_tests	1535.136s

N1ql tests took 0 hours, 25 minutes and 40 seconds.

Starting server
Starting server: attempt 1

Rebalance tests

2021/12/03 16:00:09 Started rebalance
Post "http://127.0.0.1:9000/pools/default": read tcp 127.0.0.1:48622->127.0.0.1:9000: read: connection reset by peer
Quota setup Post "http://127.0.0.1:9000/pools/default": read tcp 127.0.0.1:48622->127.0.0.1:9000: read: connection reset by peer
=== RUN   TestDuplicateWithEventingRebIn
2021/12/03 16:00:28 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
2021/12/03 16:00:28 App: TestDuplicateWithEventingRebIn got un-deployed
2021/12/03 16:00:28 No eventing-con process running
2021/12/03 16:00:28 Function: TestDuplicateWithEventingRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithEventingRebIn not found"
 }
}
2021/12/03 16:00:29 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:00:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 16:00:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 16:00:36 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:00:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 16:00:41 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:00:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 16:00:46 Deploying app: TestDuplicateWithEventingRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/03 16:00:47 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithEventingRebIn' in metakv",
  "warnings": null
 }
}
2021/12/03 16:00:52 Waiting for app: TestDuplicateWithEventingRebIn to get deployed
2021/12/03 16:00:57 Waiting for app: TestDuplicateWithEventingRebIn to get deployed
2021/12/03 16:01:02 Waiting for app: TestDuplicateWithEventingRebIn to get deployed
2021/12/03 16:01:07 Waiting for app: TestDuplicateWithEventingRebIn to get deployed
2021/12/03 16:01:07 App: TestDuplicateWithEventingRebIn got deployed
2021/12/03 16:01:07 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/03 16:01:07 Starting bucket ops to source bucket
2021/12/03 16:01:23 Removing node(s): [] from the cluster
2021/12/03 16:01:28 Rebalance progress: 25
2021/12/03 16:01:33 Rebalance progress: 25
2021/12/03 16:01:38 Rebalance progress: 25
2021/12/03 16:01:43 Rebalance progress: 25
2021/12/03 16:01:48 Rebalance progress: 75
2021/12/03 16:01:53 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:01:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:01:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 16:01:53 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9302/api/v1/stats": dial tcp 127.0.0.1:9302: connect: connection refused
2021/12/03 16:01:53 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/03 16:01:53 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/03 16:01:58 Rebalance progress: 25
2021/12/03 16:02:03 Rebalance progress: 25
2021/12/03 16:02:08 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:02:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:02:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 16:02:08 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9302/api/v1/stats": dial tcp 127.0.0.1:9302: connect: connection refused
2021/12/03 16:02:08 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/03 16:02:08 Waiting for src bucket item count to get to: 200000 curr count: 80490
2021/12/03 16:02:13 Waiting for src bucket item count to get to: 200000 curr count: 96929
2021/12/03 16:02:18 Waiting for src bucket item count to get to: 200000 curr count: 96929
2021/12/03 16:02:23 Waiting for src bucket item count to get to: 200000 curr count: 113395
2021/12/03 16:02:28 Waiting for src bucket item count to get to: 200000 curr count: 113395
2021/12/03 16:02:33 Waiting for src bucket item count to get to: 200000 curr count: 129840
2021/12/03 16:02:38 Waiting for src bucket item count to get to: 200000 curr count: 129840
2021/12/03 16:02:43 Waiting for src bucket item count to get to: 200000 curr count: 146260
2021/12/03 16:02:48 Waiting for src bucket item count to get to: 200000 curr count: 146260
2021/12/03 16:02:53 Waiting for src bucket item count to get to: 200000 curr count: 146260
2021/12/03 16:02:58 Waiting for src bucket item count to get to: 200000 curr count: 162671
2021/12/03 16:03:03 Waiting for src bucket item count to get to: 200000 curr count: 162671
2021/12/03 16:03:08 Waiting for src bucket item count to get to: 200000 curr count: 178819
2021/12/03 16:03:13 Waiting for src bucket item count to get to: 200000 curr count: 178819
2021/12/03 16:03:18 Waiting for src bucket item count to get to: 200000 curr count: 195159
2021/12/03 16:03:23 Waiting for src bucket item count to get to: 200000 curr count: 195159
2021/12/03 16:03:33 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/03 16:03:38 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/03 16:03:43 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/03 16:03:48 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/03 16:03:53 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/03 16:03:58 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/03 16:04:03 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/03 16:04:08 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/03 16:04:13 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/03 16:04:18 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/03 16:04:18 Post SET ops. Expected item count: 200000 got 200001
2021/12/03 16:04:18 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 16:04:23 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
2021/12/03 16:04:28 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
2021/12/03 16:04:33 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
2021/12/03 16:04:38 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
2021/12/03 16:04:38 App: TestDuplicateWithEventingRebIn got un-deployed
2021/12/03 16:04:38 No eventing-con process running
2021/12/03 16:04:38 Function: TestDuplicateWithEventingRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithEventingRebIn deleting in the background"
}
2021/12/03 16:04:40 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:04:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 16:04:50 Waiting for dst bucket item count to get to: 0 curr count: 42537
2021/12/03 16:04:55 Waiting for dst bucket item count to get to: 0 curr count: 42537
2021/12/03 16:04:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 16:04:57 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:05:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 16:05:07 Waiting for dst bucket item count to get to: 0 curr count: 200001
2021/12/03 16:05:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithEventingRebIn (289.30s)
=== RUN   TestDuplicateWithEventingRebOut
2021/12/03 16:05:12 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/03 16:05:12 Removing node(s): [] from the cluster
2021/12/03 16:05:18 Rebalance progress: 100
2021/12/03 16:05:23 Waiting for app: TestDuplicateWithEventingRebOut to get un-deployed
2021/12/03 16:05:23 App: TestDuplicateWithEventingRebOut got un-deployed
2021/12/03 16:05:23 No eventing-con process running
2021/12/03 16:05:23 Function: TestDuplicateWithEventingRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithEventingRebOut not found"
 }
}
2021/12/03 16:05:24 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:05:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 16:05:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 16:05:31 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:05:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 16:05:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:05:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 16:05:41 Deploying app: TestDuplicateWithEventingRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/03 16:05:42 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithEventingRebOut' in metakv",
  "warnings": null
 }
}
2021/12/03 16:05:47 Waiting for app: TestDuplicateWithEventingRebOut to get deployed
2021/12/03 16:05:52 Waiting for app: TestDuplicateWithEventingRebOut to get deployed
2021/12/03 16:05:57 Waiting for app: TestDuplicateWithEventingRebOut to get deployed
2021/12/03 16:06:02 Waiting for app: TestDuplicateWithEventingRebOut to get deployed
2021/12/03 16:06:02 App: TestDuplicateWithEventingRebOut got deployed
2021/12/03 16:06:02 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/03 16:06:02 Starting bucket ops to source bucket
2021/12/03 16:06:07 Rebalance progress: 25
2021/12/03 16:06:12 Rebalance progress: 25
2021/12/03 16:06:17 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:06:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:06:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 16:06:17 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9302/api/v1/stats": dial tcp 127.0.0.1:9302: connect: connection refused
2021/12/03 16:06:17 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/03 16:06:17 Waiting for src bucket item count to get to: 200000 curr count: 10854
2021/12/03 16:06:22 Waiting for src bucket item count to get to: 200000 curr count: 27227
2021/12/03 16:06:27 Waiting for src bucket item count to get to: 200000 curr count: 27227
2021/12/03 16:06:32 Waiting for src bucket item count to get to: 200000 curr count: 43534
2021/12/03 16:06:37 Waiting for src bucket item count to get to: 200000 curr count: 43534
2021/12/03 16:06:42 Waiting for src bucket item count to get to: 200000 curr count: 59826
2021/12/03 16:06:47 Waiting for src bucket item count to get to: 200000 curr count: 59826
2021/12/03 16:06:52 Waiting for src bucket item count to get to: 200000 curr count: 59826
2021/12/03 16:06:57 Waiting for src bucket item count to get to: 200000 curr count: 75851
2021/12/03 16:07:02 Waiting for src bucket item count to get to: 200000 curr count: 75851
2021/12/03 16:07:07 Waiting for src bucket item count to get to: 200000 curr count: 91917
2021/12/03 16:07:12 Waiting for src bucket item count to get to: 200000 curr count: 91917
2021/12/03 16:07:17 Waiting for src bucket item count to get to: 200000 curr count: 108200
2021/12/03 16:07:22 Waiting for src bucket item count to get to: 200000 curr count: 108200
2021/12/03 16:07:27 Waiting for src bucket item count to get to: 200000 curr count: 124583
2021/12/03 16:07:32 Waiting for src bucket item count to get to: 200000 curr count: 124583
2021/12/03 16:07:37 Waiting for src bucket item count to get to: 200000 curr count: 140741
2021/12/03 16:07:42 Waiting for src bucket item count to get to: 200000 curr count: 140741
2021/12/03 16:07:47 Waiting for src bucket item count to get to: 200000 curr count: 156890
2021/12/03 16:07:52 Waiting for src bucket item count to get to: 200000 curr count: 156890
2021/12/03 16:07:57 Waiting for src bucket item count to get to: 200000 curr count: 172953
2021/12/03 16:08:02 Waiting for src bucket item count to get to: 200000 curr count: 172953
2021/12/03 16:08:07 Waiting for src bucket item count to get to: 200000 curr count: 189335
2021/12/03 16:08:12 Waiting for src bucket item count to get to: 200000 curr count: 189335
2021/12/03 16:08:17 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/03 16:08:17 Post SET ops. Expected item count: 200000 got 200000
2021/12/03 16:08:17 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 16:08:22 Waiting for app: TestDuplicateWithEventingRebOut to get un-deployed
2021/12/03 16:08:27 Waiting for app: TestDuplicateWithEventingRebOut to get un-deployed
2021/12/03 16:08:32 Waiting for app: TestDuplicateWithEventingRebOut to get un-deployed
2021/12/03 16:08:37 Waiting for app: TestDuplicateWithEventingRebOut to get un-deployed
2021/12/03 16:08:37 App: TestDuplicateWithEventingRebOut got un-deployed
2021/12/03 16:08:37 No eventing-con process running
2021/12/03 16:08:37 Function: TestDuplicateWithEventingRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithEventingRebOut deleting in the background"
}
2021/12/03 16:08:39 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:08:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 16:08:49 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/03 16:08:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 16:08:51 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:08:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 16:09:01 Waiting for dst bucket item count to get to: 0 curr count: 194581
2021/12/03 16:09:06 Waiting for dst bucket item count to get to: 0 curr count: 194581
2021/12/03 16:09:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithEventingRebOut (239.04s)
=== RUN   TestDuplicateWithEventingSwapReb
2021/12/03 16:09:16 Waiting for app: TestDuplicateWithEventingSwapReb to get un-deployed
2021/12/03 16:09:16 App: TestDuplicateWithEventingSwapReb got un-deployed
2021/12/03 16:09:16 No eventing-con process running
2021/12/03 16:09:16 Function: TestDuplicateWithEventingSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithEventingSwapReb not found"
 }
}
2021/12/03 16:09:18 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:09:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 16:09:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 16:09:25 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:09:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 16:09:30 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:09:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 16:09:35 Deploying app: TestDuplicateWithEventingSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/03 16:09:36 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithEventingSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/03 16:09:41 Waiting for app: TestDuplicateWithEventingSwapReb to get deployed
2021/12/03 16:09:46 Waiting for app: TestDuplicateWithEventingSwapReb to get deployed
2021/12/03 16:09:51 Waiting for app: TestDuplicateWithEventingSwapReb to get deployed
2021/12/03 16:09:56 Waiting for app: TestDuplicateWithEventingSwapReb to get deployed
2021/12/03 16:09:56 App: TestDuplicateWithEventingSwapReb got deployed
2021/12/03 16:09:56 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/03 16:09:56 Starting bucket ops to source bucket
2021/12/03 16:09:56 Removing node(s): [] from the cluster
2021/12/03 16:10:01 Rebalance progress: 25
2021/12/03 16:10:06 Rebalance progress: 25
2021/12/03 16:10:11 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:10:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:10:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 16:10:11 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9302/api/v1/stats": dial tcp 127.0.0.1:9302: connect: connection refused
2021/12/03 16:10:11 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/03 16:10:11 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/03 16:10:18 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/03 16:10:23 Rebalance progress: 16.66666666666667
2021/12/03 16:10:28 Rebalance progress: 16.66666666666667
2021/12/03 16:10:33 Rebalance progress: 16.66666666666667
2021/12/03 16:10:38 Rebalance progress: 72.22222222222223
2021/12/03 16:10:43 Rebalance progress: 72.22222222222223
2021/12/03 16:10:48 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:10:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:10:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 16:10:48 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/03 16:10:48 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/03 16:10:54 Rebalance progress: 16.66666666666667
2021/12/03 16:10:59 Rebalance progress: 16.66666666666667
2021/12/03 16:11:04 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:11:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:11:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 16:11:04 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/03 16:11:04 Waiting for src bucket item count to get to: 200000 curr count: 94226
2021/12/03 16:11:09 Waiting for src bucket item count to get to: 200000 curr count: 94226
2021/12/03 16:11:14 Waiting for src bucket item count to get to: 200000 curr count: 109950
2021/12/03 16:11:19 Waiting for src bucket item count to get to: 200000 curr count: 109950
2021/12/03 16:11:24 Waiting for src bucket item count to get to: 200000 curr count: 125943
2021/12/03 16:11:29 Waiting for src bucket item count to get to: 200000 curr count: 125943
2021/12/03 16:11:34 Waiting for src bucket item count to get to: 200000 curr count: 141781
2021/12/03 16:11:39 Waiting for src bucket item count to get to: 200000 curr count: 141781
2021/12/03 16:11:44 Waiting for src bucket item count to get to: 200000 curr count: 158158
2021/12/03 16:11:49 Waiting for src bucket item count to get to: 200000 curr count: 158158
2021/12/03 16:11:54 Waiting for src bucket item count to get to: 200000 curr count: 173737
2021/12/03 16:11:59 Waiting for src bucket item count to get to: 200000 curr count: 173737
2021/12/03 16:12:04 Waiting for src bucket item count to get to: 200000 curr count: 189581
2021/12/03 16:12:09 Waiting for src bucket item count to get to: 200000 curr count: 189581
2021/12/03 16:12:14 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/03 16:12:14 Post SET ops. Expected item count: 200000 got 200000
2021/12/03 16:12:14 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 16:12:19 Waiting for app: TestDuplicateWithEventingSwapReb to get un-deployed
2021/12/03 16:12:24 Waiting for app: TestDuplicateWithEventingSwapReb to get un-deployed
2021/12/03 16:12:29 Waiting for app: TestDuplicateWithEventingSwapReb to get un-deployed
2021/12/03 16:12:34 Waiting for app: TestDuplicateWithEventingSwapReb to get un-deployed
2021/12/03 16:12:34 App: TestDuplicateWithEventingSwapReb got un-deployed
2021/12/03 16:12:34 No eventing-con process running
2021/12/03 16:12:34 Function: TestDuplicateWithEventingSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithEventingSwapReb deleting in the background"
}
2021/12/03 16:12:36 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:12:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 16:12:46 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/03 16:12:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 16:12:47 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:12:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 16:12:57 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/03 16:12:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithEventingSwapReb (231.39s)
=== RUN   TestDuplicateWithKVRebIn
2021/12/03 16:13:07 Waiting for app: TestDuplicateWithKVRebIn to get un-deployed
2021/12/03 16:13:07 App: TestDuplicateWithKVRebIn got un-deployed
2021/12/03 16:13:07 No eventing-con process running
2021/12/03 16:13:07 Function: TestDuplicateWithKVRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithKVRebIn not found"
 }
}
2021/12/03 16:13:09 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:13:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 16:13:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 16:13:16 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:13:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 16:13:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:13:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 16:13:26 Deploying app: TestDuplicateWithKVRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/03 16:13:27 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithKVRebIn' in metakv",
  "warnings": null
 }
}
2021/12/03 16:13:32 Waiting for app: TestDuplicateWithKVRebIn to get deployed
2021/12/03 16:13:37 Waiting for app: TestDuplicateWithKVRebIn to get deployed
2021/12/03 16:13:42 Waiting for app: TestDuplicateWithKVRebIn to get deployed
2021/12/03 16:13:47 Waiting for app: TestDuplicateWithKVRebIn to get deployed
2021/12/03 16:13:47 App: TestDuplicateWithKVRebIn got deployed
2021/12/03 16:13:47 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/03 16:13:47 Starting bucket ops to source bucket
2021/12/03 16:13:47 Removing node(s): [] from the cluster
2021/12/03 16:13:52 Rebalance progress: 16.66666666666667
2021/12/03 16:13:57 Rebalance progress: 16.66666666666667
2021/12/03 16:14:02 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:14:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:14:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 16:14:02 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/03 16:14:02 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/03 16:14:07 Rebalance progress: 16.66666666666667
2021/12/03 16:14:12 Rebalance progress: 16.66666666666667
2021/12/03 16:14:17 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:14:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:14:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 16:14:17 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/03 16:14:17 Waiting for src bucket item count to get to: 200000 curr count: 33997
2021/12/03 16:14:23 Waiting for src bucket item count to get to: 200000 curr count: 49548
2021/12/03 16:14:28 Waiting for src bucket item count to get to: 200000 curr count: 49548
2021/12/03 16:14:33 Waiting for src bucket item count to get to: 200000 curr count: 65667
2021/12/03 16:14:38 Waiting for src bucket item count to get to: 200000 curr count: 65667
2021/12/03 16:14:43 Waiting for src bucket item count to get to: 200000 curr count: 81248
2021/12/03 16:14:48 Waiting for src bucket item count to get to: 200000 curr count: 81248
2021/12/03 16:14:53 Waiting for src bucket item count to get to: 200000 curr count: 81248
2021/12/03 16:14:58 Waiting for src bucket item count to get to: 200000 curr count: 97051
2021/12/03 16:15:03 Waiting for src bucket item count to get to: 200000 curr count: 97051
2021/12/03 16:15:08 Waiting for src bucket item count to get to: 200000 curr count: 113456
2021/12/03 16:15:13 Waiting for src bucket item count to get to: 200000 curr count: 113456
2021/12/03 16:15:18 Waiting for src bucket item count to get to: 200000 curr count: 129657
2021/12/03 16:15:23 Waiting for src bucket item count to get to: 200000 curr count: 129657
2021/12/03 16:15:28 Waiting for src bucket item count to get to: 200000 curr count: 145871
2021/12/03 16:15:33 Waiting for src bucket item count to get to: 200000 curr count: 145871
2021/12/03 16:15:38 Waiting for src bucket item count to get to: 200000 curr count: 162390
2021/12/03 16:15:43 Waiting for src bucket item count to get to: 200000 curr count: 162390
2021/12/03 16:15:48 Waiting for src bucket item count to get to: 200000 curr count: 177877
2021/12/03 16:15:53 Waiting for src bucket item count to get to: 200000 curr count: 177877
2021/12/03 16:15:58 Waiting for src bucket item count to get to: 200000 curr count: 193530
2021/12/03 16:16:03 Waiting for src bucket item count to get to: 200000 curr count: 193530
2021/12/03 16:16:08 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/03 16:16:08 Post SET ops. Expected item count: 200000 got 200000
2021/12/03 16:16:08 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 16:16:13 Waiting for app: TestDuplicateWithKVRebIn to get un-deployed
2021/12/03 16:16:18 Waiting for app: TestDuplicateWithKVRebIn to get un-deployed
2021/12/03 16:16:23 Waiting for app: TestDuplicateWithKVRebIn to get un-deployed
2021/12/03 16:16:28 Waiting for app: TestDuplicateWithKVRebIn to get un-deployed
2021/12/03 16:16:28 App: TestDuplicateWithKVRebIn got un-deployed
2021/12/03 16:16:28 No eventing-con process running
2021/12/03 16:16:28 Function: TestDuplicateWithKVRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithKVRebIn deleting in the background"
}
2021/12/03 16:16:30 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:16:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 16:16:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 16:16:36 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:16:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 16:16:46 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/03 16:16:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithKVRebIn (228.98s)
=== RUN   TestDuplicateWithKVRebOut
2021/12/03 16:16:51 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/03 16:16:52 Removing node(s): [] from the cluster
2021/12/03 16:16:57 Rebalance progress: 100
2021/12/03 16:17:02 Waiting for app: TestDuplicateWithKVRebOut to get un-deployed
2021/12/03 16:17:02 App: TestDuplicateWithKVRebOut got un-deployed
2021/12/03 16:17:02 No eventing-con process running
2021/12/03 16:17:02 Function: TestDuplicateWithKVRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithKVRebOut not found"
 }
}
2021/12/03 16:17:04 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:17:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 16:17:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 16:17:10 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:17:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 16:17:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:17:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 16:17:20 Deploying app: TestDuplicateWithKVRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/03 16:17:21 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithKVRebOut' in metakv",
  "warnings": null
 }
}
2021/12/03 16:17:26 Waiting for app: TestDuplicateWithKVRebOut to get deployed
2021/12/03 16:17:31 Waiting for app: TestDuplicateWithKVRebOut to get deployed
2021/12/03 16:17:36 Waiting for app: TestDuplicateWithKVRebOut to get deployed
2021/12/03 16:17:41 Waiting for app: TestDuplicateWithKVRebOut to get deployed
2021/12/03 16:17:41 App: TestDuplicateWithKVRebOut got deployed
2021/12/03 16:17:41 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/03 16:17:41 Starting bucket ops to source bucket
2021/12/03 16:17:46 Rebalance progress: 16.66666666666667
2021/12/03 16:17:51 Rebalance progress: 16.66666666666667
2021/12/03 16:17:56 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:17:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:17:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 16:17:57 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/03 16:17:57 Waiting for src bucket item count to get to: 200000 curr count: 11923
2021/12/03 16:18:02 Waiting for src bucket item count to get to: 200000 curr count: 28045
2021/12/03 16:18:07 Waiting for src bucket item count to get to: 200000 curr count: 28045
2021/12/03 16:18:12 Waiting for src bucket item count to get to: 200000 curr count: 44200
2021/12/03 16:18:17 Waiting for src bucket item count to get to: 200000 curr count: 44200
2021/12/03 16:18:22 Waiting for src bucket item count to get to: 200000 curr count: 60380
2021/12/03 16:18:27 Waiting for src bucket item count to get to: 200000 curr count: 60380
2021/12/03 16:18:32 Waiting for src bucket item count to get to: 200000 curr count: 60380
2021/12/03 16:18:37 Waiting for src bucket item count to get to: 200000 curr count: 76047
2021/12/03 16:18:42 Waiting for src bucket item count to get to: 200000 curr count: 76047
2021/12/03 16:18:47 Waiting for src bucket item count to get to: 200000 curr count: 92060
2021/12/03 16:18:52 Waiting for src bucket item count to get to: 200000 curr count: 92060
2021/12/03 16:18:57 Waiting for src bucket item count to get to: 200000 curr count: 108274
2021/12/03 16:19:02 Waiting for src bucket item count to get to: 200000 curr count: 108274
2021/12/03 16:19:07 Waiting for src bucket item count to get to: 200000 curr count: 124505
2021/12/03 16:19:12 Waiting for src bucket item count to get to: 200000 curr count: 124505
2021/12/03 16:19:17 Waiting for src bucket item count to get to: 200000 curr count: 140745
2021/12/03 16:19:22 Waiting for src bucket item count to get to: 200000 curr count: 140745
2021/12/03 16:19:27 Waiting for src bucket item count to get to: 200000 curr count: 157087
2021/12/03 16:19:32 Waiting for src bucket item count to get to: 200000 curr count: 157087
2021/12/03 16:19:37 Waiting for src bucket item count to get to: 200000 curr count: 172518
2021/12/03 16:19:42 Waiting for src bucket item count to get to: 200000 curr count: 172518
2021/12/03 16:19:47 Waiting for src bucket item count to get to: 200000 curr count: 188301
2021/12/03 16:19:52 Waiting for src bucket item count to get to: 200000 curr count: 188301
2021/12/03 16:19:57 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/03 16:19:57 Post SET ops. Expected item count: 200000 got 200000
2021/12/03 16:19:57 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 16:20:02 Waiting for app: TestDuplicateWithKVRebOut to get un-deployed
2021/12/03 16:20:07 Waiting for app: TestDuplicateWithKVRebOut to get un-deployed
2021/12/03 16:20:12 Waiting for app: TestDuplicateWithKVRebOut to get un-deployed
2021/12/03 16:20:17 Waiting for app: TestDuplicateWithKVRebOut to get un-deployed
2021/12/03 16:20:17 App: TestDuplicateWithKVRebOut got un-deployed
2021/12/03 16:20:17 No eventing-con process running
2021/12/03 16:20:17 Function: TestDuplicateWithKVRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithKVRebOut deleting in the background"
}
2021/12/03 16:20:19 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:20:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 16:20:29 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/03 16:20:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 16:20:30 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:20:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 16:20:40 Waiting for dst bucket item count to get to: 0 curr count: 99453
2021/12/03 16:20:45 Waiting for dst bucket item count to get to: 0 curr count: 99453
2021/12/03 16:20:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithKVRebOut (239.07s)
=== RUN   TestDuplicateWithKVSwapReb
2021/12/03 16:20:55 Waiting for app: TestDuplicateWithKVSwapReb to get un-deployed
2021/12/03 16:20:55 App: TestDuplicateWithKVSwapReb got un-deployed
2021/12/03 16:20:55 No eventing-con process running
2021/12/03 16:20:55 Function: TestDuplicateWithKVSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithKVSwapReb not found"
 }
}
2021/12/03 16:20:57 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:21:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 16:21:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 16:21:04 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:21:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 16:21:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:21:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 16:21:14 Deploying app: TestDuplicateWithKVSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/03 16:21:15 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithKVSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/03 16:21:20 Waiting for app: TestDuplicateWithKVSwapReb to get deployed
2021/12/03 16:21:25 Waiting for app: TestDuplicateWithKVSwapReb to get deployed
2021/12/03 16:21:30 Waiting for app: TestDuplicateWithKVSwapReb to get deployed
2021/12/03 16:21:35 Waiting for app: TestDuplicateWithKVSwapReb to get deployed
2021/12/03 16:21:35 App: TestDuplicateWithKVSwapReb got deployed
2021/12/03 16:21:35 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/03 16:21:35 Starting bucket ops to source bucket
2021/12/03 16:21:36 Removing node(s): [] from the cluster
2021/12/03 16:21:41 Rebalance progress: 16.66666666666667
2021/12/03 16:21:46 Rebalance progress: 16.66666666666667
2021/12/03 16:21:51 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:21:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:21:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 16:21:51 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/03 16:21:51 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/03 16:21:52 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/03 16:21:57 Rebalance progress: 16.66666666666667
2021/12/03 16:22:02 Rebalance progress: 16.66666666666667
2021/12/03 16:22:07 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:22:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:22:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 16:22:07 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/03 16:22:07 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/03 16:22:12 Rebalance progress: 16.66666666666667
2021/12/03 16:22:17 Rebalance progress: 16.66666666666667
2021/12/03 16:22:22 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:22:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:22:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 16:22:22 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/03 16:22:22 Waiting for src bucket item count to get to: 200000 curr count: 68205
2021/12/03 16:22:27 Waiting for src bucket item count to get to: 200000 curr count: 68205
2021/12/03 16:22:32 Waiting for src bucket item count to get to: 200000 curr count: 83151
2021/12/03 16:22:37 Waiting for src bucket item count to get to: 200000 curr count: 83151
2021/12/03 16:22:42 Waiting for src bucket item count to get to: 200000 curr count: 98947
2021/12/03 16:22:47 Waiting for src bucket item count to get to: 200000 curr count: 98947
2021/12/03 16:22:52 Waiting for src bucket item count to get to: 200000 curr count: 115277
2021/12/03 16:22:57 Waiting for src bucket item count to get to: 200000 curr count: 115277
2021/12/03 16:23:02 Waiting for src bucket item count to get to: 200000 curr count: 131374
2021/12/03 16:23:07 Waiting for src bucket item count to get to: 200000 curr count: 131374
2021/12/03 16:23:12 Waiting for src bucket item count to get to: 200000 curr count: 147866
2021/12/03 16:23:17 Waiting for src bucket item count to get to: 200000 curr count: 147866
2021/12/03 16:23:22 Waiting for src bucket item count to get to: 200000 curr count: 163939
2021/12/03 16:23:27 Waiting for src bucket item count to get to: 200000 curr count: 163939
2021/12/03 16:23:32 Waiting for src bucket item count to get to: 200000 curr count: 179099
2021/12/03 16:23:37 Waiting for src bucket item count to get to: 200000 curr count: 179099
2021/12/03 16:23:42 Waiting for src bucket item count to get to: 200000 curr count: 194836
2021/12/03 16:23:47 Waiting for src bucket item count to get to: 200000 curr count: 194836
2021/12/03 16:23:52 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/03 16:23:52 Post SET ops. Expected item count: 200000 got 200000
2021/12/03 16:23:52 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 16:23:57 Waiting for app: TestDuplicateWithKVSwapReb to get un-deployed
2021/12/03 16:24:02 Waiting for app: TestDuplicateWithKVSwapReb to get un-deployed
2021/12/03 16:24:07 Waiting for app: TestDuplicateWithKVSwapReb to get un-deployed
2021/12/03 16:24:12 Waiting for app: TestDuplicateWithKVSwapReb to get un-deployed
2021/12/03 16:24:12 App: TestDuplicateWithKVSwapReb got un-deployed
2021/12/03 16:24:12 No eventing-con process running
2021/12/03 16:24:12 Function: TestDuplicateWithKVSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithKVSwapReb deleting in the background"
}
2021/12/03 16:24:14 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:24:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 16:24:24 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/03 16:24:29 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/03 16:24:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 16:24:31 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:24:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 16:24:41 Waiting for dst bucket item count to get to: 0 curr count: 153697
2021/12/03 16:24:46 Waiting for dst bucket item count to get to: 0 curr count: 153697
2021/12/03 16:24:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithKVSwapReb (240.35s)
=== RUN   TestEventingRebKVOpsOnUpdateBucketOpOneByOne
2021/12/03 16:24:56 Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/03 16:24:56 App: TestEventingRebKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/03 16:24:56 No eventing-con process running
2021/12/03 16:24:56 Function: TestEventingRebKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpOneByOne not found"
 }
}
2021/12/03 16:24:58 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:25:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 16:25:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 16:25:04 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:25:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 16:25:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:25:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 16:25:14 Deploying app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne. Item count src bucket: 0 dst bucket: 0
2021/12/03 16:25:15 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebKVOpsOnUpdateBucketOpOneByOne' in metakv",
  "warnings": null
 }
}
2021/12/03 16:25:20 Starting bucket ops to source bucket
2021/12/03 16:25:25 Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/03 16:25:30 Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/03 16:25:36 Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/03 16:25:36 App: TestEventingRebKVOpsOnUpdateBucketOpOneByOne got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:25:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:25:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 16:25:36 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/03 16:25:36 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/03 16:25:36 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/03 16:25:41 Rebalance progress: 16.66666666666667
2021/12/03 16:25:46 Rebalance progress: 16.66666666666667
2021/12/03 16:25:51 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:25:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:25:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 16:25:51 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/03 16:25:51 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/03 16:25:52 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/03 16:25:57 Rebalance progress: 16.66666666666667
2021/12/03 16:26:02 Rebalance progress: 16.66666666666667
2021/12/03 16:26:07 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:26:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:26:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 16:26:07 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/03 16:26:07 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/03 16:26:12 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/03 16:26:17 Rebalance progress: 12.5
2021/12/03 16:26:22 Rebalance progress: 12.5
2021/12/03 16:26:27 Rebalance progress: 12.5
2021/12/03 16:26:32 Rebalance progress: 70.83333333333334
2021/12/03 16:26:37 Rebalance progress: 70.83333333333334
2021/12/03 16:26:42 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:26:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:26:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 16:26:42 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/03 16:26:48 Rebalance progress: 12.5
2021/12/03 16:26:53 Rebalance progress: 12.5
2021/12/03 16:26:58 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:26:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:26:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 16:26:58 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/03 16:27:03 Rebalance progress: 12.5
2021/12/03 16:27:08 Rebalance progress: 12.5
2021/12/03 16:27:13 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:27:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:27:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 16:27:13 clusterSetup.go:253: Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/03 16:27:18 Rebalance progress: 12.5
2021/12/03 16:27:23 Rebalance progress: 12.5
2021/12/03 16:27:28 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:27:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:27:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 16:27:28 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 16:27:33 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/03 16:27:38 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/03 16:27:43 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/03 16:27:48 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/03 16:27:48 clusterSetup.go:405: App: TestEventingRebKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/03 16:27:48 eventingSetup.go:46: No eventing-con process running
2021/12/03 16:27:48 eventingSetup.go:418: Function: TestEventingRebKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpOneByOne deleting in the background"
}
2021/12/03 16:27:50 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:27:55 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 16:27:55 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 16:27:57 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:28:02 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 16:28:07 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12714
2021/12/03 16:28:07 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebKVOpsOnUpdateBucketOpOneByOne (200.75s)
=== RUN   TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce
2021/12/03 16:28:17 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get un-deployed
2021/12/03 16:28:17 clusterSetup.go:405: App: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce got un-deployed
2021/12/03 16:28:17 eventingSetup.go:46: No eventing-con process running
2021/12/03 16:28:17 eventingSetup.go:418: Function: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce not found"
 }
}
2021/12/03 16:28:18 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:28:23 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 16:28:23 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 16:28:25 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:28:30 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 16:28:30 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:28:35 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 16:28:35 eventingSetup.go:125: Deploying app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce. Item count src bucket: 0 dst bucket: 0
2021/12/03 16:28:36 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce' in metakv",
  "warnings": null
 }
}
2021/12/03 16:28:41 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/03 16:28:46 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get deployed
2021/12/03 16:28:51 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get deployed
2021/12/03 16:28:56 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get deployed
2021/12/03 16:28:56 clusterSetup.go:354: App: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:28:56 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:28:56 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 16:28:56 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/03 16:28:57 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/03 16:28:57 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/03 16:28:58 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/03 16:29:03 Rebalance progress: 12.5
2021/12/03 16:29:08 Rebalance progress: 12.5
2021/12/03 16:29:13 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:29:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:29:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 16:29:13 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002 127.0.0.1:19003] from the cluster
2021/12/03 16:29:18 Rebalance progress: 12.5
2021/12/03 16:29:23 Rebalance progress: 12.5
2021/12/03 16:29:28 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:29:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:29:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 16:29:28 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 16:29:33 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get un-deployed
2021/12/03 16:29:38 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get un-deployed
2021/12/03 16:29:43 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get un-deployed
2021/12/03 16:29:48 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get un-deployed
2021/12/03 16:29:48 clusterSetup.go:405: App: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce got un-deployed
2021/12/03 16:29:48 eventingSetup.go:46: No eventing-con process running
2021/12/03 16:29:48 eventingSetup.go:418: Function: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce deleting in the background"
}
2021/12/03 16:29:50 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:29:55 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 16:29:55 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 16:29:57 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:30:02 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 16:30:07 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4629
2021/12/03 16:30:07 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce (120.01s)
=== RUN   TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings
2021/12/03 16:30:17 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/03 16:30:17 clusterSetup.go:405: App: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings got un-deployed
2021/12/03 16:30:17 eventingSetup.go:46: No eventing-con process running
2021/12/03 16:30:17 eventingSetup.go:418: Function: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings not found"
 }
}
2021/12/03 16:30:18 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:30:23 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 16:30:23 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 16:30:25 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:30:30 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 16:30:30 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:30:35 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 16:30:35 eventingSetup.go:125: Deploying app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/03 16:30:36 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/03 16:30:41 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/03 16:30:46 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get deployed
2021/12/03 16:30:51 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get deployed
2021/12/03 16:30:56 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get deployed
2021/12/03 16:30:56 clusterSetup.go:354: App: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:30:56 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:30:56 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 16:30:56 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/03 16:30:57 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/03 16:31:02 Rebalance progress: 12.5
2021/12/03 16:31:07 Rebalance progress: 12.5
2021/12/03 16:31:12 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:31:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:31:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 16:31:12 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/03 16:31:12 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/03 16:31:17 Rebalance progress: 12.5
2021/12/03 16:31:22 Rebalance progress: 12.5
2021/12/03 16:31:27 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:31:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:31:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 16:31:27 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/03 16:31:27 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/03 16:31:32 Rebalance progress: 12.5
2021/12/03 16:31:37 Rebalance progress: 12.5
2021/12/03 16:31:42 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:31:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:31:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 16:31:42 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/03 16:31:47 Rebalance progress: 12.5
2021/12/03 16:31:52 Rebalance progress: 12.5
2021/12/03 16:31:57 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:31:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:31:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 16:31:57 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/03 16:32:02 Rebalance progress: 12.5
2021/12/03 16:32:07 Rebalance progress: 12.5
2021/12/03 16:32:12 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:32:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:32:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 16:32:12 clusterSetup.go:253: Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/03 16:32:18 Rebalance progress: 12.5
2021/12/03 16:32:23 Rebalance progress: 12.5
2021/12/03 16:32:28 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:32:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:32:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 16:32:28 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 16:32:33 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/03 16:32:38 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/03 16:32:43 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/03 16:32:48 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/03 16:32:48 clusterSetup.go:405: App: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings got un-deployed
2021/12/03 16:32:48 eventingSetup.go:46: No eventing-con process running
2021/12/03 16:32:48 eventingSetup.go:418: Function: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings deleting in the background"
}
2021/12/03 16:32:50 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:32:55 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 16:32:55 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 16:32:56 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:33:01 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 16:33:06 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 10618
2021/12/03 16:33:06 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings (179.81s)
=== RUN   TestEventingSwapRebOnUpdateBucketOp
2021/12/03 16:33:16 clusterSetup.go:397: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get un-deployed
2021/12/03 16:33:16 clusterSetup.go:405: App: TestEventingSwapRebOnUpdateBucketOp got un-deployed
2021/12/03 16:33:16 eventingSetup.go:46: No eventing-con process running
2021/12/03 16:33:16 eventingSetup.go:418: Function: TestEventingSwapRebOnUpdateBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingSwapRebOnUpdateBucketOp not found"
 }
}
2021/12/03 16:33:18 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:33:23 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 16:33:23 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 16:33:25 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:33:30 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 16:33:30 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:33:30 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 16:33:30 eventingSetup.go:125: Deploying app: TestEventingSwapRebOnUpdateBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/03 16:33:31 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingSwapRebOnUpdateBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/03 16:33:36 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/03 16:33:41 clusterSetup.go:346: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get deployed
2021/12/03 16:33:46 clusterSetup.go:346: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get deployed
2021/12/03 16:33:51 clusterSetup.go:346: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get deployed
2021/12/03 16:33:51 clusterSetup.go:354: App: TestEventingSwapRebOnUpdateBucketOp got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:33:51 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:33:51 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 16:33:51 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/03 16:33:51 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/03 16:33:57 Rebalance progress: 12.5
2021/12/03 16:34:02 Rebalance progress: 12.5
2021/12/03 16:34:07 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:34:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:34:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 16:34:07 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/03 16:34:07 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/03 16:34:12 Rebalance progress: 12.5
2021/12/03 16:34:17 Rebalance progress: 12.5
2021/12/03 16:34:22 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:34:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:34:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 16:34:22 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/03 16:34:27 Rebalance progress: 12.5
2021/12/03 16:34:32 Rebalance progress: 12.5
2021/12/03 16:34:37 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:34:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:34:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 16:34:37 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 16:34:42 clusterSetup.go:397: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get un-deployed
2021/12/03 16:34:47 clusterSetup.go:397: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get un-deployed
2021/12/03 16:34:52 clusterSetup.go:397: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get un-deployed
2021/12/03 16:34:57 clusterSetup.go:397: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get un-deployed
2021/12/03 16:34:57 clusterSetup.go:405: App: TestEventingSwapRebOnUpdateBucketOp got un-deployed
2021/12/03 16:34:57 eventingSetup.go:46: No eventing-con process running
2021/12/03 16:34:57 eventingSetup.go:418: Function: TestEventingSwapRebOnUpdateBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingSwapRebOnUpdateBucketOp deleting in the background"
}
2021/12/03 16:34:59 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:35:04 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 16:35:09 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6093
2021/12/03 16:35:09 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 16:35:11 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:35:16 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 16:35:21 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5148
2021/12/03 16:35:26 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5148
2021/12/03 16:35:26 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingSwapRebOnUpdateBucketOp (139.53s)
=== RUN   TestMetaRollbackWithEventingReb
2021/12/03 16:35:31 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/03 16:35:32 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/03 16:35:37 Rebalance progress: 100
2021/12/03 16:35:42 Waiting for app: TestMetaRollbackWithEventingReb to get un-deployed
2021/12/03 16:35:42 App: TestMetaRollbackWithEventingReb got un-deployed
2021/12/03 16:35:42 No eventing-con process running
2021/12/03 16:35:42 Function: TestMetaRollbackWithEventingReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMetaRollbackWithEventingReb not found"
 }
}
2021/12/03 16:35:43 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:35:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 16:35:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 16:35:50 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:35:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 16:35:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:35:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 16:35:55 Deploying app: TestMetaRollbackWithEventingReb. Item count src bucket: 0 dst bucket: 0
2021/12/03 16:35:56 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMetaRollbackWithEventingReb' in metakv",
  "warnings": null
 }
}
2021/12/03 16:36:02 Starting bucket ops to source bucket
2021/12/03 16:36:07 Waiting for app: TestMetaRollbackWithEventingReb to get deployed
2021/12/03 16:36:12 Waiting for app: TestMetaRollbackWithEventingReb to get deployed
2021/12/03 16:36:17 Waiting for app: TestMetaRollbackWithEventingReb to get deployed
2021/12/03 16:36:17 App: TestMetaRollbackWithEventingReb got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:36:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:36:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 16:36:17 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/03 16:36:17 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/03 16:36:22 Rebalance progress: 12.5
2021/12/03 16:36:27 Rebalance progress: 12.5
2021/12/03 16:36:32 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:36:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:36:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 16:36:32 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002] from the cluster
2021/12/03 16:36:32 bucketOps.go:134: Metakv lookup failed, err: Get "/_metakv/eventing/tempApps/TestMetaRollbackWithEventingReb/0": Unable to initialize cbauth's revrpc: cbauth environment variable CBAUTH_REVRPC_URL is not set
2021/12/03 16:36:37 Rebalance progress: 12.5
2021/12/03 16:36:42 Rebalance progress: 12.5
2021/12/03 16:36:47 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:36:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:36:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 16:36:47 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 16:36:52 clusterSetup.go:397: Waiting for app: TestMetaRollbackWithEventingReb to get un-deployed
2021/12/03 16:36:57 clusterSetup.go:397: Waiting for app: TestMetaRollbackWithEventingReb to get un-deployed
2021/12/03 16:37:02 clusterSetup.go:397: Waiting for app: TestMetaRollbackWithEventingReb to get un-deployed
2021/12/03 16:37:07 clusterSetup.go:397: Waiting for app: TestMetaRollbackWithEventingReb to get un-deployed
2021/12/03 16:37:07 clusterSetup.go:405: App: TestMetaRollbackWithEventingReb got un-deployed
2021/12/03 16:37:07 eventingSetup.go:46: No eventing-con process running
2021/12/03 16:37:07 eventingSetup.go:418: Function: TestMetaRollbackWithEventingReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMetaRollbackWithEventingReb deleting in the background"
}
2021/12/03 16:37:09 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:37:14 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 16:37:19 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4538
2021/12/03 16:37:19 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 16:37:21 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:37:26 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 16:37:31 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3556
2021/12/03 16:37:36 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3556
2021/12/03 16:37:36 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestMetaRollbackWithEventingReb (129.93s)
=== RUN   TestMetaPartialRollbackWithEventingReb
2021/12/03 16:37:41 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/03 16:37:41 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/03 16:37:46 Rebalance progress: 100
2021/12/03 16:37:51 Waiting for app: TestMetaPartialRollbackWithEventingReb to get un-deployed
2021/12/03 16:37:51 App: TestMetaPartialRollbackWithEventingReb got un-deployed
2021/12/03 16:37:51 No eventing-con process running
2021/12/03 16:37:51 Function: TestMetaPartialRollbackWithEventingReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMetaPartialRollbackWithEventingReb not found"
 }
}
2021/12/03 16:37:53 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:37:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 16:37:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 16:38:00 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:38:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 16:38:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:38:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 16:38:05 Deploying app: TestMetaPartialRollbackWithEventingReb. Item count src bucket: 0 dst bucket: 0
2021/12/03 16:38:06 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMetaPartialRollbackWithEventingReb' in metakv",
  "warnings": null
 }
}
2021/12/03 16:38:11 Starting bucket ops to source bucket
2021/12/03 16:38:16 Waiting for app: TestMetaPartialRollbackWithEventingReb to get deployed
2021/12/03 16:38:21 Waiting for app: TestMetaPartialRollbackWithEventingReb to get deployed
2021/12/03 16:38:26 Waiting for app: TestMetaPartialRollbackWithEventingReb to get deployed
2021/12/03 16:38:26 App: TestMetaPartialRollbackWithEventingReb got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:38:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:38:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 16:38:26 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/03 16:38:27 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/03 16:38:32 Rebalance progress: 12.5
2021/12/03 16:38:37 Rebalance progress: 12.5
2021/12/03 16:38:42 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:38:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:38:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 16:38:42 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002] from the cluster
2021/12/03 16:38:42 bucketOps.go:65: Metakv lookup failed, err: Get "/_metakv/eventing/tempApps/TestMetaPartialRollbackWithEventingReb/0": Unable to initialize cbauth's revrpc: cbauth environment variable CBAUTH_REVRPC_URL is not set
2021/12/03 16:38:47 Rebalance progress: 12.5
2021/12/03 16:38:52 Rebalance progress: 12.5
2021/12/03 16:38:57 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:38:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:38:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 16:38:57 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 16:38:57 bucketOps.go:65: Metakv lookup failed, err: Get "/_metakv/eventing/tempApps/TestMetaPartialRollbackWithEventingReb/0": Unable to initialize cbauth's revrpc: cbauth environment variable CBAUTH_REVRPC_URL is not set
2021/12/03 16:39:02 clusterSetup.go:397: Waiting for app: TestMetaPartialRollbackWithEventingReb to get un-deployed
2021/12/03 16:39:07 clusterSetup.go:397: Waiting for app: TestMetaPartialRollbackWithEventingReb to get un-deployed
2021/12/03 16:39:12 clusterSetup.go:397: Waiting for app: TestMetaPartialRollbackWithEventingReb to get un-deployed
2021/12/03 16:39:17 clusterSetup.go:397: Waiting for app: TestMetaPartialRollbackWithEventingReb to get un-deployed
2021/12/03 16:39:17 clusterSetup.go:405: App: TestMetaPartialRollbackWithEventingReb got un-deployed
2021/12/03 16:39:17 eventingSetup.go:46: No eventing-con process running
2021/12/03 16:39:17 eventingSetup.go:418: Function: TestMetaPartialRollbackWithEventingReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMetaPartialRollbackWithEventingReb deleting in the background"
}
2021/12/03 16:39:19 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:39:24 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 16:39:29 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 37883
2021/12/03 16:39:29 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 16:39:31 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:39:36 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 16:39:41 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 24772
2021/12/03 16:39:46 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 24772
2021/12/03 16:39:46 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestMetaPartialRollbackWithEventingReb (129.76s)
=== RUN   TestEventingRebMultipleHandlersOneByOne
2021/12/03 16:39:56 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get un-deployed
2021/12/03 16:39:56 clusterSetup.go:405: App: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op got un-deployed
2021/12/03 16:39:56 eventingSetup.go:46: No eventing-con process running
2021/12/03 16:39:56 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op not found"
 }
}
2021/12/03 16:39:57 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:40:02 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 16:40:02 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 16:40:04 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:40:09 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 16:40:09 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:40:14 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 16:40:14 eventingSetup.go:125: Deploying app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op. Item count src bucket: 0 dst bucket: 0
2021/12/03 16:40:15 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op' in metakv",
  "warnings": null
 }
}
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:40:15 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 16:40:15 eventingSetup.go:125: Deploying app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/03 16:40:16 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebMultipleHandlersOneByOnebucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/03 16:40:21 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/03 16:40:26 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get deployed
2021/12/03 16:40:32 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get deployed
2021/12/03 16:40:37 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get deployed
2021/12/03 16:40:37 clusterSetup.go:354: App: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op got deployed
2021/12/03 16:40:42 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get deployed
2021/12/03 16:40:47 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get deployed
2021/12/03 16:40:52 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get deployed
2021/12/03 16:40:52 clusterSetup.go:354: App: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:40:52 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:40:52 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 16:40:52 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/03 16:40:52 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/03 16:40:57 Rebalance progress: 12.5
2021/12/03 16:41:02 Rebalance progress: 12.5
2021/12/03 16:41:07 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:41:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:41:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 16:41:07 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/03 16:41:07 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/03 16:41:12 Rebalance progress: 12.5
2021/12/03 16:41:17 Rebalance progress: 12.5
2021/12/03 16:41:22 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:41:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:41:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 16:41:23 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/03 16:41:23 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/03 16:41:28 Rebalance progress: 12.5
2021/12/03 16:41:33 Rebalance progress: 12.5
2021/12/03 16:41:38 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:41:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:41:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 16:41:38 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/03 16:41:43 Rebalance progress: 12.5
2021/12/03 16:41:48 Rebalance progress: 12.5
2021/12/03 16:41:53 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:41:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:41:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 16:41:53 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/03 16:41:58 Rebalance progress: 12.5
2021/12/03 16:42:03 Rebalance progress: 12.5
2021/12/03 16:42:08 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:42:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:42:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 16:42:08 clusterSetup.go:253: Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/03 16:42:13 Rebalance progress: 12.5
2021/12/03 16:42:18 Rebalance progress: 12.5
2021/12/03 16:42:23 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:42:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:42:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 16:42:23 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 16:42:28 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get un-deployed
2021/12/03 16:42:33 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get un-deployed
2021/12/03 16:42:38 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get un-deployed
2021/12/03 16:42:43 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get un-deployed
2021/12/03 16:42:43 clusterSetup.go:405: App: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op got un-deployed
2021/12/03 16:42:43 eventingSetup.go:46: No eventing-con process running
2021/12/03 16:42:43 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op deleting in the background"
}
2021/12/03 16:42:47 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:42:52 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 16:42:57 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12165
2021/12/03 16:42:57 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 16:42:59 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:43:04 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 16:43:09 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12165
2021/12/03 16:43:09 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 16:43:09 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 16:43:14 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get un-deployed
2021/12/03 16:43:19 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get un-deployed
2021/12/03 16:43:24 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get un-deployed
2021/12/03 16:43:29 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get un-deployed
2021/12/03 16:43:29 clusterSetup.go:405: App: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update got un-deployed
2021/12/03 16:43:29 eventingSetup.go:46: No eventing-con process running
2021/12/03 16:43:29 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update deleting in the background"
}
2021/12/03 16:43:31 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:43:36 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 16:43:36 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 16:43:38 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:43:43 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 16:43:43 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebMultipleHandlersOneByOne (237.11s)
=== RUN   TestEventingRebMultipleHandlersAllAtOnce
2021/12/03 16:43:53 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get un-deployed
2021/12/03 16:43:53 clusterSetup.go:405: App: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op got un-deployed
2021/12/03 16:43:53 eventingSetup.go:46: No eventing-con process running
2021/12/03 16:43:53 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op not found"
 }
}
2021/12/03 16:43:55 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:44:00 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 16:44:00 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 16:44:01 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:44:06 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 16:44:06 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:44:11 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 16:44:11 eventingSetup.go:125: Deploying app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op. Item count src bucket: 0 dst bucket: 0
2021/12/03 16:44:12 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op' in metakv",
  "warnings": null
 }
}
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:44:12 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 16:44:12 eventingSetup.go:125: Deploying app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/03 16:44:14 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/03 16:44:19 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/03 16:44:24 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get deployed
2021/12/03 16:44:29 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get deployed
2021/12/03 16:44:34 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get deployed
2021/12/03 16:44:34 clusterSetup.go:354: App: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op got deployed
2021/12/03 16:44:39 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get deployed
2021/12/03 16:44:44 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get deployed
2021/12/03 16:44:49 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get deployed
2021/12/03 16:44:49 clusterSetup.go:354: App: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:44:49 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:44:49 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 16:44:49 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/03 16:44:49 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/03 16:44:49 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/03 16:44:50 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/03 16:44:55 Rebalance progress: 12.5
2021/12/03 16:45:00 Rebalance progress: 12.5
2021/12/03 16:45:05 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:45:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:45:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 16:45:05 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002 127.0.0.1:19003] from the cluster
2021/12/03 16:45:10 Rebalance progress: 12.5
2021/12/03 16:45:15 Rebalance progress: 12.5
2021/12/03 16:45:20 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:45:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:45:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 16:45:20 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 16:45:25 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get un-deployed
2021/12/03 16:45:30 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get un-deployed
2021/12/03 16:45:35 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get un-deployed
2021/12/03 16:45:40 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get un-deployed
2021/12/03 16:45:40 clusterSetup.go:405: App: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op got un-deployed
2021/12/03 16:45:40 eventingSetup.go:46: No eventing-con process running
2021/12/03 16:45:40 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op deleting in the background"
}
2021/12/03 16:45:44 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:45:49 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 16:45:54 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6135
2021/12/03 16:45:59 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6135
2021/12/03 16:45:59 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 16:46:01 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:46:06 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 16:46:11 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5124
2021/12/03 16:46:16 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5124
2021/12/03 16:46:16 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 16:46:16 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 16:46:21 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get un-deployed
2021/12/03 16:46:26 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get un-deployed
2021/12/03 16:46:31 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get un-deployed
2021/12/03 16:46:36 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get un-deployed
2021/12/03 16:46:36 clusterSetup.go:405: App: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update got un-deployed
2021/12/03 16:46:36 eventingSetup.go:46: No eventing-con process running
2021/12/03 16:46:36 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update deleting in the background"
}
2021/12/03 16:46:38 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:46:43 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 16:46:43 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 16:46:45 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:46:50 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 16:46:50 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebMultipleHandlersAllAtOnce (187.01s)
=== RUN   TestEventingSwapRebMultipleHandlers
2021/12/03 16:47:00 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get un-deployed
2021/12/03 16:47:00 clusterSetup.go:405: App: TestEventingSwapRebMultipleHandlersbucket_op_on_update got un-deployed
2021/12/03 16:47:00 eventingSetup.go:46: No eventing-con process running
2021/12/03 16:47:00 eventingSetup.go:418: Function: TestEventingSwapRebMultipleHandlersbucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingSwapRebMultipleHandlersbucket_op_on_update not found"
 }
}
2021/12/03 16:47:02 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:47:07 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 16:47:07 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 16:47:08 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:47:13 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 16:47:13 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 16:47:18 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get un-deployed
2021/12/03 16:47:18 clusterSetup.go:405: App: TestEventingSwapRebMultipleHandlerssys_test_bucket_op got un-deployed
2021/12/03 16:47:18 eventingSetup.go:46: No eventing-con process running
2021/12/03 16:47:18 eventingSetup.go:418: Function: TestEventingSwapRebMultipleHandlerssys_test_bucket_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingSwapRebMultipleHandlerssys_test_bucket_op not found"
 }
}
2021/12/03 16:47:20 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:47:25 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 16:47:25 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 16:47:27 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:47:32 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 16:47:32 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:47:32 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 16:47:32 eventingSetup.go:125: Deploying app: TestEventingSwapRebMultipleHandlersbucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/03 16:47:33 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingSwapRebMultipleHandlersbucket_op_on_update' in metakv",
  "warnings": null
 }
}
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:47:33 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 16:47:33 eventingSetup.go:125: Deploying app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op. Item count src bucket: 0 dst bucket: 0
2021/12/03 16:47:34 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingSwapRebMultipleHandlerssys_test_bucket_op' in metakv",
  "warnings": null
 }
}
2021/12/03 16:47:39 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/03 16:47:44 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get deployed
2021/12/03 16:47:49 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get deployed
2021/12/03 16:47:54 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get deployed
2021/12/03 16:47:54 clusterSetup.go:354: App: TestEventingSwapRebMultipleHandlersbucket_op_on_update got deployed
2021/12/03 16:47:59 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get deployed
2021/12/03 16:48:04 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get deployed
2021/12/03 16:48:09 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get deployed
2021/12/03 16:48:09 clusterSetup.go:354: App: TestEventingSwapRebMultipleHandlerssys_test_bucket_op got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:48:09 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:48:09 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 16:48:09 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/03 16:48:10 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/03 16:48:15 Rebalance progress: 12.5
2021/12/03 16:48:20 Rebalance progress: 12.5
2021/12/03 16:48:25 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:48:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:48:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 16:48:25 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/03 16:48:25 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/03 16:48:30 Rebalance progress: 12.5
2021/12/03 16:48:35 Rebalance progress: 12.5
2021/12/03 16:48:40 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:48:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:48:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 16:48:40 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/03 16:48:45 Rebalance progress: 12.5
2021/12/03 16:48:50 Rebalance progress: 12.5
2021/12/03 16:48:55 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:48:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:48:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 16:48:55 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 16:49:00 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get un-deployed
2021/12/03 16:49:05 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get un-deployed
2021/12/03 16:49:10 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get un-deployed
2021/12/03 16:49:15 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get un-deployed
2021/12/03 16:49:15 clusterSetup.go:405: App: TestEventingSwapRebMultipleHandlersbucket_op_on_update got un-deployed
2021/12/03 16:49:15 eventingSetup.go:46: No eventing-con process running
2021/12/03 16:49:15 eventingSetup.go:418: Function: TestEventingSwapRebMultipleHandlersbucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingSwapRebMultipleHandlersbucket_op_on_update deleting in the background"
}
2021/12/03 16:49:19 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:49:24 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 16:49:29 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7592
2021/12/03 16:49:29 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 16:49:31 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:49:36 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 16:49:41 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6948
2021/12/03 16:49:46 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6948
2021/12/03 16:49:46 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 16:49:46 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 16:49:51 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get un-deployed
2021/12/03 16:49:56 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get un-deployed
2021/12/03 16:50:01 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get un-deployed
2021/12/03 16:50:06 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get un-deployed
2021/12/03 16:50:06 clusterSetup.go:405: App: TestEventingSwapRebMultipleHandlerssys_test_bucket_op got un-deployed
2021/12/03 16:50:06 eventingSetup.go:46: No eventing-con process running
2021/12/03 16:50:06 eventingSetup.go:418: Function: TestEventingSwapRebMultipleHandlerssys_test_bucket_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingSwapRebMultipleHandlerssys_test_bucket_op deleting in the background"
}
2021/12/03 16:50:08 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:50:13 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 16:50:13 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 16:50:15 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:50:20 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 16:50:20 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingSwapRebMultipleHandlers (210.14s)
=== RUN   TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne
2021/12/03 16:50:30 clusterSetup.go:397: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/03 16:50:30 clusterSetup.go:405: App: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/03 16:50:30 eventingSetup.go:46: No eventing-con process running
2021/12/03 16:50:30 eventingSetup.go:418: Function: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne not found"
 }
}
2021/12/03 16:50:32 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:50:37 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 16:50:37 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 16:50:38 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:50:43 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 16:50:43 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:50:48 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 16:50:48 eventingSetup.go:125: Deploying app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne. Item count src bucket: 0 dst bucket: 0
2021/12/03 16:50:50 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne' in metakv",
  "warnings": null
 }
}
2021/12/03 16:50:55 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/03 16:51:00 clusterSetup.go:346: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/03 16:51:05 clusterSetup.go:346: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/03 16:51:10 clusterSetup.go:346: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/03 16:51:10 clusterSetup.go:354: App: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:51:10 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:51:10 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 16:51:10 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/03 16:51:10 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/03 16:51:30 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:51:31 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:51:31 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:52:01 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:52:01 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 16:52:01 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/03 16:52:06 Rebalance progress: 12.5
2021/12/03 16:52:11 Rebalance progress: 12.5
2021/12/03 16:52:16 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:52:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:52:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 16:52:16 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/03 16:52:16 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/03 16:52:21 Rebalance progress: 12.5
2021/12/03 16:52:26 Rebalance progress: 12.5
2021/12/03 16:52:31 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:52:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:52:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 16:52:31 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/03 16:52:37 Rebalance progress: 12.5
2021/12/03 16:52:42 Rebalance progress: 12.5
2021/12/03 16:52:47 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:52:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:52:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 16:52:47 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 16:52:52 clusterSetup.go:397: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/03 16:52:57 clusterSetup.go:397: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/03 16:53:02 clusterSetup.go:397: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/03 16:53:07 clusterSetup.go:397: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/03 16:53:07 clusterSetup.go:405: App: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/03 16:53:07 eventingSetup.go:46: No eventing-con process running
2021/12/03 16:53:07 eventingSetup.go:418: Function: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne deleting in the background"
}
2021/12/03 16:53:09 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:53:14 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 16:53:19 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 11182
2021/12/03 16:53:19 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 16:53:20 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:53:25 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 16:53:30 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4850
2021/12/03 16:53:35 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4850
2021/12/03 16:53:35 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne (195.53s)
=== RUN   TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne
2021/12/03 16:53:45 clusterSetup.go:397: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/03 16:53:45 clusterSetup.go:405: App: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/03 16:53:45 eventingSetup.go:46: No eventing-con process running
2021/12/03 16:53:45 eventingSetup.go:418: Function: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne not found"
 }
}
2021/12/03 16:53:47 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:53:52 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 16:53:52 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 16:53:54 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:53:59 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 16:53:59 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:54:04 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 16:54:04 eventingSetup.go:125: Deploying app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne. Item count src bucket: 0 dst bucket: 0
2021/12/03 16:54:05 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne' in metakv",
  "warnings": null
 }
}
2021/12/03 16:54:10 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/03 16:54:15 clusterSetup.go:346: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/03 16:54:20 clusterSetup.go:346: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/03 16:54:25 clusterSetup.go:346: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/03 16:54:25 clusterSetup.go:354: App: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:54:25 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:54:25 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 16:54:25 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/03 16:54:25 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/03 16:54:45 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:54:46 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:54:46 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:54:56 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:54:56 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 16:54:56 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/03 16:54:56 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/03 16:55:16 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:55:17 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:55:17 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:55:27 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:55:27 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 16:55:27 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/03 16:55:27 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/03 16:55:47 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:55:48 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:55:48 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:55:58 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:55:58 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 16:55:58 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/03 16:55:58 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/03 16:56:18 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:56:19 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:56:19 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:56:29 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:56:29 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 16:56:29 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/03 16:56:29 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/03 16:56:49 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:56:50 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:56:50 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:57:00 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:57:00 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 16:57:00 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/03 16:57:05 Rebalance progress: 12.5
2021/12/03 16:57:10 Rebalance progress: 12.5
2021/12/03 16:57:15 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:57:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:57:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 16:57:15 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/03 16:57:15 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/03 16:57:20 Rebalance progress: 12.5
2021/12/03 16:57:25 Rebalance progress: 12.5
2021/12/03 16:57:30 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:57:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:57:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 16:57:30 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/03 16:57:35 Rebalance progress: 12.5
2021/12/03 16:57:40 Rebalance progress: 12.5
2021/12/03 16:57:45 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:57:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:57:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 16:57:45 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 16:57:50 clusterSetup.go:397: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/03 16:57:56 clusterSetup.go:397: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/03 16:58:01 clusterSetup.go:397: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/03 16:58:06 clusterSetup.go:397: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/03 16:58:06 clusterSetup.go:405: App: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/03 16:58:06 eventingSetup.go:46: No eventing-con process running
2021/12/03 16:58:06 eventingSetup.go:418: Function: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne deleting in the background"
}
2021/12/03 16:58:07 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:58:12 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 16:58:17 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 21516
2021/12/03 16:58:17 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 16:58:19 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:58:24 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 16:58:29 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 21516
2021/12/03 16:58:29 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne (293.88s)
=== RUN   TestEventingFailoverOnUpdateBucketOp
2021/12/03 16:58:39 clusterSetup.go:397: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get un-deployed
2021/12/03 16:58:39 clusterSetup.go:405: App: TestEventingFailoverOnUpdateBucketOp got un-deployed
2021/12/03 16:58:39 eventingSetup.go:46: No eventing-con process running
2021/12/03 16:58:39 eventingSetup.go:418: Function: TestEventingFailoverOnUpdateBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingFailoverOnUpdateBucketOp not found"
 }
}
2021/12/03 16:58:41 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:58:46 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 16:58:46 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 16:58:48 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:58:53 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 16:58:53 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:58:58 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 16:58:58 eventingSetup.go:125: Deploying app: TestEventingFailoverOnUpdateBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/03 16:58:59 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingFailoverOnUpdateBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/03 16:59:04 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/03 16:59:09 clusterSetup.go:346: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get deployed
2021/12/03 16:59:14 clusterSetup.go:346: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get deployed
2021/12/03 16:59:19 clusterSetup.go:346: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get deployed
2021/12/03 16:59:19 clusterSetup.go:354: App: TestEventingFailoverOnUpdateBucketOp got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:59:19 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:59:19 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 16:59:19 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/03 16:59:19 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/03 16:59:24 Rebalance progress: 12.5
2021/12/03 16:59:29 Rebalance progress: 12.5
2021/12/03 16:59:34 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:59:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:59:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 16:59:34 clusterSetup.go:229: Failing over: [127.0.0.1:19001]
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:59:44 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:59:44 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 16:59:44 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/03 16:59:49 Rebalance progress: 12.5
2021/12/03 16:59:54 Rebalance progress: 12.5
2021/12/03 16:59:59 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:59:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 16:59:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 17:00:00 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/03 17:00:00 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/03 17:00:05 Rebalance progress: 12.5
2021/12/03 17:00:10 Rebalance progress: 12.5
2021/12/03 17:00:15 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 17:00:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 17:00:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 17:00:15 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/03 17:00:20 Rebalance progress: 12.5
2021/12/03 17:00:25 Rebalance progress: 12.5
2021/12/03 17:00:30 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 17:00:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 17:00:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 17:00:30 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 17:00:35 clusterSetup.go:397: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get un-deployed
2021/12/03 17:00:40 clusterSetup.go:397: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get un-deployed
2021/12/03 17:00:45 clusterSetup.go:397: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get un-deployed
2021/12/03 17:00:50 clusterSetup.go:397: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get un-deployed
2021/12/03 17:00:50 clusterSetup.go:405: App: TestEventingFailoverOnUpdateBucketOp got un-deployed
2021/12/03 17:00:50 eventingSetup.go:46: No eventing-con process running
2021/12/03 17:00:50 eventingSetup.go:418: Function: TestEventingFailoverOnUpdateBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingFailoverOnUpdateBucketOp deleting in the background"
}
2021/12/03 17:00:52 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 17:00:57 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 17:01:02 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 8586
2021/12/03 17:01:07 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 8586
2021/12/03 17:01:07 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 17:01:09 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 17:01:14 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 17:01:19 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 8586
2021/12/03 17:01:19 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingFailoverOnUpdateBucketOp (169.42s)
=== RUN   TestEventingKVRebalanceOnUpdateBucketOp
2021/12/03 17:01:29 clusterSetup.go:397: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get un-deployed
2021/12/03 17:01:29 clusterSetup.go:405: App: TestEventingKVRebalanceOnUpdateBucketOp got un-deployed
2021/12/03 17:01:29 eventingSetup.go:46: No eventing-con process running
2021/12/03 17:01:29 eventingSetup.go:418: Function: TestEventingKVRebalanceOnUpdateBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingKVRebalanceOnUpdateBucketOp not found"
 }
}
2021/12/03 17:01:30 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 17:01:35 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 17:01:35 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 17:01:37 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 17:01:42 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 17:01:42 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 17:01:47 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 17:01:47 eventingSetup.go:125: Deploying app: TestEventingKVRebalanceOnUpdateBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/03 17:01:48 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingKVRebalanceOnUpdateBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/03 17:01:53 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/03 17:01:58 clusterSetup.go:346: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get deployed
2021/12/03 17:02:03 clusterSetup.go:346: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get deployed
2021/12/03 17:02:08 clusterSetup.go:346: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get deployed
2021/12/03 17:02:08 clusterSetup.go:354: App: TestEventingKVRebalanceOnUpdateBucketOp got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 17:02:08 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 17:02:08 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 17:02:08 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing,kv to the cluster
2021/12/03 17:02:09 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/03 17:02:14 Rebalance progress: 12.5
2021/12/03 17:02:19 Rebalance progress: 12.5
2021/12/03 17:02:24 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 17:02:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 17:02:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 17:02:24 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing,kv to the cluster
2021/12/03 17:02:24 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/03 17:02:30 Rebalance progress: 12.5
2021/12/03 17:02:35 Rebalance progress: 12.5
2021/12/03 17:02:40 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 17:02:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 17:02:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 17:02:40 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/03 17:02:45 Rebalance progress: 12.5
2021/12/03 17:02:50 Rebalance progress: 12.5
2021/12/03 17:02:55 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 17:02:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 17:02:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 17:02:55 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 17:03:00 clusterSetup.go:397: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get un-deployed
2021/12/03 17:03:05 clusterSetup.go:397: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get un-deployed
2021/12/03 17:03:10 clusterSetup.go:397: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get un-deployed
2021/12/03 17:03:15 clusterSetup.go:397: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get un-deployed
2021/12/03 17:03:15 clusterSetup.go:405: App: TestEventingKVRebalanceOnUpdateBucketOp got un-deployed
2021/12/03 17:03:15 eventingSetup.go:46: No eventing-con process running
2021/12/03 17:03:15 eventingSetup.go:418: Function: TestEventingKVRebalanceOnUpdateBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingKVRebalanceOnUpdateBucketOp deleting in the background"
}
2021/12/03 17:03:17 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 17:03:22 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 17:03:27 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6113
2021/12/03 17:03:27 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 17:03:29 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 17:03:34 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 17:03:39 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6113
2021/12/03 17:03:39 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingKVRebalanceOnUpdateBucketOp (139.86s)
=== RUN   TestEventingRebInWithOps
2021/12/03 17:03:49 clusterSetup.go:397: Waiting for app: TestEventingRebInWithOps to get un-deployed
2021/12/03 17:03:49 clusterSetup.go:405: App: TestEventingRebInWithOps got un-deployed
2021/12/03 17:03:49 eventingSetup.go:46: No eventing-con process running
2021/12/03 17:03:49 eventingSetup.go:418: Function: TestEventingRebInWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebInWithOps not found"
 }
}
2021/12/03 17:03:50 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 17:03:55 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 17:03:55 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 17:03:57 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 17:04:02 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 17:04:02 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 17:04:07 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 17:04:07 eventingSetup.go:125: Deploying app: TestEventingRebInWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/03 17:04:08 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebInWithOps' in metakv",
  "warnings": null
 }
}
2021/12/03 17:04:13 clusterSetup.go:346: Waiting for app: TestEventingRebInWithOps to get deployed
2021/12/03 17:04:18 clusterSetup.go:346: Waiting for app: TestEventingRebInWithOps to get deployed
2021/12/03 17:04:23 clusterSetup.go:346: Waiting for app: TestEventingRebInWithOps to get deployed
2021/12/03 17:04:28 clusterSetup.go:346: Waiting for app: TestEventingRebInWithOps to get deployed
2021/12/03 17:04:28 clusterSetup.go:354: App: TestEventingRebInWithOps got deployed
2021/12/03 17:04:28 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/03 17:04:28 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/03 17:04:30 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/03 17:04:35 Rebalance progress: 12.5
2021/12/03 17:04:40 Rebalance progress: 12.5
2021/12/03 17:04:45 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 17:04:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 17:04:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 17:04:45 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/03 17:04:50 Rebalance progress: 12.5
2021/12/03 17:04:55 Rebalance progress: 12.5
2021/12/03 17:05:00 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 17:05:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 17:05:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 17:05:00 Waiting for src bucket item count to get to: 200000 curr count: 30841
2021/12/03 17:05:05 Waiting for src bucket item count to get to: 200000 curr count: 46856
2021/12/03 17:05:10 Waiting for src bucket item count to get to: 200000 curr count: 46856
2021/12/03 17:05:15 Waiting for src bucket item count to get to: 200000 curr count: 62314
2021/12/03 17:05:20 Waiting for src bucket item count to get to: 200000 curr count: 62314
2021/12/03 17:05:25 Waiting for src bucket item count to get to: 200000 curr count: 77163
2021/12/03 17:05:30 Waiting for src bucket item count to get to: 200000 curr count: 77163
2021/12/03 17:05:35 Waiting for src bucket item count to get to: 200000 curr count: 92585
2021/12/03 17:05:40 Waiting for src bucket item count to get to: 200000 curr count: 92585
2021/12/03 17:05:45 Waiting for src bucket item count to get to: 200000 curr count: 108320
2021/12/03 17:05:50 Waiting for src bucket item count to get to: 200000 curr count: 108320
2021/12/03 17:05:55 Waiting for src bucket item count to get to: 200000 curr count: 124124
2021/12/03 17:06:00 Waiting for src bucket item count to get to: 200000 curr count: 124124
2021/12/03 17:06:05 Waiting for src bucket item count to get to: 200000 curr count: 139995
2021/12/03 17:06:10 Waiting for src bucket item count to get to: 200000 curr count: 139995
2021/12/03 17:06:15 Waiting for src bucket item count to get to: 200000 curr count: 155913
2021/12/03 17:06:20 Waiting for src bucket item count to get to: 200000 curr count: 155913
2021/12/03 17:06:25 Waiting for src bucket item count to get to: 200000 curr count: 170869
2021/12/03 17:06:30 Waiting for src bucket item count to get to: 200000 curr count: 186824
2021/12/03 17:06:35 Waiting for src bucket item count to get to: 200000 curr count: 186824
2021/12/03 17:06:40 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/03 17:06:40 Post SET ops. Expected item count: 200000 got 200000
2021/12/03 17:06:40 Starting bucket ops to source bucket
2021/12/03 17:10:31 Waiting for dst bucket item count to get to: 0 curr count: 12056
2021/12/03 17:10:36 Waiting for dst bucket item count to get to: 0 curr count: 12056
2021/12/03 17:10:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 17:10:36 Post DELETE ops. Expected item count: 0 got 0
2021/12/03 17:10:36 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 17:10:41 Waiting for app: TestEventingRebInWithOps to get un-deployed
2021/12/03 17:10:46 Waiting for app: TestEventingRebInWithOps to get un-deployed
2021/12/03 17:10:51 Waiting for app: TestEventingRebInWithOps to get un-deployed
2021/12/03 17:10:56 Waiting for app: TestEventingRebInWithOps to get un-deployed
2021/12/03 17:10:56 App: TestEventingRebInWithOps got un-deployed
2021/12/03 17:10:56 No eventing-con process running
2021/12/03 17:10:56 Function: TestEventingRebInWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebInWithOps deleting in the background"
}
2021/12/03 17:10:58 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 17:11:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 17:11:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 17:11:05 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 17:11:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 17:11:10 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebInWithOps (451.27s)
=== RUN   TestEventingRebOutWithOps
2021/12/03 17:11:15 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/03 17:11:15 Removing node(s): [] from the cluster
2021/12/03 17:11:20 Rebalance progress: 100
2021/12/03 17:11:25 Waiting for app: TestEventingRebOutWithOps to get un-deployed
2021/12/03 17:11:25 App: TestEventingRebOutWithOps got un-deployed
2021/12/03 17:11:25 No eventing-con process running
2021/12/03 17:11:25 Function: TestEventingRebOutWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebOutWithOps not found"
 }
}
2021/12/03 17:11:27 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 17:11:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 17:11:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 17:11:34 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 17:11:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 17:11:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 17:11:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 17:11:44 Deploying app: TestEventingRebOutWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/03 17:11:45 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebOutWithOps' in metakv",
  "warnings": null
 }
}
2021/12/03 17:11:50 Waiting for app: TestEventingRebOutWithOps to get deployed
2021/12/03 17:11:55 Waiting for app: TestEventingRebOutWithOps to get deployed
2021/12/03 17:12:00 Waiting for app: TestEventingRebOutWithOps to get deployed
2021/12/03 17:12:05 Waiting for app: TestEventingRebOutWithOps to get deployed
2021/12/03 17:12:05 App: TestEventingRebOutWithOps got deployed
2021/12/03 17:12:05 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/03 17:12:05 Starting bucket ops to source bucket
2021/12/03 17:12:10 Rebalance progress: 12.5
2021/12/03 17:12:15 Rebalance progress: 12.5
2021/12/03 17:12:20 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 17:12:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 17:12:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 17:12:20 Waiting for src bucket item count to get to: 200000 curr count: 6156
2021/12/03 17:12:25 Waiting for src bucket item count to get to: 200000 curr count: 22492
2021/12/03 17:12:30 Waiting for src bucket item count to get to: 200000 curr count: 38379
2021/12/03 17:12:35 Waiting for src bucket item count to get to: 200000 curr count: 38379
2021/12/03 17:12:40 Waiting for src bucket item count to get to: 200000 curr count: 54582
2021/12/03 17:12:45 Waiting for src bucket item count to get to: 200000 curr count: 54582
2021/12/03 17:12:50 Waiting for src bucket item count to get to: 200000 curr count: 54582
2021/12/03 17:12:55 Waiting for src bucket item count to get to: 200000 curr count: 70831
2021/12/03 17:13:00 Waiting for src bucket item count to get to: 200000 curr count: 70831
2021/12/03 17:13:05 Waiting for src bucket item count to get to: 200000 curr count: 86038
2021/12/03 17:13:10 Waiting for src bucket item count to get to: 200000 curr count: 86038
2021/12/03 17:13:15 Waiting for src bucket item count to get to: 200000 curr count: 102025
2021/12/03 17:13:20 Waiting for src bucket item count to get to: 200000 curr count: 102025
2021/12/03 17:13:25 Waiting for src bucket item count to get to: 200000 curr count: 118325
2021/12/03 17:13:30 Waiting for src bucket item count to get to: 200000 curr count: 118325
2021/12/03 17:13:35 Waiting for src bucket item count to get to: 200000 curr count: 134187
2021/12/03 17:13:40 Waiting for src bucket item count to get to: 200000 curr count: 134187
2021/12/03 17:13:45 Waiting for src bucket item count to get to: 200000 curr count: 150257
2021/12/03 17:13:50 Waiting for src bucket item count to get to: 200000 curr count: 150257
2021/12/03 17:13:55 Waiting for src bucket item count to get to: 200000 curr count: 166381
2021/12/03 17:14:00 Waiting for src bucket item count to get to: 200000 curr count: 166381
2021/12/03 17:14:05 Waiting for src bucket item count to get to: 200000 curr count: 181433
2021/12/03 17:14:10 Waiting for src bucket item count to get to: 200000 curr count: 181433
2021/12/03 17:14:15 Waiting for src bucket item count to get to: 200000 curr count: 197240
2021/12/03 17:14:20 Waiting for src bucket item count to get to: 200000 curr count: 197240
2021/12/03 17:14:25 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/03 17:14:25 Post SET ops. Expected item count: 200000 got 200000
2021/12/03 17:14:25 Starting bucket ops to source bucket
2021/12/03 17:18:17 Waiting for dst bucket item count to get to: 0 curr count: 4243
2021/12/03 17:18:22 Waiting for dst bucket item count to get to: 0 curr count: 4243
2021/12/03 17:18:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 17:18:22 Post DELETE ops. Expected item count: 0 got 0
2021/12/03 17:18:22 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 17:18:27 Waiting for app: TestEventingRebOutWithOps to get un-deployed
2021/12/03 17:18:32 Waiting for app: TestEventingRebOutWithOps to get un-deployed
2021/12/03 17:18:37 Waiting for app: TestEventingRebOutWithOps to get un-deployed
2021/12/03 17:18:42 Waiting for app: TestEventingRebOutWithOps to get un-deployed
2021/12/03 17:18:42 App: TestEventingRebOutWithOps got un-deployed
2021/12/03 17:18:42 No eventing-con process running
2021/12/03 17:18:42 Function: TestEventingRebOutWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebOutWithOps deleting in the background"
}
2021/12/03 17:18:44 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 17:18:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 17:18:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 17:18:50 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 17:18:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 17:18:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebOutWithOps (465.66s)
=== RUN   TestEventingSwapRebWithOps
2021/12/03 17:19:05 Waiting for app: TestEventingSwapRebWithOps to get un-deployed
2021/12/03 17:19:05 App: TestEventingSwapRebWithOps got un-deployed
2021/12/03 17:19:05 No eventing-con process running
2021/12/03 17:19:05 Function: TestEventingSwapRebWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingSwapRebWithOps not found"
 }
}
2021/12/03 17:19:07 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 17:19:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 17:19:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 17:19:14 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 17:19:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 17:19:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 17:19:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 17:19:24 Deploying app: TestEventingSwapRebWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/03 17:19:25 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingSwapRebWithOps' in metakv",
  "warnings": null
 }
}
2021/12/03 17:19:30 Waiting for app: TestEventingSwapRebWithOps to get deployed
2021/12/03 17:19:35 Waiting for app: TestEventingSwapRebWithOps to get deployed
2021/12/03 17:19:40 Waiting for app: TestEventingSwapRebWithOps to get deployed
2021/12/03 17:19:45 Waiting for app: TestEventingSwapRebWithOps to get deployed
2021/12/03 17:19:45 App: TestEventingSwapRebWithOps got deployed
2021/12/03 17:19:45 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/03 17:19:45 Starting bucket ops to source bucket
2021/12/03 17:19:46 Removing node(s): [] from the cluster
2021/12/03 17:19:51 Rebalance progress: 12.5
2021/12/03 17:19:56 Rebalance progress: 12.5
2021/12/03 17:20:01 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 17:20:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 17:20:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 17:20:01 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/03 17:20:01 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/03 17:20:06 Rebalance progress: 12.5
2021/12/03 17:20:11 Rebalance progress: 12.5
2021/12/03 17:20:16 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 17:20:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 17:20:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 17:20:16 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/03 17:20:21 Rebalance progress: 12.5
2021/12/03 17:20:26 Rebalance progress: 12.5
2021/12/03 17:20:31 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 17:20:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 17:20:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 17:20:31 Waiting for src bucket item count to get to: 200000 curr count: 67029
2021/12/03 17:20:36 Waiting for src bucket item count to get to: 200000 curr count: 67029
2021/12/03 17:20:41 Waiting for src bucket item count to get to: 200000 curr count: 81845
2021/12/03 17:20:46 Waiting for src bucket item count to get to: 200000 curr count: 81845
2021/12/03 17:20:51 Waiting for src bucket item count to get to: 200000 curr count: 97742
2021/12/03 17:20:56 Waiting for src bucket item count to get to: 200000 curr count: 97742
2021/12/03 17:21:01 Waiting for src bucket item count to get to: 200000 curr count: 113595
2021/12/03 17:21:06 Waiting for src bucket item count to get to: 200000 curr count: 113595
2021/12/03 17:21:11 Waiting for src bucket item count to get to: 200000 curr count: 129691
2021/12/03 17:21:16 Waiting for src bucket item count to get to: 200000 curr count: 129691
2021/12/03 17:21:21 Waiting for src bucket item count to get to: 200000 curr count: 145733
2021/12/03 17:21:26 Waiting for src bucket item count to get to: 200000 curr count: 145733
2021/12/03 17:21:31 Waiting for src bucket item count to get to: 200000 curr count: 145733
2021/12/03 17:21:36 Waiting for src bucket item count to get to: 200000 curr count: 161840
2021/12/03 17:21:41 Waiting for src bucket item count to get to: 200000 curr count: 161840
2021/12/03 17:21:46 Waiting for src bucket item count to get to: 200000 curr count: 177124
2021/12/03 17:21:51 Waiting for src bucket item count to get to: 200000 curr count: 177124
2021/12/03 17:21:56 Waiting for src bucket item count to get to: 200000 curr count: 193285
2021/12/03 17:22:01 Waiting for src bucket item count to get to: 200000 curr count: 193285
2021/12/03 17:22:06 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/03 17:22:06 Post SET ops. Expected item count: 200000 got 200000
2021/12/03 17:22:06 Starting bucket ops to source bucket
2021/12/03 17:25:58 Waiting for dst bucket item count to get to: 0 curr count: 7060
2021/12/03 17:26:03 Waiting for dst bucket item count to get to: 0 curr count: 7060
2021/12/03 17:26:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 17:26:03 Post DELETE ops. Expected item count: 0 got 0
2021/12/03 17:26:03 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 17:26:08 Waiting for app: TestEventingSwapRebWithOps to get un-deployed
2021/12/03 17:26:13 Waiting for app: TestEventingSwapRebWithOps to get un-deployed
2021/12/03 17:26:18 Waiting for app: TestEventingSwapRebWithOps to get un-deployed
2021/12/03 17:26:23 Waiting for app: TestEventingSwapRebWithOps to get un-deployed
2021/12/03 17:26:23 App: TestEventingSwapRebWithOps got un-deployed
2021/12/03 17:26:23 No eventing-con process running
2021/12/03 17:26:23 Function: TestEventingSwapRebWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingSwapRebWithOps deleting in the background"
}
2021/12/03 17:26:25 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 17:26:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 17:26:30 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 17:26:32 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 17:26:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 17:26:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingSwapRebWithOps (461.52s)
=== RUN   TestKVRebInWithOps
2021/12/03 17:26:47 Waiting for app: TestKVRebInWithOps to get un-deployed
2021/12/03 17:26:47 App: TestKVRebInWithOps got un-deployed
2021/12/03 17:26:47 No eventing-con process running
2021/12/03 17:26:47 Function: TestKVRebInWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVRebInWithOps not found"
 }
}
2021/12/03 17:26:49 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 17:26:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 17:26:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 17:26:56 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 17:27:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 17:27:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 17:27:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 17:27:06 Deploying app: TestKVRebInWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/03 17:27:07 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVRebInWithOps' in metakv",
  "warnings": null
 }
}
2021/12/03 17:27:12 Waiting for app: TestKVRebInWithOps to get deployed
2021/12/03 17:27:17 Waiting for app: TestKVRebInWithOps to get deployed
2021/12/03 17:27:22 Waiting for app: TestKVRebInWithOps to get deployed
2021/12/03 17:27:27 Waiting for app: TestKVRebInWithOps to get deployed
2021/12/03 17:27:27 App: TestKVRebInWithOps got deployed
2021/12/03 17:27:27 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/03 17:27:27 Starting bucket ops to source bucket
2021/12/03 17:27:27 Removing node(s): [] from the cluster
2021/12/03 17:27:32 Rebalance progress: 12.5
2021/12/03 17:27:37 Rebalance progress: 12.5
2021/12/03 17:27:42 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 17:27:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 17:27:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 17:27:42 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/03 17:27:47 Rebalance progress: 12.5
2021/12/03 17:27:52 Rebalance progress: 12.5
2021/12/03 17:27:57 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 17:27:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 17:27:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 17:27:57 Waiting for src bucket item count to get to: 200000 curr count: 33527
2021/12/03 17:28:02 Waiting for src bucket item count to get to: 200000 curr count: 49311
2021/12/03 17:28:07 Waiting for src bucket item count to get to: 200000 curr count: 49311
2021/12/03 17:28:12 Waiting for src bucket item count to get to: 200000 curr count: 65560
2021/12/03 17:28:17 Waiting for src bucket item count to get to: 200000 curr count: 65560
2021/12/03 17:28:22 Waiting for src bucket item count to get to: 200000 curr count: 80808
2021/12/03 17:28:27 Waiting for src bucket item count to get to: 200000 curr count: 80808
2021/12/03 17:28:32 Waiting for src bucket item count to get to: 200000 curr count: 96465
2021/12/03 17:28:37 Waiting for src bucket item count to get to: 200000 curr count: 96465
2021/12/03 17:28:42 Waiting for src bucket item count to get to: 200000 curr count: 96465
2021/12/03 17:28:47 Waiting for src bucket item count to get to: 200000 curr count: 112515
2021/12/03 17:28:52 Waiting for src bucket item count to get to: 200000 curr count: 112515
2021/12/03 17:28:57 Waiting for src bucket item count to get to: 200000 curr count: 128327
2021/12/03 17:29:02 Waiting for src bucket item count to get to: 200000 curr count: 128327
2021/12/03 17:29:07 Waiting for src bucket item count to get to: 200000 curr count: 144157
2021/12/03 17:29:12 Waiting for src bucket item count to get to: 200000 curr count: 144157
2021/12/03 17:29:17 Waiting for src bucket item count to get to: 200000 curr count: 160195
2021/12/03 17:29:22 Waiting for src bucket item count to get to: 200000 curr count: 160195
2021/12/03 17:29:27 Waiting for src bucket item count to get to: 200000 curr count: 175034
2021/12/03 17:29:32 Waiting for src bucket item count to get to: 200000 curr count: 175034
2021/12/03 17:29:37 Waiting for src bucket item count to get to: 200000 curr count: 190671
2021/12/03 17:29:42 Waiting for src bucket item count to get to: 200000 curr count: 190671
2021/12/03 17:29:47 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/03 17:29:47 Post SET ops. Expected item count: 200000 got 200000
2021/12/03 17:29:47 Starting bucket ops to source bucket
2021/12/03 17:33:39 Waiting for dst bucket item count to get to: 0 curr count: 8108
2021/12/03 17:33:44 Waiting for dst bucket item count to get to: 0 curr count: 8108
2021/12/03 17:33:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 17:33:44 Post DELETE ops. Expected item count: 0 got 0
2021/12/03 17:33:44 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 17:33:49 Waiting for app: TestKVRebInWithOps to get un-deployed
2021/12/03 17:33:54 Waiting for app: TestKVRebInWithOps to get un-deployed
2021/12/03 17:33:59 Waiting for app: TestKVRebInWithOps to get un-deployed
2021/12/03 17:34:04 Waiting for app: TestKVRebInWithOps to get un-deployed
2021/12/03 17:34:04 App: TestKVRebInWithOps got un-deployed
2021/12/03 17:34:04 No eventing-con process running
2021/12/03 17:34:04 Function: TestKVRebInWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVRebInWithOps deleting in the background"
}
2021/12/03 17:34:06 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 17:34:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 17:34:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 17:34:13 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 17:34:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 17:34:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebInWithOps (460.54s)
=== RUN   TestKVRebOutWithOps
2021/12/03 17:34:23 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/03 17:34:23 Removing node(s): [] from the cluster
2021/12/03 17:34:28 Rebalance progress: 100
2021/12/03 17:34:33 Waiting for app: TestKVRebOutWithOps to get un-deployed
2021/12/03 17:34:33 App: TestKVRebOutWithOps got un-deployed
2021/12/03 17:34:33 No eventing-con process running
2021/12/03 17:34:33 Function: TestKVRebOutWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVRebOutWithOps not found"
 }
}
2021/12/03 17:34:35 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 17:34:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 17:34:40 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 17:34:42 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 17:34:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 17:34:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 17:34:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 17:34:52 Deploying app: TestKVRebOutWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/03 17:34:53 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVRebOutWithOps' in metakv",
  "warnings": null
 }
}
2021/12/03 17:34:58 Waiting for app: TestKVRebOutWithOps to get deployed
2021/12/03 17:35:03 Waiting for app: TestKVRebOutWithOps to get deployed
2021/12/03 17:35:08 Waiting for app: TestKVRebOutWithOps to get deployed
2021/12/03 17:35:13 Waiting for app: TestKVRebOutWithOps to get deployed
2021/12/03 17:35:13 App: TestKVRebOutWithOps got deployed
2021/12/03 17:35:13 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/03 17:35:13 Starting bucket ops to source bucket
2021/12/03 17:35:18 Rebalance progress: 12.5
2021/12/03 17:35:23 Rebalance progress: 12.5
2021/12/03 17:35:28 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 17:35:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 17:35:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 17:35:28 Waiting for src bucket item count to get to: 200000 curr count: 9013
2021/12/03 17:35:33 Waiting for src bucket item count to get to: 200000 curr count: 24576
2021/12/03 17:35:38 Waiting for src bucket item count to get to: 200000 curr count: 24576
2021/12/03 17:35:43 Waiting for src bucket item count to get to: 200000 curr count: 40595
2021/12/03 17:35:48 Waiting for src bucket item count to get to: 200000 curr count: 40595
2021/12/03 17:35:53 Waiting for src bucket item count to get to: 200000 curr count: 56689
2021/12/03 17:35:58 Waiting for src bucket item count to get to: 200000 curr count: 56689
2021/12/03 17:36:03 Waiting for src bucket item count to get to: 200000 curr count: 56689
2021/12/03 17:36:08 Waiting for src bucket item count to get to: 200000 curr count: 72121
2021/12/03 17:36:13 Waiting for src bucket item count to get to: 200000 curr count: 72121
2021/12/03 17:36:18 Waiting for src bucket item count to get to: 200000 curr count: 87648
2021/12/03 17:36:23 Waiting for src bucket item count to get to: 200000 curr count: 87648
2021/12/03 17:36:28 Waiting for src bucket item count to get to: 200000 curr count: 103120
2021/12/03 17:36:33 Waiting for src bucket item count to get to: 200000 curr count: 103120
2021/12/03 17:36:38 Waiting for src bucket item count to get to: 200000 curr count: 119137
2021/12/03 17:36:43 Waiting for src bucket item count to get to: 200000 curr count: 119137
2021/12/03 17:36:48 Waiting for src bucket item count to get to: 200000 curr count: 135223
2021/12/03 17:36:53 Waiting for src bucket item count to get to: 200000 curr count: 135223
2021/12/03 17:36:58 Waiting for src bucket item count to get to: 200000 curr count: 151355
2021/12/03 17:37:03 Waiting for src bucket item count to get to: 200000 curr count: 151355
2021/12/03 17:37:08 Waiting for src bucket item count to get to: 200000 curr count: 167175
2021/12/03 17:37:13 Waiting for src bucket item count to get to: 200000 curr count: 167175
2021/12/03 17:37:18 Waiting for src bucket item count to get to: 200000 curr count: 182553
2021/12/03 17:37:23 Waiting for src bucket item count to get to: 200000 curr count: 182553
2021/12/03 17:37:28 Waiting for src bucket item count to get to: 200000 curr count: 198105
2021/12/03 17:37:33 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/03 17:37:33 Post SET ops. Expected item count: 200000 got 200000
2021/12/03 17:37:33 Starting bucket ops to source bucket
2021/12/03 17:41:25 Waiting for dst bucket item count to get to: 0 curr count: 19343
2021/12/03 17:41:30 Waiting for dst bucket item count to get to: 0 curr count: 1604
2021/12/03 17:41:35 Waiting for dst bucket item count to get to: 0 curr count: 1604
2021/12/03 17:41:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 17:41:35 Post DELETE ops. Expected item count: 0 got 0
2021/12/03 17:41:35 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 17:41:40 Waiting for app: TestKVRebOutWithOps to get un-deployed
2021/12/03 17:41:45 Waiting for app: TestKVRebOutWithOps to get un-deployed
2021/12/03 17:41:50 Waiting for app: TestKVRebOutWithOps to get un-deployed
2021/12/03 17:41:55 Waiting for app: TestKVRebOutWithOps to get un-deployed
2021/12/03 17:41:55 App: TestKVRebOutWithOps got un-deployed
2021/12/03 17:41:55 No eventing-con process running
2021/12/03 17:41:55 Function: TestKVRebOutWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVRebOutWithOps deleting in the background"
}
2021/12/03 17:41:57 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 17:42:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 17:42:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 17:42:04 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 17:42:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 17:42:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebOutWithOps (471.45s)
=== RUN   TestKVSwapRebWithOps
2021/12/03 17:42:19 Waiting for app: TestKVSwapRebWithOps to get un-deployed
2021/12/03 17:42:19 App: TestKVSwapRebWithOps got un-deployed
2021/12/03 17:42:19 No eventing-con process running
2021/12/03 17:42:19 Function: TestKVSwapRebWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVSwapRebWithOps not found"
 }
}
2021/12/03 17:42:21 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 17:42:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 17:42:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 17:42:28 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 17:42:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 17:42:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 17:42:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 17:42:38 Deploying app: TestKVSwapRebWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/03 17:42:39 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVSwapRebWithOps' in metakv",
  "warnings": null
 }
}
2021/12/03 17:42:44 Waiting for app: TestKVSwapRebWithOps to get deployed
2021/12/03 17:42:49 Waiting for app: TestKVSwapRebWithOps to get deployed
2021/12/03 17:42:54 Waiting for app: TestKVSwapRebWithOps to get deployed
2021/12/03 17:42:59 Waiting for app: TestKVSwapRebWithOps to get deployed
2021/12/03 17:42:59 App: TestKVSwapRebWithOps got deployed
2021/12/03 17:42:59 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/03 17:42:59 Starting bucket ops to source bucket
2021/12/03 17:42:59 Removing node(s): [] from the cluster
2021/12/03 17:43:04 Rebalance progress: 12.5
2021/12/03 17:43:09 Rebalance progress: 12.5
2021/12/03 17:43:14 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 17:43:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 17:43:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 17:43:14 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/03 17:43:15 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/03 17:43:20 Rebalance progress: 12.5
2021/12/03 17:43:25 Rebalance progress: 12.5
2021/12/03 17:43:30 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 17:43:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 17:43:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 17:43:30 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/03 17:43:35 Rebalance progress: 12.5
2021/12/03 17:43:40 Rebalance progress: 12.5
2021/12/03 17:43:45 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 17:43:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 17:43:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 17:43:45 Waiting for src bucket item count to get to: 200000 curr count: 61247
2021/12/03 17:43:50 Waiting for src bucket item count to get to: 200000 curr count: 61247
2021/12/03 17:43:55 Waiting for src bucket item count to get to: 200000 curr count: 76175
2021/12/03 17:44:00 Waiting for src bucket item count to get to: 200000 curr count: 76175
2021/12/03 17:44:05 Waiting for src bucket item count to get to: 200000 curr count: 91988
2021/12/03 17:44:10 Waiting for src bucket item count to get to: 200000 curr count: 91988
2021/12/03 17:44:15 Waiting for src bucket item count to get to: 200000 curr count: 108170
2021/12/03 17:44:20 Waiting for src bucket item count to get to: 200000 curr count: 108170
2021/12/03 17:44:25 Waiting for src bucket item count to get to: 200000 curr count: 124331
2021/12/03 17:44:30 Waiting for src bucket item count to get to: 200000 curr count: 140405
2021/12/03 17:44:35 Waiting for src bucket item count to get to: 200000 curr count: 140405
2021/12/03 17:44:40 Waiting for src bucket item count to get to: 200000 curr count: 156570
2021/12/03 17:44:45 Waiting for src bucket item count to get to: 200000 curr count: 156570
2021/12/03 17:44:50 Waiting for src bucket item count to get to: 200000 curr count: 156570
2021/12/03 17:44:55 Waiting for src bucket item count to get to: 200000 curr count: 171653
2021/12/03 17:45:00 Waiting for src bucket item count to get to: 200000 curr count: 171653
2021/12/03 17:45:05 Waiting for src bucket item count to get to: 200000 curr count: 187286
2021/12/03 17:45:10 Waiting for src bucket item count to get to: 200000 curr count: 187286
2021/12/03 17:45:15 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/03 17:45:15 Post SET ops. Expected item count: 200000 got 200000
2021/12/03 17:45:15 Starting bucket ops to source bucket
2021/12/03 17:49:07 Waiting for dst bucket item count to get to: 0 curr count: 22209
2021/12/03 17:49:12 Waiting for dst bucket item count to get to: 0 curr count: 4037
2021/12/03 17:49:17 Waiting for dst bucket item count to get to: 0 curr count: 4037
2021/12/03 17:49:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 17:49:17 Post DELETE ops. Expected item count: 0 got 0
2021/12/03 17:49:17 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 17:49:22 Waiting for app: TestKVSwapRebWithOps to get un-deployed
2021/12/03 17:49:27 Waiting for app: TestKVSwapRebWithOps to get un-deployed
2021/12/03 17:49:32 Waiting for app: TestKVSwapRebWithOps to get un-deployed
2021/12/03 17:49:37 Waiting for app: TestKVSwapRebWithOps to get un-deployed
2021/12/03 17:49:37 App: TestKVSwapRebWithOps got un-deployed
2021/12/03 17:49:37 No eventing-con process running
2021/12/03 17:49:37 Function: TestKVSwapRebWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVSwapRebWithOps deleting in the background"
}
2021/12/03 17:49:39 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 17:49:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 17:49:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 17:49:45 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 17:49:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 17:49:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVSwapRebWithOps (461.32s)
=== RUN   TestEventingRebInWithTimers
2021/12/03 17:50:00 Waiting for app: TestEventingRebInWithTimers to get un-deployed
2021/12/03 17:50:00 App: TestEventingRebInWithTimers got un-deployed
2021/12/03 17:50:00 No eventing-con process running
2021/12/03 17:50:00 Function: TestEventingRebInWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebInWithTimers not found"
 }
}
2021/12/03 17:50:02 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 17:50:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 17:50:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 17:50:09 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 17:50:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 17:50:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 17:50:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 17:50:19 Deploying app: TestEventingRebInWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/03 17:50:20 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebInWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/03 17:50:25 Waiting for app: TestEventingRebInWithTimers to get deployed
2021/12/03 17:50:30 Waiting for app: TestEventingRebInWithTimers to get deployed
2021/12/03 17:50:35 Waiting for app: TestEventingRebInWithTimers to get deployed
2021/12/03 17:50:40 Waiting for app: TestEventingRebInWithTimers to get deployed
2021/12/03 17:50:40 App: TestEventingRebInWithTimers got deployed
2021/12/03 17:50:40 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/03 17:50:40 Starting bucket ops to source bucket
2021/12/03 17:50:41 Removing node(s): [] from the cluster
2021/12/03 17:50:46 Rebalance progress: 12.5
2021/12/03 17:50:51 Rebalance progress: 12.5
2021/12/03 17:50:56 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 17:50:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 17:50:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 17:50:56 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/03 17:51:01 Rebalance progress: 12.5
2021/12/03 17:51:06 Rebalance progress: 12.5
2021/12/03 17:51:11 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 17:51:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 17:51:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 17:51:11 Waiting for src bucket item count to get to: 50000 curr count: 34866
2021/12/03 17:51:16 Waiting for src bucket item count to get to: 50000 curr count: 34866
2021/12/03 17:51:21 Waiting for src bucket item count to get to: 50000 curr count: 46291
2021/12/03 17:51:26 Waiting for src bucket item count to get to: 50000 curr count: 46291
2021/12/03 17:51:36 Waiting for dst bucket item count to get to: 50000 curr count: 41622
2021/12/03 17:51:41 Waiting for dst bucket item count to get to: 50000 curr count: 41622
2021/12/03 17:51:41 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/03 17:51:41 Post SET ops. Expected item count: 50000 got 50000
2021/12/03 17:51:41 Starting bucket ops to source bucket
2021/12/03 17:52:52 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/03 17:52:57 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/03 17:53:02 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/03 17:53:07 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/03 17:53:12 Waiting for dst bucket item count to get to: 0 curr count: 44774
2021/12/03 17:53:17 Waiting for dst bucket item count to get to: 0 curr count: 44774
2021/12/03 17:53:22 Waiting for dst bucket item count to get to: 0 curr count: 22796
2021/12/03 17:53:27 Waiting for dst bucket item count to get to: 0 curr count: 22796
2021/12/03 17:53:32 Waiting for dst bucket item count to get to: 0 curr count: 8539
2021/12/03 17:53:37 Waiting for dst bucket item count to get to: 0 curr count: 8539
2021/12/03 17:53:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 17:53:37 Post DELETE ops. Expected item count: 0 got 0
2021/12/03 17:53:37 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 17:53:42 Waiting for app: TestEventingRebInWithTimers to get un-deployed
2021/12/03 17:53:47 Waiting for app: TestEventingRebInWithTimers to get un-deployed
2021/12/03 17:53:52 Waiting for app: TestEventingRebInWithTimers to get un-deployed
2021/12/03 17:53:57 Waiting for app: TestEventingRebInWithTimers to get un-deployed
2021/12/03 17:53:57 App: TestEventingRebInWithTimers got un-deployed
2021/12/03 17:53:57 No eventing-con process running
2021/12/03 17:53:57 Function: TestEventingRebInWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebInWithTimers deleting in the background"
}
2021/12/03 17:53:59 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 17:54:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 17:54:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 17:54:06 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 17:54:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 17:54:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebInWithTimers (260.21s)
=== RUN   TestEventingRebOutWithTimers
2021/12/03 17:54:16 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/03 17:54:16 Removing node(s): [] from the cluster
2021/12/03 17:54:21 Rebalance progress: 100
2021/12/03 17:54:26 Waiting for app: TestEventingRebOutWithTimers to get un-deployed
2021/12/03 17:54:26 App: TestEventingRebOutWithTimers got un-deployed
2021/12/03 17:54:26 No eventing-con process running
2021/12/03 17:54:26 Function: TestEventingRebOutWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebOutWithTimers not found"
 }
}
2021/12/03 17:54:28 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 17:54:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 17:54:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 17:54:34 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 17:54:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 17:54:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 17:54:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 17:54:44 Deploying app: TestEventingRebOutWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/03 17:54:46 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebOutWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/03 17:54:51 Waiting for app: TestEventingRebOutWithTimers to get deployed
2021/12/03 17:54:56 Waiting for app: TestEventingRebOutWithTimers to get deployed
2021/12/03 17:55:01 Waiting for app: TestEventingRebOutWithTimers to get deployed
2021/12/03 17:55:06 Waiting for app: TestEventingRebOutWithTimers to get deployed
2021/12/03 17:55:06 App: TestEventingRebOutWithTimers got deployed
2021/12/03 17:55:06 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/03 17:55:06 Starting bucket ops to source bucket
2021/12/03 17:55:11 Rebalance progress: 12.5
2021/12/03 17:55:16 Rebalance progress: 12.5
2021/12/03 17:55:21 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 17:55:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 17:55:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 17:55:21 Waiting for src bucket item count to get to: 50000 curr count: 18653
2021/12/03 17:55:26 Waiting for src bucket item count to get to: 50000 curr count: 18653
2021/12/03 17:55:31 Waiting for src bucket item count to get to: 50000 curr count: 29056
2021/12/03 17:55:36 Waiting for src bucket item count to get to: 50000 curr count: 29056
2021/12/03 17:55:41 Waiting for src bucket item count to get to: 50000 curr count: 41210
2021/12/03 17:55:46 Waiting for src bucket item count to get to: 50000 curr count: 41210
2021/12/03 17:55:51 Waiting for src bucket item count to get to: 50000 curr count: 41210
2021/12/03 17:56:01 Waiting for dst bucket item count to get to: 50000 curr count: 39979
2021/12/03 17:56:06 Waiting for dst bucket item count to get to: 50000 curr count: 39979
2021/12/03 17:56:11 Waiting for dst bucket item count to get to: 50000 curr count: 48225
2021/12/03 17:56:16 Waiting for dst bucket item count to get to: 50000 curr count: 48225
2021/12/03 17:56:16 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/03 17:56:16 Post SET ops. Expected item count: 50000 got 50000
2021/12/03 17:56:16 Starting bucket ops to source bucket
2021/12/03 17:57:26 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/03 17:57:31 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/03 17:57:36 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/03 17:57:41 Waiting for dst bucket item count to get to: 0 curr count: 48446
2021/12/03 17:57:46 Waiting for dst bucket item count to get to: 0 curr count: 48446
2021/12/03 17:57:51 Waiting for dst bucket item count to get to: 0 curr count: 37637
2021/12/03 17:57:56 Waiting for dst bucket item count to get to: 0 curr count: 37637
2021/12/03 17:58:01 Waiting for dst bucket item count to get to: 0 curr count: 11308
2021/12/03 17:58:06 Waiting for dst bucket item count to get to: 0 curr count: 11308
2021/12/03 17:58:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 17:58:06 Post DELETE ops. Expected item count: 0 got 0
2021/12/03 17:58:06 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 17:58:11 Waiting for app: TestEventingRebOutWithTimers to get un-deployed
2021/12/03 17:58:16 Waiting for app: TestEventingRebOutWithTimers to get un-deployed
2021/12/03 17:58:21 Waiting for app: TestEventingRebOutWithTimers to get un-deployed
2021/12/03 17:58:26 Waiting for app: TestEventingRebOutWithTimers to get un-deployed
2021/12/03 17:58:26 App: TestEventingRebOutWithTimers got un-deployed
2021/12/03 17:58:26 No eventing-con process running
2021/12/03 17:58:26 Function: TestEventingRebOutWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebOutWithTimers deleting in the background"
}
2021/12/03 17:58:28 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 17:58:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 17:58:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 17:58:35 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 17:58:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 17:58:40 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebOutWithTimers (269.50s)
=== RUN   TestEventingSwapRebWithTimers
2021/12/03 17:58:50 Waiting for app: TestEventingSwapRebWithTimers to get un-deployed
2021/12/03 17:58:50 App: TestEventingSwapRebWithTimers got un-deployed
2021/12/03 17:58:50 No eventing-con process running
2021/12/03 17:58:50 Function: TestEventingSwapRebWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingSwapRebWithTimers not found"
 }
}
2021/12/03 17:58:52 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 17:58:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 17:58:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 17:58:59 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 17:59:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 17:59:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 17:59:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 17:59:09 Deploying app: TestEventingSwapRebWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/03 17:59:10 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingSwapRebWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/03 17:59:15 Waiting for app: TestEventingSwapRebWithTimers to get deployed
2021/12/03 17:59:20 Waiting for app: TestEventingSwapRebWithTimers to get deployed
2021/12/03 17:59:25 Waiting for app: TestEventingSwapRebWithTimers to get deployed
2021/12/03 17:59:30 Waiting for app: TestEventingSwapRebWithTimers to get deployed
2021/12/03 17:59:30 App: TestEventingSwapRebWithTimers got deployed
2021/12/03 17:59:30 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/03 17:59:30 Starting bucket ops to source bucket
2021/12/03 17:59:30 Removing node(s): [] from the cluster
2021/12/03 17:59:35 Rebalance progress: 12.5
2021/12/03 17:59:40 Rebalance progress: 12.5
2021/12/03 17:59:45 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 17:59:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 17:59:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 17:59:45 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/03 17:59:45 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/03 17:59:50 Rebalance progress: 12.5
2021/12/03 17:59:55 Rebalance progress: 12.5
2021/12/03 18:00:00 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:00:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:00:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 18:00:01 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/03 18:00:06 Rebalance progress: 12.5
2021/12/03 18:00:11 Rebalance progress: 12.5
2021/12/03 18:00:16 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:00:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:00:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 18:00:16 Waiting for src bucket item count to get to: 50000 curr count: 44102
2021/12/03 18:00:26 Waiting for dst bucket item count to get to: 50000 curr count: 43170
2021/12/03 18:00:31 Waiting for dst bucket item count to get to: 50000 curr count: 43170
2021/12/03 18:00:31 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/03 18:00:31 Post SET ops. Expected item count: 50000 got 50000
2021/12/03 18:00:31 Starting bucket ops to source bucket
2021/12/03 18:01:41 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/03 18:01:46 Waiting for dst bucket item count to get to: 0 curr count: 48325
2021/12/03 18:01:51 Waiting for dst bucket item count to get to: 0 curr count: 48325
2021/12/03 18:01:56 Waiting for dst bucket item count to get to: 0 curr count: 48325
2021/12/03 18:02:01 Waiting for dst bucket item count to get to: 0 curr count: 37360
2021/12/03 18:02:06 Waiting for dst bucket item count to get to: 0 curr count: 37360
2021/12/03 18:02:11 Waiting for dst bucket item count to get to: 0 curr count: 16207
2021/12/03 18:02:16 Waiting for dst bucket item count to get to: 0 curr count: 16207
2021/12/03 18:02:21 Waiting for dst bucket item count to get to: 0 curr count: 4545
2021/12/03 18:02:26 Waiting for dst bucket item count to get to: 0 curr count: 4545
2021/12/03 18:02:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 18:02:26 Post DELETE ops. Expected item count: 0 got 0
2021/12/03 18:02:27 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 18:02:32 Waiting for app: TestEventingSwapRebWithTimers to get un-deployed
2021/12/03 18:02:37 Waiting for app: TestEventingSwapRebWithTimers to get un-deployed
2021/12/03 18:02:42 Waiting for app: TestEventingSwapRebWithTimers to get un-deployed
2021/12/03 18:02:47 Waiting for app: TestEventingSwapRebWithTimers to get un-deployed
2021/12/03 18:02:47 App: TestEventingSwapRebWithTimers got un-deployed
2021/12/03 18:02:47 No eventing-con process running
2021/12/03 18:02:47 Function: TestEventingSwapRebWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingSwapRebWithTimers deleting in the background"
}
2021/12/03 18:02:49 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:02:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 18:02:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 18:02:55 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:03:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 18:03:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingSwapRebWithTimers (260.34s)
=== RUN   TestKVRebInWithTimers
2021/12/03 18:03:10 Waiting for app: TestKVRebInWithTimers to get un-deployed
2021/12/03 18:03:10 App: TestKVRebInWithTimers got un-deployed
2021/12/03 18:03:10 No eventing-con process running
2021/12/03 18:03:10 Function: TestKVRebInWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVRebInWithTimers not found"
 }
}
2021/12/03 18:03:12 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:03:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 18:03:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 18:03:19 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:03:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 18:03:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:03:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 18:03:29 Deploying app: TestKVRebInWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/03 18:03:30 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVRebInWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/03 18:03:35 Waiting for app: TestKVRebInWithTimers to get deployed
2021/12/03 18:03:40 Waiting for app: TestKVRebInWithTimers to get deployed
2021/12/03 18:03:45 Waiting for app: TestKVRebInWithTimers to get deployed
2021/12/03 18:03:50 Waiting for app: TestKVRebInWithTimers to get deployed
2021/12/03 18:03:50 App: TestKVRebInWithTimers got deployed
2021/12/03 18:03:50 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/03 18:03:50 Starting bucket ops to source bucket
2021/12/03 18:03:51 Removing node(s): [] from the cluster
2021/12/03 18:03:56 Rebalance progress: 12.5
2021/12/03 18:04:01 Rebalance progress: 12.5
2021/12/03 18:04:06 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:04:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:04:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 18:04:06 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/03 18:04:11 Rebalance progress: 12.5
2021/12/03 18:04:16 Rebalance progress: 12.5
2021/12/03 18:04:21 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:04:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:04:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 18:04:21 Waiting for src bucket item count to get to: 50000 curr count: 34014
2021/12/03 18:04:26 Waiting for src bucket item count to get to: 50000 curr count: 34014
2021/12/03 18:04:31 Waiting for src bucket item count to get to: 50000 curr count: 45681
2021/12/03 18:04:36 Waiting for src bucket item count to get to: 50000 curr count: 45681
2021/12/03 18:04:46 Waiting for dst bucket item count to get to: 50000 curr count: 42408
2021/12/03 18:04:51 Waiting for dst bucket item count to get to: 50000 curr count: 42408
2021/12/03 18:04:51 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/03 18:04:51 Post SET ops. Expected item count: 50000 got 50000
2021/12/03 18:04:51 Starting bucket ops to source bucket
2021/12/03 18:06:01 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/03 18:06:06 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/03 18:06:12 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/03 18:06:17 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/03 18:06:22 Waiting for dst bucket item count to get to: 0 curr count: 41952
2021/12/03 18:06:27 Waiting for dst bucket item count to get to: 0 curr count: 41952
2021/12/03 18:06:32 Waiting for dst bucket item count to get to: 0 curr count: 20066
2021/12/03 18:06:37 Waiting for dst bucket item count to get to: 0 curr count: 20066
2021/12/03 18:06:42 Waiting for dst bucket item count to get to: 0 curr count: 6700
2021/12/03 18:06:47 Waiting for dst bucket item count to get to: 0 curr count: 6700
2021/12/03 18:06:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 18:06:47 Post DELETE ops. Expected item count: 0 got 0
2021/12/03 18:06:47 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 18:06:52 Waiting for app: TestKVRebInWithTimers to get un-deployed
2021/12/03 18:06:57 Waiting for app: TestKVRebInWithTimers to get un-deployed
2021/12/03 18:07:02 Waiting for app: TestKVRebInWithTimers to get un-deployed
2021/12/03 18:07:07 Waiting for app: TestKVRebInWithTimers to get un-deployed
2021/12/03 18:07:07 App: TestKVRebInWithTimers got un-deployed
2021/12/03 18:07:07 No eventing-con process running
2021/12/03 18:07:07 Function: TestKVRebInWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVRebInWithTimers deleting in the background"
}
2021/12/03 18:07:09 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:07:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 18:07:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 18:07:15 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:07:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 18:07:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebInWithTimers (260.01s)
=== RUN   TestKVRebOutWithTimers
2021/12/03 18:07:25 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/03 18:07:26 Removing node(s): [] from the cluster
2021/12/03 18:07:31 Rebalance progress: 100
2021/12/03 18:07:36 Waiting for app: TestKVRebOutWithTimers to get un-deployed
2021/12/03 18:07:36 App: TestKVRebOutWithTimers got un-deployed
2021/12/03 18:07:36 No eventing-con process running
2021/12/03 18:07:36 Function: TestKVRebOutWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVRebOutWithTimers not found"
 }
}
2021/12/03 18:07:37 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:07:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 18:07:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 18:07:44 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:07:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 18:07:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:07:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 18:07:54 Deploying app: TestKVRebOutWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/03 18:07:55 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVRebOutWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/03 18:08:00 Waiting for app: TestKVRebOutWithTimers to get deployed
2021/12/03 18:08:05 Waiting for app: TestKVRebOutWithTimers to get deployed
2021/12/03 18:08:10 Waiting for app: TestKVRebOutWithTimers to get deployed
2021/12/03 18:08:15 Waiting for app: TestKVRebOutWithTimers to get deployed
2021/12/03 18:08:15 App: TestKVRebOutWithTimers got deployed
2021/12/03 18:08:15 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/03 18:08:15 Starting bucket ops to source bucket
2021/12/03 18:08:20 Rebalance progress: 12.5
2021/12/03 18:08:25 Rebalance progress: 12.5
2021/12/03 18:08:30 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:08:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:08:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 18:08:30 Waiting for src bucket item count to get to: 50000 curr count: 18940
2021/12/03 18:08:35 Waiting for src bucket item count to get to: 50000 curr count: 18940
2021/12/03 18:08:41 Waiting for src bucket item count to get to: 50000 curr count: 28840
2021/12/03 18:08:46 Waiting for src bucket item count to get to: 50000 curr count: 28840
2021/12/03 18:08:51 Waiting for src bucket item count to get to: 50000 curr count: 41066
2021/12/03 18:08:56 Waiting for src bucket item count to get to: 50000 curr count: 41066
2021/12/03 18:09:06 Waiting for dst bucket item count to get to: 50000 curr count: 41344
2021/12/03 18:09:11 Waiting for dst bucket item count to get to: 50000 curr count: 41344
2021/12/03 18:09:16 Waiting for dst bucket item count to get to: 50000 curr count: 41344
2021/12/03 18:09:21 Waiting for dst bucket item count to get to: 50000 curr count: 49559
2021/12/03 18:09:26 Waiting for dst bucket item count to get to: 50000 curr count: 49559
2021/12/03 18:09:26 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/03 18:09:26 Post SET ops. Expected item count: 50000 got 50000
2021/12/03 18:09:26 Starting bucket ops to source bucket
2021/12/03 18:10:36 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/03 18:10:41 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/03 18:10:46 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/03 18:10:51 Waiting for dst bucket item count to get to: 0 curr count: 45533
2021/12/03 18:10:56 Waiting for dst bucket item count to get to: 0 curr count: 45533
2021/12/03 18:11:01 Waiting for dst bucket item count to get to: 0 curr count: 30955
2021/12/03 18:11:06 Waiting for dst bucket item count to get to: 0 curr count: 30955
2021/12/03 18:11:11 Waiting for dst bucket item count to get to: 0 curr count: 9483
2021/12/03 18:11:16 Waiting for dst bucket item count to get to: 0 curr count: 9483
2021/12/03 18:11:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 18:11:16 Post DELETE ops. Expected item count: 0 got 0
2021/12/03 18:11:16 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 18:11:21 Waiting for app: TestKVRebOutWithTimers to get un-deployed
2021/12/03 18:11:26 Waiting for app: TestKVRebOutWithTimers to get un-deployed
2021/12/03 18:11:31 Waiting for app: TestKVRebOutWithTimers to get un-deployed
2021/12/03 18:11:36 Waiting for app: TestKVRebOutWithTimers to get un-deployed
2021/12/03 18:11:36 App: TestKVRebOutWithTimers got un-deployed
2021/12/03 18:11:36 No eventing-con process running
2021/12/03 18:11:36 Function: TestKVRebOutWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVRebOutWithTimers deleting in the background"
}
2021/12/03 18:11:38 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:11:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 18:11:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 18:11:45 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:11:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 18:11:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebOutWithTimers (269.56s)
=== RUN   TestKVSwapRebWithTimers
2021/12/03 18:12:00 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/03 18:12:00 App: TestKVSwapRebWithTimers got un-deployed
2021/12/03 18:12:00 No eventing-con process running
2021/12/03 18:12:00 Function: TestKVSwapRebWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVSwapRebWithTimers not found"
 }
}
2021/12/03 18:12:02 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:12:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 18:12:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 18:12:09 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:12:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 18:12:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:12:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 18:12:19 Deploying app: TestKVSwapRebWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/03 18:12:20 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVSwapRebWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/03 18:12:25 Waiting for app: TestKVSwapRebWithTimers to get deployed
2021/12/03 18:12:30 Waiting for app: TestKVSwapRebWithTimers to get deployed
2021/12/03 18:12:35 Waiting for app: TestKVSwapRebWithTimers to get deployed
2021/12/03 18:12:40 Waiting for app: TestKVSwapRebWithTimers to get deployed
2021/12/03 18:12:40 App: TestKVSwapRebWithTimers got deployed
2021/12/03 18:12:40 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/03 18:12:40 Starting bucket ops to source bucket
2021/12/03 18:12:40 Removing node(s): [] from the cluster
2021/12/03 18:12:45 Rebalance progress: 12.5
2021/12/03 18:12:50 Rebalance progress: 12.5
2021/12/03 18:12:55 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:12:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:12:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 18:12:55 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/03 18:12:56 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/03 18:13:01 Rebalance progress: 12.5
2021/12/03 18:13:06 Rebalance progress: 12.5
2021/12/03 18:13:11 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:13:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:13:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 18:13:11 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/03 18:13:16 Rebalance progress: 12.5
2021/12/03 18:13:21 Rebalance progress: 12.5
2021/12/03 18:13:26 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:13:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:13:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 18:13:26 Waiting for src bucket item count to get to: 50000 curr count: 46167
2021/12/03 18:13:36 Waiting for dst bucket item count to get to: 50000 curr count: 39192
2021/12/03 18:13:41 Waiting for dst bucket item count to get to: 50000 curr count: 39192
2021/12/03 18:13:46 Waiting for dst bucket item count to get to: 50000 curr count: 48798
2021/12/03 18:13:51 Waiting for dst bucket item count to get to: 50000 curr count: 48798
2021/12/03 18:13:51 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/03 18:13:51 Post SET ops. Expected item count: 50000 got 50000
2021/12/03 18:13:51 Starting bucket ops to source bucket
2021/12/03 18:15:01 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/03 18:15:06 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/03 18:15:12 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/03 18:15:17 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/03 18:15:22 Waiting for dst bucket item count to get to: 0 curr count: 40919
2021/12/03 18:15:27 Waiting for dst bucket item count to get to: 0 curr count: 40919
2021/12/03 18:15:32 Waiting for dst bucket item count to get to: 0 curr count: 30788
2021/12/03 18:15:37 Waiting for dst bucket item count to get to: 0 curr count: 30788
2021/12/03 18:15:42 Waiting for dst bucket item count to get to: 0 curr count: 11453
2021/12/03 18:15:47 Waiting for dst bucket item count to get to: 0 curr count: 11453
2021/12/03 18:15:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 18:15:47 Post DELETE ops. Expected item count: 0 got 0
2021/12/03 18:15:47 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 18:15:52 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/03 18:15:57 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/03 18:16:02 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/03 18:16:07 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/03 18:16:12 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/03 18:16:17 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/03 18:16:17 App: TestKVSwapRebWithTimers got un-deployed
2021/12/03 18:16:17 No eventing-con process running
2021/12/03 18:16:17 Function: TestKVSwapRebWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVSwapRebWithTimers deleting in the background"
}
2021/12/03 18:16:19 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:16:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 18:16:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 18:16:25 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:16:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 18:16:30 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVSwapRebWithTimers (280.48s)
=== RUN   TestDuplicateTimerCreationEventingRebIn
2021/12/03 18:16:40 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/03 18:16:40 App: TestDuplicateTimerCreationEventingRebIn got un-deployed
2021/12/03 18:16:40 No eventing-con process running
2021/12/03 18:16:40 Function: TestDuplicateTimerCreationEventingRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationEventingRebIn not found"
 }
}
2021/12/03 18:16:42 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:16:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 18:16:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 18:16:49 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:16:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 18:16:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:16:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 18:16:59 Deploying app: TestDuplicateTimerCreationEventingRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/03 18:17:00 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationEventingRebIn' in metakv",
  "warnings": null
 }
}
2021/12/03 18:17:05 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get deployed
2021/12/03 18:17:10 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get deployed
2021/12/03 18:17:15 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get deployed
2021/12/03 18:17:20 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get deployed
2021/12/03 18:17:20 App: TestDuplicateTimerCreationEventingRebIn got deployed
2021/12/03 18:17:20 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/03 18:17:20 Starting bucket ops to source bucket
2021/12/03 18:17:21 Removing node(s): [] from the cluster
2021/12/03 18:17:26 Rebalance progress: 12.5
2021/12/03 18:17:31 Rebalance progress: 12.5
2021/12/03 18:17:36 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:17:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:17:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 18:17:36 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/03 18:17:41 Rebalance progress: 12.5
2021/12/03 18:17:46 Rebalance progress: 12.5
2021/12/03 18:17:51 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:17:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:17:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 18:17:51 Waiting for src bucket item count to get to: 50000 curr count: 35713
2021/12/03 18:17:56 Waiting for src bucket item count to get to: 50000 curr count: 35713
2021/12/03 18:18:01 Waiting for src bucket item count to get to: 50000 curr count: 45412
2021/12/03 18:18:06 Waiting for src bucket item count to get to: 50000 curr count: 45412
2021/12/03 18:18:16 Waiting for dst bucket item count to get to: 50000 curr count: 33971
2021/12/03 18:18:21 Waiting for dst bucket item count to get to: 50000 curr count: 33971
2021/12/03 18:18:26 Waiting for dst bucket item count to get to: 50000 curr count: 45317
2021/12/03 18:18:31 Waiting for dst bucket item count to get to: 50000 curr count: 45317
2021/12/03 18:18:36 Waiting for dst bucket item count to get to: 50000 curr count: 49317
2021/12/03 18:18:41 Waiting for dst bucket item count to get to: 50000 curr count: 49317
2021/12/03 18:18:41 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/03 18:18:41 Post SET ops. Expected item count: 50000 got 50000
2021/12/03 18:18:41 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 18:18:46 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/03 18:18:51 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/03 18:18:56 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/03 18:19:01 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/03 18:19:06 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/03 18:19:11 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/03 18:19:11 App: TestDuplicateTimerCreationEventingRebIn got un-deployed
2021/12/03 18:19:11 No eventing-con process running
2021/12/03 18:19:11 Function: TestDuplicateTimerCreationEventingRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationEventingRebIn deleting in the background"
}
2021/12/03 18:19:13 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:19:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 18:19:23 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/03 18:19:28 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/03 18:19:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 18:19:30 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:19:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 18:19:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationEventingRebIn (184.45s)
=== RUN   TestDuplicateTimerCreationEventingRebOut
2021/12/03 18:19:40 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/03 18:19:41 Removing node(s): [] from the cluster
2021/12/03 18:19:46 Rebalance progress: 100
2021/12/03 18:19:51 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/03 18:19:51 App: TestDuplicateTimerCreationEventingRebOut got un-deployed
2021/12/03 18:19:51 No eventing-con process running
2021/12/03 18:19:51 Function: TestDuplicateTimerCreationEventingRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationEventingRebOut not found"
 }
}
2021/12/03 18:19:52 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:19:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 18:19:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 18:19:59 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:20:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 18:20:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:20:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 18:20:09 Deploying app: TestDuplicateTimerCreationEventingRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/03 18:20:10 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationEventingRebOut' in metakv",
  "warnings": null
 }
}
2021/12/03 18:20:15 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get deployed
2021/12/03 18:20:20 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get deployed
2021/12/03 18:20:25 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get deployed
2021/12/03 18:20:30 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get deployed
2021/12/03 18:20:30 App: TestDuplicateTimerCreationEventingRebOut got deployed
2021/12/03 18:20:30 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/03 18:20:30 Starting bucket ops to source bucket
2021/12/03 18:20:35 Rebalance progress: 12.5
2021/12/03 18:20:40 Rebalance progress: 12.5
2021/12/03 18:20:45 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:20:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:20:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 18:20:45 Waiting for src bucket item count to get to: 50000 curr count: 11478
2021/12/03 18:20:50 Waiting for src bucket item count to get to: 50000 curr count: 25435
2021/12/03 18:20:55 Waiting for src bucket item count to get to: 50000 curr count: 25435
2021/12/03 18:21:01 Waiting for src bucket item count to get to: 50000 curr count: 36870
2021/12/03 18:21:06 Waiting for src bucket item count to get to: 50000 curr count: 36870
2021/12/03 18:21:11 Waiting for src bucket item count to get to: 50000 curr count: 46372
2021/12/03 18:21:16 Waiting for src bucket item count to get to: 50000 curr count: 46372
2021/12/03 18:21:21 Waiting for src bucket item count to get to: 50000 curr count: 46372
2021/12/03 18:21:31 Waiting for dst bucket item count to get to: 50000 curr count: 36389
2021/12/03 18:21:36 Waiting for dst bucket item count to get to: 50000 curr count: 36389
2021/12/03 18:21:41 Waiting for dst bucket item count to get to: 50000 curr count: 48010
2021/12/03 18:21:46 Waiting for dst bucket item count to get to: 50000 curr count: 48010
2021/12/03 18:21:46 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/03 18:21:46 Post SET ops. Expected item count: 50000 got 50000
2021/12/03 18:21:46 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 18:21:51 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/03 18:21:56 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/03 18:22:01 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/03 18:22:06 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/03 18:22:11 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/03 18:22:16 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/03 18:22:16 App: TestDuplicateTimerCreationEventingRebOut got un-deployed
2021/12/03 18:22:16 No eventing-con process running
2021/12/03 18:22:16 Function: TestDuplicateTimerCreationEventingRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationEventingRebOut deleting in the background"
}
2021/12/03 18:22:18 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:22:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 18:22:28 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/03 18:22:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 18:22:29 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:22:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 18:22:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationEventingRebOut (179.62s)
=== RUN   TestDuplicateTimerCreationEventingSwapReb
2021/12/03 18:22:44 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/03 18:22:44 App: TestDuplicateTimerCreationEventingSwapReb got un-deployed
2021/12/03 18:22:44 No eventing-con process running
2021/12/03 18:22:44 Function: TestDuplicateTimerCreationEventingSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationEventingSwapReb not found"
 }
}
2021/12/03 18:22:46 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:22:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 18:22:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 18:22:53 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:22:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 18:22:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:23:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 18:23:03 Deploying app: TestDuplicateTimerCreationEventingSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/03 18:23:04 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationEventingSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/03 18:23:09 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get deployed
2021/12/03 18:23:14 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get deployed
2021/12/03 18:23:19 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get deployed
2021/12/03 18:23:24 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get deployed
2021/12/03 18:23:24 App: TestDuplicateTimerCreationEventingSwapReb got deployed
2021/12/03 18:23:24 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/03 18:23:24 Starting bucket ops to source bucket
2021/12/03 18:23:25 Removing node(s): [] from the cluster
2021/12/03 18:23:30 Rebalance progress: 12.5
2021/12/03 18:23:35 Rebalance progress: 12.5
2021/12/03 18:23:40 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:23:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:23:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 18:23:40 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/03 18:23:41 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/03 18:23:46 Rebalance progress: 12.5
2021/12/03 18:23:51 Rebalance progress: 12.5
2021/12/03 18:23:56 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:23:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:23:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 18:23:56 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/03 18:24:01 Rebalance progress: 12.5
2021/12/03 18:24:06 Rebalance progress: 12.5
2021/12/03 18:24:11 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:24:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:24:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 18:24:11 Waiting for src bucket item count to get to: 50000 curr count: 40554
2021/12/03 18:24:21 Waiting for dst bucket item count to get to: 50000 curr count: 30147
2021/12/03 18:24:26 Waiting for dst bucket item count to get to: 50000 curr count: 30147
2021/12/03 18:24:31 Waiting for dst bucket item count to get to: 50000 curr count: 37341
2021/12/03 18:24:36 Waiting for dst bucket item count to get to: 50000 curr count: 48845
2021/12/03 18:24:41 Waiting for dst bucket item count to get to: 50000 curr count: 48845
2021/12/03 18:24:41 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/03 18:24:41 Post SET ops. Expected item count: 50000 got 50000
2021/12/03 18:24:41 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 18:24:46 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/03 18:24:51 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/03 18:24:56 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/03 18:25:01 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/03 18:25:06 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/03 18:25:11 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/03 18:25:11 App: TestDuplicateTimerCreationEventingSwapReb got un-deployed
2021/12/03 18:25:11 No eventing-con process running
2021/12/03 18:25:11 Function: TestDuplicateTimerCreationEventingSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationEventingSwapReb deleting in the background"
}
2021/12/03 18:25:13 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:25:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 18:25:23 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/03 18:25:28 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/03 18:25:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 18:25:30 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:25:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 18:25:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationEventingSwapReb (180.21s)
=== RUN   TestDuplicateTimerCreationKVRebIn
2021/12/03 18:25:45 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/03 18:25:45 App: TestDuplicateTimerCreationKVRebIn got un-deployed
2021/12/03 18:25:45 No eventing-con process running
2021/12/03 18:25:45 Function: TestDuplicateTimerCreationKVRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationKVRebIn not found"
 }
}
2021/12/03 18:25:46 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:25:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 18:25:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 18:25:53 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:25:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 18:25:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:26:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 18:26:03 Deploying app: TestDuplicateTimerCreationKVRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/03 18:26:04 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationKVRebIn' in metakv",
  "warnings": null
 }
}
2021/12/03 18:26:09 Waiting for app: TestDuplicateTimerCreationKVRebIn to get deployed
2021/12/03 18:26:14 Waiting for app: TestDuplicateTimerCreationKVRebIn to get deployed
2021/12/03 18:26:19 Waiting for app: TestDuplicateTimerCreationKVRebIn to get deployed
2021/12/03 18:26:24 Waiting for app: TestDuplicateTimerCreationKVRebIn to get deployed
2021/12/03 18:26:24 App: TestDuplicateTimerCreationKVRebIn got deployed
2021/12/03 18:26:24 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/03 18:26:24 Starting bucket ops to source bucket
2021/12/03 18:26:25 Removing node(s): [] from the cluster
2021/12/03 18:26:30 Rebalance progress: 12.5
2021/12/03 18:26:35 Rebalance progress: 12.5
2021/12/03 18:26:40 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:26:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:26:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 18:26:40 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/03 18:26:45 Rebalance progress: 12.5
2021/12/03 18:26:50 Rebalance progress: 12.5
2021/12/03 18:26:55 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:26:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:26:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 18:26:55 Waiting for src bucket item count to get to: 50000 curr count: 31931
2021/12/03 18:27:00 Waiting for src bucket item count to get to: 50000 curr count: 31931
2021/12/03 18:27:05 Waiting for src bucket item count to get to: 50000 curr count: 42862
2021/12/03 18:27:10 Waiting for src bucket item count to get to: 50000 curr count: 42862
2021/12/03 18:27:20 Waiting for dst bucket item count to get to: 50000 curr count: 27913
2021/12/03 18:27:25 Waiting for dst bucket item count to get to: 50000 curr count: 35430
2021/12/03 18:27:30 Waiting for dst bucket item count to get to: 50000 curr count: 35430
2021/12/03 18:27:35 Waiting for dst bucket item count to get to: 50000 curr count: 49382
2021/12/03 18:27:40 Waiting for dst bucket item count to get to: 50000 curr count: 49382
2021/12/03 18:27:40 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/03 18:27:40 Post SET ops. Expected item count: 50000 got 50000
2021/12/03 18:27:40 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 18:27:45 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/03 18:27:50 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/03 18:27:55 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/03 18:28:00 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/03 18:28:05 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/03 18:28:05 App: TestDuplicateTimerCreationKVRebIn got un-deployed
2021/12/03 18:28:05 No eventing-con process running
2021/12/03 18:28:05 Function: TestDuplicateTimerCreationKVRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationKVRebIn deleting in the background"
}
2021/12/03 18:28:07 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:28:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 18:28:17 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/03 18:28:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 18:28:19 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:28:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 18:28:29 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/03 18:28:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationKVRebIn (174.32s)
=== RUN   TestDuplicateTimerCreationKVRebOut
2021/12/03 18:28:34 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/03 18:28:34 Removing node(s): [] from the cluster
2021/12/03 18:28:39 Rebalance progress: 100
2021/12/03 18:28:44 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/03 18:28:44 App: TestDuplicateTimerCreationKVRebOut got un-deployed
2021/12/03 18:28:44 No eventing-con process running
2021/12/03 18:28:44 Function: TestDuplicateTimerCreationKVRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationKVRebOut not found"
 }
}
2021/12/03 18:28:46 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:28:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 18:28:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 18:28:53 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:28:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 18:28:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:29:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 18:29:03 Deploying app: TestDuplicateTimerCreationKVRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/03 18:29:04 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationKVRebOut' in metakv",
  "warnings": null
 }
}
2021/12/03 18:29:09 Waiting for app: TestDuplicateTimerCreationKVRebOut to get deployed
2021/12/03 18:29:14 Waiting for app: TestDuplicateTimerCreationKVRebOut to get deployed
2021/12/03 18:29:19 Waiting for app: TestDuplicateTimerCreationKVRebOut to get deployed
2021/12/03 18:29:24 Waiting for app: TestDuplicateTimerCreationKVRebOut to get deployed
2021/12/03 18:29:24 App: TestDuplicateTimerCreationKVRebOut got deployed
2021/12/03 18:29:24 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/03 18:29:24 Starting bucket ops to source bucket
2021/12/03 18:29:29 Rebalance progress: 12.5
2021/12/03 18:29:34 Rebalance progress: 12.5
2021/12/03 18:29:39 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:29:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:29:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 18:29:39 Waiting for src bucket item count to get to: 50000 curr count: 6180
2021/12/03 18:29:44 Waiting for src bucket item count to get to: 50000 curr count: 19931
2021/12/03 18:29:49 Waiting for src bucket item count to get to: 50000 curr count: 19931
2021/12/03 18:29:54 Waiting for src bucket item count to get to: 50000 curr count: 33003
2021/12/03 18:29:59 Waiting for src bucket item count to get to: 50000 curr count: 33003
2021/12/03 18:30:04 Waiting for src bucket item count to get to: 50000 curr count: 43625
2021/12/03 18:30:09 Waiting for src bucket item count to get to: 50000 curr count: 43625
2021/12/03 18:30:14 Waiting for src bucket item count to get to: 50000 curr count: 43625
2021/12/03 18:30:24 Waiting for dst bucket item count to get to: 50000 curr count: 31902
2021/12/03 18:30:29 Waiting for dst bucket item count to get to: 50000 curr count: 31902
2021/12/03 18:30:34 Waiting for dst bucket item count to get to: 50000 curr count: 39183
2021/12/03 18:30:39 Waiting for dst bucket item count to get to: 50000 curr count: 39183
2021/12/03 18:30:39 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/03 18:30:39 Post SET ops. Expected item count: 50000 got 50000
2021/12/03 18:30:39 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 18:30:44 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/03 18:30:49 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/03 18:30:54 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/03 18:30:59 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/03 18:31:04 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/03 18:31:09 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/03 18:31:09 App: TestDuplicateTimerCreationKVRebOut got un-deployed
2021/12/03 18:31:09 No eventing-con process running
2021/12/03 18:31:10 Function: TestDuplicateTimerCreationKVRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationKVRebOut deleting in the background"
}
2021/12/03 18:31:11 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:31:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 18:31:21 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/03 18:31:26 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/03 18:31:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 18:31:28 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:31:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 18:31:38 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/03 18:31:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationKVRebOut (189.18s)
=== RUN   TestDuplicateTimerCreationKVSwapReb
2021/12/03 18:31:48 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/03 18:31:48 App: TestDuplicateTimerCreationKVSwapReb got un-deployed
2021/12/03 18:31:48 No eventing-con process running
2021/12/03 18:31:48 Function: TestDuplicateTimerCreationKVSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationKVSwapReb not found"
 }
}
2021/12/03 18:31:50 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:31:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 18:31:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 18:31:57 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:32:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 18:32:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:32:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 18:32:07 Deploying app: TestDuplicateTimerCreationKVSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/03 18:32:08 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationKVSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/03 18:32:13 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get deployed
2021/12/03 18:32:18 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get deployed
2021/12/03 18:32:23 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get deployed
2021/12/03 18:32:28 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get deployed
2021/12/03 18:32:28 App: TestDuplicateTimerCreationKVSwapReb got deployed
2021/12/03 18:32:28 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/03 18:32:28 Starting bucket ops to source bucket
2021/12/03 18:32:28 Removing node(s): [] from the cluster
2021/12/03 18:32:33 Rebalance progress: 12.5
2021/12/03 18:32:38 Rebalance progress: 12.5
2021/12/03 18:32:43 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:32:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:32:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 18:32:43 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/03 18:32:44 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/03 18:32:49 Rebalance progress: 12.5
2021/12/03 18:32:54 Rebalance progress: 12.5
2021/12/03 18:32:59 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:32:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:32:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 18:32:59 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/03 18:33:04 Rebalance progress: 12.5
2021/12/03 18:33:09 Rebalance progress: 12.5
2021/12/03 18:33:14 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:33:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:33:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 18:33:14 Waiting for src bucket item count to get to: 50000 curr count: 48400
2021/12/03 18:33:19 Waiting for src bucket item count to get to: 50000 curr count: 48400
2021/12/03 18:33:29 Waiting for dst bucket item count to get to: 50000 curr count: 35049
2021/12/03 18:33:34 Waiting for dst bucket item count to get to: 50000 curr count: 35049
2021/12/03 18:33:39 Waiting for dst bucket item count to get to: 50000 curr count: 41153
2021/12/03 18:33:44 Waiting for dst bucket item count to get to: 50000 curr count: 41153
2021/12/03 18:33:44 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/03 18:33:44 Post SET ops. Expected item count: 50000 got 50000
2021/12/03 18:33:44 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 18:33:49 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/03 18:33:54 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/03 18:33:59 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/03 18:34:04 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/03 18:34:09 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/03 18:34:14 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/03 18:34:14 App: TestDuplicateTimerCreationKVSwapReb got un-deployed
2021/12/03 18:34:14 No eventing-con process running
2021/12/03 18:34:14 Function: TestDuplicateTimerCreationKVSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationKVSwapReb deleting in the background"
}
2021/12/03 18:34:16 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:34:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 18:34:26 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/03 18:34:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 18:34:28 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:34:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 18:34:38 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/03 18:34:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationKVSwapReb (179.71s)
=== RUN   TestDuplicateTimerFiringEventingRebIn
2021/12/03 18:34:48 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/03 18:34:48 App: TestDuplicateTimerFiringEventingRebIn got un-deployed
2021/12/03 18:34:48 No eventing-con process running
2021/12/03 18:34:48 Function: TestDuplicateTimerFiringEventingRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringEventingRebIn not found"
 }
}
2021/12/03 18:34:50 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:34:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 18:34:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 18:34:57 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:35:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 18:35:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:35:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 18:35:07 Deploying app: TestDuplicateTimerFiringEventingRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/03 18:35:08 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringEventingRebIn' in metakv",
  "warnings": null
 }
}
2021/12/03 18:35:13 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get deployed
2021/12/03 18:35:18 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get deployed
2021/12/03 18:35:23 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get deployed
2021/12/03 18:35:28 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get deployed
2021/12/03 18:35:28 App: TestDuplicateTimerFiringEventingRebIn got deployed
2021/12/03 18:35:28 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/03 18:35:28 Starting bucket ops to source bucket
2021/12/03 18:35:28 Removing node(s): [] from the cluster
2021/12/03 18:35:33 Rebalance progress: 12.5
2021/12/03 18:35:38 Rebalance progress: 12.5
2021/12/03 18:35:43 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:35:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:35:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 18:35:43 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/03 18:35:48 Rebalance progress: 12.5
2021/12/03 18:35:53 Rebalance progress: 12.5
2021/12/03 18:35:58 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:35:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:35:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 18:35:58 Waiting for src bucket item count to get to: 50000 curr count: 26062
2021/12/03 18:36:03 Waiting for src bucket item count to get to: 50000 curr count: 36073
2021/12/03 18:36:08 Waiting for src bucket item count to get to: 50000 curr count: 36073
2021/12/03 18:36:13 Waiting for src bucket item count to get to: 50000 curr count: 48029
2021/12/03 18:36:18 Waiting for src bucket item count to get to: 50000 curr count: 48029
2021/12/03 18:36:28 Waiting for dst bucket item count to get to: 50000 curr count: 45686
2021/12/03 18:36:33 Waiting for dst bucket item count to get to: 50000 curr count: 45686
2021/12/03 18:36:33 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/03 18:36:33 Post SET ops. Expected item count: 50000 got 50000
2021/12/03 18:36:34 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 18:36:39 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/03 18:36:44 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/03 18:36:49 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/03 18:36:54 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/03 18:36:59 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/03 18:37:04 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/03 18:37:04 App: TestDuplicateTimerFiringEventingRebIn got un-deployed
2021/12/03 18:37:04 No eventing-con process running
2021/12/03 18:37:04 Function: TestDuplicateTimerFiringEventingRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringEventingRebIn deleting in the background"
}
2021/12/03 18:37:06 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:37:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 18:37:16 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/03 18:37:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 18:37:17 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:37:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 18:37:27 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/03 18:37:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringEventingRebIn (169.57s)
=== RUN   TestDuplicateTimerFiringEventingRebOut
2021/12/03 18:37:32 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/03 18:37:33 Removing node(s): [] from the cluster
2021/12/03 18:37:38 Rebalance progress: 100
2021/12/03 18:37:43 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/03 18:37:43 App: TestDuplicateTimerFiringEventingRebOut got un-deployed
2021/12/03 18:37:43 No eventing-con process running
2021/12/03 18:37:43 Function: TestDuplicateTimerFiringEventingRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringEventingRebOut not found"
 }
}
2021/12/03 18:37:45 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:37:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 18:37:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 18:37:52 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:37:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 18:37:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:38:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 18:38:02 Deploying app: TestDuplicateTimerFiringEventingRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/03 18:38:03 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringEventingRebOut' in metakv",
  "warnings": null
 }
}
2021/12/03 18:38:08 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get deployed
2021/12/03 18:38:13 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get deployed
2021/12/03 18:38:18 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get deployed
2021/12/03 18:38:23 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get deployed
2021/12/03 18:38:23 App: TestDuplicateTimerFiringEventingRebOut got deployed
2021/12/03 18:38:23 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/03 18:38:23 Starting bucket ops to source bucket
2021/12/03 18:38:28 Rebalance progress: 12.5
2021/12/03 18:38:33 Rebalance progress: 12.5
2021/12/03 18:38:38 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:38:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:38:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 18:38:38 Waiting for src bucket item count to get to: 50000 curr count: 7703
2021/12/03 18:38:43 Waiting for src bucket item count to get to: 50000 curr count: 20337
2021/12/03 18:38:48 Waiting for src bucket item count to get to: 50000 curr count: 20337
2021/12/03 18:38:53 Waiting for src bucket item count to get to: 50000 curr count: 31784
2021/12/03 18:38:58 Waiting for src bucket item count to get to: 50000 curr count: 31784
2021/12/03 18:39:03 Waiting for src bucket item count to get to: 50000 curr count: 42401
2021/12/03 18:39:08 Waiting for src bucket item count to get to: 50000 curr count: 42401
2021/12/03 18:39:13 Waiting for src bucket item count to get to: 50000 curr count: 42401
2021/12/03 18:39:23 Waiting for dst bucket item count to get to: 50000 curr count: 38209
2021/12/03 18:39:28 Waiting for dst bucket item count to get to: 50000 curr count: 38209
2021/12/03 18:39:28 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/03 18:39:28 Post SET ops. Expected item count: 50000 got 50000
2021/12/03 18:39:28 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 18:39:33 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/03 18:39:38 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/03 18:39:43 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/03 18:39:48 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/03 18:39:53 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/03 18:39:58 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/03 18:39:58 App: TestDuplicateTimerFiringEventingRebOut got un-deployed
2021/12/03 18:39:58 No eventing-con process running
2021/12/03 18:39:58 Function: TestDuplicateTimerFiringEventingRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringEventingRebOut deleting in the background"
}
2021/12/03 18:40:00 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:40:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 18:40:10 Waiting for dst bucket item count to get to: 0 curr count: 17635
2021/12/03 18:40:15 Waiting for dst bucket item count to get to: 0 curr count: 17635
2021/12/03 18:40:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 18:40:17 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:40:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 18:40:27 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/03 18:40:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringEventingRebOut (179.68s)
=== RUN   TestDuplicateTimerFiringEventingSwapReb
2021/12/03 18:40:37 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/03 18:40:37 App: TestDuplicateTimerFiringEventingSwapReb got un-deployed
2021/12/03 18:40:37 No eventing-con process running
2021/12/03 18:40:37 Function: TestDuplicateTimerFiringEventingSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringEventingSwapReb not found"
 }
}
2021/12/03 18:40:39 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:40:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 18:40:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 18:40:46 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:40:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 18:40:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:40:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 18:40:56 Deploying app: TestDuplicateTimerFiringEventingSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/03 18:40:57 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringEventingSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/03 18:41:02 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get deployed
2021/12/03 18:41:07 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get deployed
2021/12/03 18:41:12 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get deployed
2021/12/03 18:41:17 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get deployed
2021/12/03 18:41:17 App: TestDuplicateTimerFiringEventingSwapReb got deployed
2021/12/03 18:41:17 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/03 18:41:17 Starting bucket ops to source bucket
2021/12/03 18:41:17 Removing node(s): [] from the cluster
2021/12/03 18:41:22 Rebalance progress: 12.5
2021/12/03 18:41:27 Rebalance progress: 12.5
2021/12/03 18:41:32 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:41:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:41:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 18:41:32 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/03 18:41:33 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/03 18:41:38 Rebalance progress: 12.5
2021/12/03 18:41:43 Rebalance progress: 12.5
2021/12/03 18:41:48 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:41:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:41:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 18:41:48 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/03 18:41:53 Rebalance progress: 12.5
2021/12/03 18:41:58 Rebalance progress: 12.5
2021/12/03 18:42:03 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:42:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:42:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 18:42:03 Waiting for src bucket item count to get to: 50000 curr count: 47728
2021/12/03 18:42:08 Waiting for src bucket item count to get to: 50000 curr count: 47728
2021/12/03 18:42:18 Waiting for dst bucket item count to get to: 50000 curr count: 45849
2021/12/03 18:42:23 Waiting for dst bucket item count to get to: 50000 curr count: 45849
2021/12/03 18:42:23 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/03 18:42:23 Post SET ops. Expected item count: 50000 got 50000
2021/12/03 18:42:23 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 18:42:28 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/03 18:42:33 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/03 18:42:38 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/03 18:42:43 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/03 18:42:48 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/03 18:42:53 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/03 18:42:53 App: TestDuplicateTimerFiringEventingSwapReb got un-deployed
2021/12/03 18:42:53 No eventing-con process running
2021/12/03 18:42:53 Function: TestDuplicateTimerFiringEventingSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringEventingSwapReb deleting in the background"
}
2021/12/03 18:42:55 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:43:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 18:43:05 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/03 18:43:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 18:43:07 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:43:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 18:43:17 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/03 18:43:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringEventingSwapReb (169.67s)
=== RUN   TestDuplicateTimerFiringKVRebIn
2021/12/03 18:43:27 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/03 18:43:27 App: TestDuplicateTimerFiringKVRebIn got un-deployed
2021/12/03 18:43:27 No eventing-con process running
2021/12/03 18:43:27 Function: TestDuplicateTimerFiringKVRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringKVRebIn not found"
 }
}
2021/12/03 18:43:29 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:43:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 18:43:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 18:43:35 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:43:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 18:43:40 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:43:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 18:43:45 Deploying app: TestDuplicateTimerFiringKVRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/03 18:43:47 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringKVRebIn' in metakv",
  "warnings": null
 }
}
2021/12/03 18:43:52 Waiting for app: TestDuplicateTimerFiringKVRebIn to get deployed
2021/12/03 18:43:57 Waiting for app: TestDuplicateTimerFiringKVRebIn to get deployed
2021/12/03 18:44:02 Waiting for app: TestDuplicateTimerFiringKVRebIn to get deployed
2021/12/03 18:44:07 Waiting for app: TestDuplicateTimerFiringKVRebIn to get deployed
2021/12/03 18:44:07 App: TestDuplicateTimerFiringKVRebIn got deployed
2021/12/03 18:44:07 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/03 18:44:07 Starting bucket ops to source bucket
2021/12/03 18:44:07 Removing node(s): [] from the cluster
2021/12/03 18:44:12 Rebalance progress: 12.5
2021/12/03 18:44:17 Rebalance progress: 12.5
2021/12/03 18:44:22 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:44:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:44:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 18:44:22 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/03 18:44:27 Rebalance progress: 12.5
2021/12/03 18:44:32 Rebalance progress: 12.5
2021/12/03 18:44:37 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:44:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:44:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 18:44:37 Waiting for src bucket item count to get to: 50000 curr count: 26502
2021/12/03 18:44:42 Waiting for src bucket item count to get to: 50000 curr count: 37291
2021/12/03 18:44:47 Waiting for src bucket item count to get to: 50000 curr count: 37291
2021/12/03 18:44:52 Waiting for src bucket item count to get to: 50000 curr count: 48406
2021/12/03 18:44:57 Waiting for src bucket item count to get to: 50000 curr count: 48406
2021/12/03 18:45:07 Waiting for dst bucket item count to get to: 50000 curr count: 44204
2021/12/03 18:45:12 Waiting for dst bucket item count to get to: 50000 curr count: 44204
2021/12/03 18:45:12 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/03 18:45:12 Post SET ops. Expected item count: 50000 got 50000
2021/12/03 18:45:12 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 18:45:17 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/03 18:45:23 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/03 18:45:28 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/03 18:45:33 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/03 18:45:38 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/03 18:45:43 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/03 18:45:43 App: TestDuplicateTimerFiringKVRebIn got un-deployed
2021/12/03 18:45:43 No eventing-con process running
2021/12/03 18:45:43 Function: TestDuplicateTimerFiringKVRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringKVRebIn deleting in the background"
}
2021/12/03 18:45:45 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:45:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 18:45:55 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/03 18:45:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 18:45:56 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:46:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 18:46:06 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/03 18:46:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringKVRebIn (169.63s)
=== RUN   TestDuplicateTimerFiringKVRebOut
2021/12/03 18:46:11 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/03 18:46:12 Removing node(s): [] from the cluster
2021/12/03 18:46:17 Rebalance progress: 100
2021/12/03 18:46:22 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/03 18:46:22 App: TestDuplicateTimerFiringKVRebOut got un-deployed
2021/12/03 18:46:22 No eventing-con process running
2021/12/03 18:46:22 Function: TestDuplicateTimerFiringKVRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringKVRebOut not found"
 }
}
2021/12/03 18:46:24 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:46:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 18:46:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 18:46:31 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:46:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 18:46:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:46:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 18:46:41 Deploying app: TestDuplicateTimerFiringKVRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/03 18:46:42 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringKVRebOut' in metakv",
  "warnings": null
 }
}
2021/12/03 18:46:47 Waiting for app: TestDuplicateTimerFiringKVRebOut to get deployed
2021/12/03 18:46:52 Waiting for app: TestDuplicateTimerFiringKVRebOut to get deployed
2021/12/03 18:46:57 Waiting for app: TestDuplicateTimerFiringKVRebOut to get deployed
2021/12/03 18:47:02 Waiting for app: TestDuplicateTimerFiringKVRebOut to get deployed
2021/12/03 18:47:02 App: TestDuplicateTimerFiringKVRebOut got deployed
2021/12/03 18:47:02 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/03 18:47:02 Starting bucket ops to source bucket
2021/12/03 18:47:07 Rebalance progress: 12.5
2021/12/03 18:47:12 Rebalance progress: 12.5
2021/12/03 18:47:17 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:47:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:47:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 18:47:17 Waiting for src bucket item count to get to: 50000 curr count: 9524
2021/12/03 18:47:22 Waiting for src bucket item count to get to: 50000 curr count: 22408
2021/12/03 18:47:27 Waiting for src bucket item count to get to: 50000 curr count: 22408
2021/12/03 18:47:32 Waiting for src bucket item count to get to: 50000 curr count: 33635
2021/12/03 18:47:37 Waiting for src bucket item count to get to: 50000 curr count: 33635
2021/12/03 18:47:42 Waiting for src bucket item count to get to: 50000 curr count: 43357
2021/12/03 18:47:47 Waiting for src bucket item count to get to: 50000 curr count: 43357
2021/12/03 18:47:52 Waiting for src bucket item count to get to: 50000 curr count: 43357
2021/12/03 18:48:02 Waiting for dst bucket item count to get to: 50000 curr count: 37100
2021/12/03 18:48:07 Waiting for dst bucket item count to get to: 50000 curr count: 37100
2021/12/03 18:48:07 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/03 18:48:07 Post SET ops. Expected item count: 50000 got 50000
2021/12/03 18:48:07 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 18:48:12 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/03 18:48:17 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/03 18:48:22 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/03 18:48:27 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/03 18:48:32 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/03 18:48:37 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/03 18:48:37 App: TestDuplicateTimerFiringKVRebOut got un-deployed
2021/12/03 18:48:38 No eventing-con process running
2021/12/03 18:48:38 Function: TestDuplicateTimerFiringKVRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringKVRebOut deleting in the background"
}
2021/12/03 18:48:39 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:48:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 18:48:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 18:48:46 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:48:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 18:48:56 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/03 18:48:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringKVRebOut (169.70s)
=== RUN   TestDuplicateTimerFiringKVSwapReb
2021/12/03 18:49:06 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/03 18:49:06 App: TestDuplicateTimerFiringKVSwapReb got un-deployed
2021/12/03 18:49:06 No eventing-con process running
2021/12/03 18:49:06 Function: TestDuplicateTimerFiringKVSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringKVSwapReb not found"
 }
}
2021/12/03 18:49:08 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:49:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 18:49:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 18:49:15 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:49:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 18:49:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:49:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 18:49:25 Deploying app: TestDuplicateTimerFiringKVSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/03 18:49:26 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringKVSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/03 18:49:31 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get deployed
2021/12/03 18:49:36 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get deployed
2021/12/03 18:49:41 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get deployed
2021/12/03 18:49:46 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get deployed
2021/12/03 18:49:46 App: TestDuplicateTimerFiringKVSwapReb got deployed
2021/12/03 18:49:46 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/03 18:49:46 Starting bucket ops to source bucket
2021/12/03 18:49:46 Removing node(s): [] from the cluster
2021/12/03 18:49:51 Rebalance progress: 12.5
2021/12/03 18:49:56 Rebalance progress: 12.5
2021/12/03 18:50:01 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:50:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:50:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 18:50:01 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/03 18:50:02 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/03 18:50:07 Rebalance progress: 12.5
2021/12/03 18:50:12 Rebalance progress: 12.5
2021/12/03 18:50:17 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:50:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:50:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 18:50:17 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/03 18:50:22 Rebalance progress: 12.5
2021/12/03 18:50:27 Rebalance progress: 12.5
2021/12/03 18:50:32 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:50:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:50:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 18:50:32 Waiting for src bucket item count to get to: 50000 curr count: 48872
2021/12/03 18:50:37 Waiting for src bucket item count to get to: 50000 curr count: 48872
2021/12/03 18:50:47 Waiting for dst bucket item count to get to: 50000 curr count: 39353
2021/12/03 18:50:52 Waiting for dst bucket item count to get to: 50000 curr count: 39353
2021/12/03 18:50:52 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/03 18:50:52 Post SET ops. Expected item count: 50000 got 50000
2021/12/03 18:50:52 Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 18:50:57 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/03 18:51:02 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/03 18:51:07 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/03 18:51:12 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/03 18:51:17 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/03 18:51:22 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/03 18:51:22 App: TestDuplicateTimerFiringKVSwapReb got un-deployed
2021/12/03 18:51:22 No eventing-con process running
2021/12/03 18:51:22 Function: TestDuplicateTimerFiringKVSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringKVSwapReb deleting in the background"
}
2021/12/03 18:51:24 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:51:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 18:51:34 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/03 18:51:39 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/03 18:51:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 18:51:41 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:51:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 18:51:51 Waiting for dst bucket item count to get to: 0 curr count: 48714
2021/12/03 18:51:56 Waiting for dst bucket item count to get to: 0 curr count: 48714
2021/12/03 18:51:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringKVSwapReb (179.87s)
=== RUN   TestKVRebOnUpdateBucketOpOneByOne
2021/12/03 18:52:06 Waiting for app: bucket_op_on_update to get un-deployed
2021/12/03 18:52:06 App: bucket_op_on_update got un-deployed
2021/12/03 18:52:06 No eventing-con process running
2021/12/03 18:52:06 Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/03 18:52:08 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:52:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 18:52:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 18:52:15 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:52:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 18:52:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:52:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 18:52:25 Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/03 18:52:26 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/03 18:52:31 Starting bucket ops to source bucket
2021/12/03 18:52:36 Waiting for app: bucket_op_on_update to get deployed
2021/12/03 18:52:41 Waiting for app: bucket_op_on_update to get deployed
2021/12/03 18:52:46 Waiting for app: bucket_op_on_update to get deployed
2021/12/03 18:52:46 App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:52:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:52:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 18:52:46 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/03 18:52:46 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/03 18:52:51 Rebalance progress: 12.5
2021/12/03 18:52:56 Rebalance progress: 12.5
2021/12/03 18:53:01 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:53:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:53:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 18:53:01 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/03 18:53:02 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/03 18:53:07 Rebalance progress: 12.5
2021/12/03 18:53:12 Rebalance progress: 12.5
2021/12/03 18:53:17 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:53:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:53:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 18:53:17 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: kv to the cluster
2021/12/03 18:53:17 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/03 18:53:22 Rebalance progress: 12.5
2021/12/03 18:53:27 Rebalance progress: 12.5
2021/12/03 18:53:32 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:53:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:53:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 18:53:32 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/03 18:53:37 Rebalance progress: 12.5
2021/12/03 18:53:42 Rebalance progress: 12.5
2021/12/03 18:53:47 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:53:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:53:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 18:53:47 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/03 18:53:52 Rebalance progress: 12.5
2021/12/03 18:53:57 Rebalance progress: 12.5
2021/12/03 18:54:02 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:54:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:54:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 18:54:02 clusterSetup.go:253: Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/03 18:54:07 Rebalance progress: 12.5
2021/12/03 18:54:12 Rebalance progress: 12.5
2021/12/03 18:54:17 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:54:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:54:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 18:54:18 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 18:54:23 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/03 18:54:28 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/03 18:54:33 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/03 18:54:38 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/03 18:54:43 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/03 18:54:48 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/03 18:54:48 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/03 18:54:48 eventingSetup.go:46: No eventing-con process running
2021/12/03 18:54:48 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/03 18:54:50 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:54:55 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 18:54:55 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 18:54:56 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:55:01 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 18:55:06 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 10654
2021/12/03 18:55:06 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebOnUpdateBucketOpOneByOne (190.34s)
=== RUN   TestKVForceFailover
2021/12/03 18:55:21 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/03 18:55:21 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/03 18:55:21 eventingSetup.go:46: No eventing-con process running
2021/12/03 18:55:21 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/03 18:55:23 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:55:28 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 18:55:28 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 18:55:30 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:55:35 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 18:55:35 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 18:55:35 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/03 18:55:35 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/03 18:55:40 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:55:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 18:55:40 Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/03 18:55:42 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/03 18:55:42 Starting bucket ops to source bucket
2021/12/03 18:55:47 Waiting for app: bucket_op_on_update to get deployed
2021/12/03 18:55:52 Waiting for app: bucket_op_on_update to get deployed
2021/12/03 18:55:57 Waiting for app: bucket_op_on_update to get deployed
2021/12/03 18:56:02 Waiting for app: bucket_op_on_update to get deployed
2021/12/03 18:56:02 App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:56:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:56:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 18:56:02 clusterSetup.go:229: Failing over: [127.0.0.1:19001]
2021/12/03 18:56:22 clusterSetup.go:237: Kicking off failover recovery, type: full
2021/12/03 18:56:22 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/03 18:56:27 Rebalance progress: 12.5
2021/12/03 18:56:32 Rebalance progress: 12.5
2021/12/03 18:56:37 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:56:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:56:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 18:56:37 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002] from the cluster
2021/12/03 18:56:42 Rebalance progress: 12.5
2021/12/03 18:56:47 Rebalance progress: 12.5
2021/12/03 18:56:52 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:56:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:56:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 18:56:53 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 18:56:58 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/03 18:57:03 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/03 18:57:08 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/03 18:57:13 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/03 18:57:18 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/03 18:57:23 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/03 18:57:23 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/03 18:57:23 eventingSetup.go:46: No eventing-con process running
2021/12/03 18:57:23 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/03 18:57:25 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:57:30 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 18:57:35 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7068
2021/12/03 18:57:35 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 18:57:36 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:57:41 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 18:57:46 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7068
2021/12/03 18:57:46 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVForceFailover (160.14s)
=== RUN   TestKVHardFailoverRecovery
2021/12/03 18:57:56 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/03 18:57:57 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/03 18:57:57 eventingSetup.go:46: No eventing-con process running
2021/12/03 18:57:57 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/03 18:57:58 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:58:03 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 18:58:03 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 18:58:05 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:58:10 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 18:58:10 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:58:15 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 18:58:15 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/03 18:58:16 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/03 18:58:21 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/03 18:58:26 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/03 18:58:31 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/03 18:58:36 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/03 18:58:36 clusterSetup.go:354: App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:58:36 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:58:36 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 18:58:36 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/03 18:58:36 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/03 18:58:42 Rebalance progress: 12.5
2021/12/03 18:58:47 Rebalance progress: 12.5
2021/12/03 18:58:52 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:58:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 18:58:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 18:58:52 clusterSetup.go:229: Failing over: [127.0.0.1:19001]
2021/12/03 18:59:52 clusterSetup.go:237: Kicking off failover recovery, type: full
2021/12/03 18:59:57 Rebalance progress: 12.5
2021/12/03 19:00:02 Rebalance progress: 12.5
2021/12/03 19:00:07 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 19:00:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 19:00:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 19:00:07 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/03 19:00:12 Rebalance progress: 12.5
2021/12/03 19:00:17 Rebalance progress: 12.5
2021/12/03 19:00:22 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 19:00:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 19:00:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 19:00:22 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 19:00:27 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/03 19:00:32 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/03 19:00:37 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/03 19:00:42 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/03 19:00:47 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/03 19:00:52 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/03 19:00:52 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/03 19:00:52 eventingSetup.go:46: No eventing-con process running
2021/12/03 19:00:52 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/03 19:00:54 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 19:00:59 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 19:01:04 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12046
2021/12/03 19:01:09 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12046
2021/12/03 19:01:09 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 19:01:11 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 19:01:16 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 19:01:21 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 10205
2021/12/03 19:01:26 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 10205
2021/12/03 19:01:26 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVHardFailoverRecovery (219.36s)
=== RUN   TestKVRebOnUpdateBucketOpAllAtOnce
2021/12/03 19:01:36 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/03 19:01:36 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/03 19:01:36 eventingSetup.go:46: No eventing-con process running
2021/12/03 19:01:36 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/03 19:01:38 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 19:01:43 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 19:01:43 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 19:01:44 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 19:01:49 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 19:01:49 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 19:01:54 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 19:01:55 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/03 19:01:56 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/03 19:02:01 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/03 19:02:06 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/03 19:02:11 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/03 19:02:16 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/03 19:02:16 clusterSetup.go:354: App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 19:02:16 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 19:02:16 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 19:02:16 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/03 19:02:16 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/03 19:02:17 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: kv to the cluster
2021/12/03 19:02:17 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/03 19:02:22 Rebalance progress: 12.5
2021/12/03 19:02:27 Rebalance progress: 12.5
2021/12/03 19:02:32 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 19:02:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 19:02:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 19:02:32 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002 127.0.0.1:19003] from the cluster
2021/12/03 19:02:37 Rebalance progress: 12.5
2021/12/03 19:02:42 Rebalance progress: 12.5
2021/12/03 19:02:47 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 19:02:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 19:02:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 19:02:47 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 19:02:52 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/03 19:02:58 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/03 19:03:03 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/03 19:03:08 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/03 19:03:13 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/03 19:03:18 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/03 19:03:18 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/03 19:03:18 eventingSetup.go:46: No eventing-con process running
2021/12/03 19:03:18 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/03 19:03:20 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 19:03:25 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 19:03:25 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 19:03:26 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 19:03:31 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 19:03:36 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4666
2021/12/03 19:03:36 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebOnUpdateBucketOpAllAtOnce (130.61s)
=== RUN   TestKVSwapRebOnUpdateBucketOp
2021/12/03 19:03:46 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/03 19:03:46 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/03 19:03:46 eventingSetup.go:46: No eventing-con process running
2021/12/03 19:03:46 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/03 19:03:48 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 19:03:53 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 19:03:53 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 19:03:55 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 19:04:00 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 19:04:00 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 19:04:00 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 19:04:00 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/03 19:04:01 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/03 19:04:06 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/03 19:04:11 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/03 19:04:16 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/03 19:04:21 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/03 19:04:21 clusterSetup.go:354: App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 19:04:21 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 19:04:21 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 19:04:21 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/03 19:04:22 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/03 19:04:27 Rebalance progress: 12.5
2021/12/03 19:04:32 Rebalance progress: 12.5
2021/12/03 19:04:37 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 19:04:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 19:04:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 19:04:37 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/03 19:04:37 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/03 19:04:42 Rebalance progress: 12.5
2021/12/03 19:04:47 Rebalance progress: 12.5
2021/12/03 19:04:52 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 19:04:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 19:04:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 19:04:53 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/03 19:04:58 Rebalance progress: 12.5
2021/12/03 19:05:03 Rebalance progress: 12.5
2021/12/03 19:05:08 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 19:05:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 19:05:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 19:05:08 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 19:05:13 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/03 19:05:18 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/03 19:05:23 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/03 19:05:28 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/03 19:05:33 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/03 19:05:33 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/03 19:05:33 eventingSetup.go:46: No eventing-con process running
2021/12/03 19:05:33 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/03 19:05:35 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 19:05:40 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 19:05:45 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6131
2021/12/03 19:05:45 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 19:05:47 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 19:05:52 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 19:05:57 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6131
2021/12/03 19:05:57 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 19:06:02 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/03 19:06:02 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/03 19:06:02 eventingSetup.go:46: No eventing-con process running
2021/12/03 19:06:02 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/03 19:06:04 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 19:06:09 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 19:06:09 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 19:06:10 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 19:06:15 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 19:06:15 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVSwapRebOnUpdateBucketOp (158.87s)
=== RUN   TestKVRebalanceWithMultipleHandlers
2021/12/03 19:06:25 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/03 19:06:25 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/03 19:06:25 eventingSetup.go:46: No eventing-con process running
2021/12/03 19:06:25 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/03 19:06:27 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 19:06:32 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 19:06:32 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 19:06:34 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 19:06:39 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 19:06:39 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 19:06:44 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/03 19:06:44 clusterSetup.go:405: App: sys_test_bucket_op got un-deployed
2021/12/03 19:06:44 eventingSetup.go:46: No eventing-con process running
2021/12/03 19:06:44 eventingSetup.go:418: Function: sys_test_bucket_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: sys_test_bucket_op not found"
 }
}
2021/12/03 19:06:46 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 19:06:51 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 19:06:51 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 19:06:53 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 19:06:58 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 19:06:58 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 19:06:58 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 19:06:58 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/03 19:06:59 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 19:06:59 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 19:06:59 eventingSetup.go:125: Deploying app: sys_test_bucket_op. Item count src bucket: 0 dst bucket: 0
2021/12/03 19:07:00 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'sys_test_bucket_op' in metakv",
  "warnings": null
 }
}
2021/12/03 19:07:05 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/03 19:07:10 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/03 19:07:15 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/03 19:07:20 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/03 19:07:20 clusterSetup.go:354: App: bucket_op_on_update got deployed
2021/12/03 19:07:25 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/03 19:07:30 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/03 19:07:35 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/03 19:07:35 clusterSetup.go:354: App: sys_test_bucket_op got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 19:07:35 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 19:07:35 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 19:07:35 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/03 19:07:35 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/03 19:07:40 Rebalance progress: 12.5
2021/12/03 19:07:45 Rebalance progress: 12.5
2021/12/03 19:07:50 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 19:07:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 19:07:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 19:07:50 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/03 19:07:51 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/03 19:07:56 Rebalance progress: 12.5
2021/12/03 19:08:01 Rebalance progress: 12.5
2021/12/03 19:08:06 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 19:08:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 19:08:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 19:08:06 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: kv to the cluster
2021/12/03 19:08:06 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/03 19:08:11 Rebalance progress: 12.5
2021/12/03 19:08:16 Rebalance progress: 12.5
2021/12/03 19:08:21 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 19:08:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 19:08:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 19:08:21 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/03 19:08:27 Rebalance progress: 12.5
2021/12/03 19:08:32 Rebalance progress: 12.5
2021/12/03 19:08:37 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 19:08:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 19:08:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 19:08:37 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/03 19:08:42 Rebalance progress: 12.5
2021/12/03 19:08:47 Rebalance progress: 12.5
2021/12/03 19:08:52 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 19:08:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 19:08:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 19:08:52 clusterSetup.go:253: Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/03 19:08:57 Rebalance progress: 12.5
2021/12/03 19:09:02 Rebalance progress: 12.5
2021/12/03 19:09:07 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 19:09:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 19:09:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 19:09:07 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 19:09:12 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/03 19:09:17 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/03 19:09:22 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/03 19:09:27 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/03 19:09:32 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/03 19:09:37 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/03 19:09:37 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/03 19:09:37 eventingSetup.go:46: No eventing-con process running
2021/12/03 19:09:37 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/03 19:09:40 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 19:09:45 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 19:09:50 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2384
2021/12/03 19:09:55 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2384
2021/12/03 19:09:55 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 19:09:57 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 19:10:02 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 19:10:07 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12160
2021/12/03 19:10:07 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 19:10:07 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 19:10:12 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/03 19:10:17 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/03 19:10:22 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/03 19:10:28 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/03 19:10:33 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/03 19:10:33 clusterSetup.go:405: App: sys_test_bucket_op got un-deployed
2021/12/03 19:10:33 eventingSetup.go:46: No eventing-con process running
2021/12/03 19:10:33 eventingSetup.go:418: Function: sys_test_bucket_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: sys_test_bucket_op deleting in the background"
}
2021/12/03 19:10:35 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 19:10:40 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 19:10:40 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 19:10:41 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 19:10:46 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 19:10:46 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebalanceWithMultipleHandlers (270.97s)
=== RUN   TestKVSwapRebalanceWithMultipleHandlers
2021/12/03 19:10:56 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/03 19:10:56 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/03 19:10:56 eventingSetup.go:46: No eventing-con process running
2021/12/03 19:10:56 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/03 19:10:58 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 19:11:03 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 19:11:03 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 19:11:05 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 19:11:10 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 19:11:10 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 19:11:15 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/03 19:11:15 clusterSetup.go:405: App: sys_test_bucket_op got un-deployed
2021/12/03 19:11:15 eventingSetup.go:46: No eventing-con process running
2021/12/03 19:11:15 eventingSetup.go:418: Function: sys_test_bucket_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: sys_test_bucket_op not found"
 }
}
2021/12/03 19:11:17 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 19:11:22 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 19:11:22 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 19:11:23 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 19:11:28 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 19:11:28 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 19:11:28 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 19:11:28 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/03 19:11:30 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 19:11:30 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 19:11:30 eventingSetup.go:125: Deploying app: sys_test_bucket_op. Item count src bucket: 0 dst bucket: 0
2021/12/03 19:11:31 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'sys_test_bucket_op' in metakv",
  "warnings": null
 }
}
2021/12/03 19:11:36 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/03 19:11:41 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/03 19:11:46 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/03 19:11:51 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/03 19:11:51 clusterSetup.go:354: App: bucket_op_on_update got deployed
2021/12/03 19:11:56 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/03 19:12:01 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/03 19:12:06 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/03 19:12:06 clusterSetup.go:354: App: sys_test_bucket_op got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 19:12:06 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 19:12:06 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 19:12:06 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/03 19:12:06 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/03 19:12:11 Rebalance progress: 12.5
2021/12/03 19:12:16 Rebalance progress: 12.5
2021/12/03 19:12:21 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 19:12:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 19:12:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 19:12:21 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/03 19:12:22 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/03 19:12:27 Rebalance progress: 12.5
2021/12/03 19:12:32 Rebalance progress: 12.5
2021/12/03 19:12:37 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 19:12:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 19:12:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 19:12:37 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/03 19:12:42 Rebalance progress: 12.5
2021/12/03 19:12:47 Rebalance progress: 12.5
2021/12/03 19:12:52 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 19:12:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 19:12:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 19:12:52 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 19:12:57 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/03 19:13:02 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/03 19:13:07 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/03 19:13:12 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/03 19:13:17 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/03 19:13:22 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/03 19:13:22 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/03 19:13:22 eventingSetup.go:46: No eventing-con process running
2021/12/03 19:13:22 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/03 19:13:25 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 19:13:30 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 19:13:35 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7589
2021/12/03 19:13:35 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 19:13:37 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 19:13:42 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 19:13:47 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7589
2021/12/03 19:13:47 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 19:13:48 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 19:13:53 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/03 19:13:58 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/03 19:14:03 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/03 19:14:08 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/03 19:14:13 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/03 19:14:18 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/03 19:14:18 clusterSetup.go:405: App: sys_test_bucket_op got un-deployed
2021/12/03 19:14:18 eventingSetup.go:46: No eventing-con process running
2021/12/03 19:14:18 eventingSetup.go:418: Function: sys_test_bucket_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: sys_test_bucket_op deleting in the background"
}
2021/12/03 19:14:20 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 19:14:25 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 19:14:25 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 19:14:26 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 19:14:31 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 19:14:31 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVSwapRebalanceWithMultipleHandlers (225.18s)
=== RUN   TestKVRebStopStartKVOpsOnUpdateBucketOpOneByOne
2021/12/03 19:14:41 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/03 19:14:42 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/03 19:14:42 eventingSetup.go:46: No eventing-con process running
2021/12/03 19:14:42 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/03 19:14:43 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 19:14:48 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 19:14:48 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 19:14:50 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 19:14:55 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 19:14:55 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 19:15:00 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 19:15:00 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/03 19:15:01 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/03 19:15:06 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/03 19:15:11 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/03 19:15:16 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/03 19:15:21 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/03 19:15:21 clusterSetup.go:354: App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 19:15:21 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 19:15:21 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 19:15:21 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/03 19:15:22 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/03 19:15:42 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 19:15:42 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 19:15:42 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 19:15:52 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 19:15:52 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 19:15:52 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/03 19:15:57 Rebalance progress: 12.5
2021/12/03 19:16:02 Rebalance progress: 12.5
2021/12/03 19:16:07 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 19:16:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 19:16:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 19:16:07 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/03 19:16:08 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/03 19:16:13 Rebalance progress: 12.5
2021/12/03 19:16:18 Rebalance progress: 12.5
2021/12/03 19:16:23 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 19:16:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 19:16:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 19:16:23 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/03 19:16:28 Rebalance progress: 12.5
2021/12/03 19:16:33 Rebalance progress: 12.5
2021/12/03 19:16:38 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 19:16:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 19:16:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 19:16:38 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 19:16:43 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/03 19:16:48 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/03 19:16:53 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/03 19:16:58 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/03 19:17:03 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/03 19:17:03 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/03 19:17:03 eventingSetup.go:46: No eventing-con process running
2021/12/03 19:17:03 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/03 19:17:05 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 19:17:10 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 19:17:15 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 9145
2021/12/03 19:17:15 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 19:17:17 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 19:17:22 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 19:17:27 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 9145
2021/12/03 19:17:27 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebStopStartKVOpsOnUpdateBucketOpOneByOne (175.50s)
=== RUN   TestKVFailoverOnUpdateBucketOp
2021/12/03 19:17:37 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/03 19:17:37 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/03 19:17:37 eventingSetup.go:46: No eventing-con process running
2021/12/03 19:17:37 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/03 19:17:39 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 19:17:44 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 19:17:44 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 19:17:46 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 19:17:51 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 19:17:51 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 19:17:56 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 19:17:56 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/03 19:17:57 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/03 19:18:02 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/03 19:18:07 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/03 19:18:12 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/03 19:18:17 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/03 19:18:17 clusterSetup.go:354: App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 19:18:17 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 19:18:17 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 19:18:17 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/03 19:18:17 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/03 19:18:22 Rebalance progress: 12.5
2021/12/03 19:18:27 Rebalance progress: 12.5
2021/12/03 19:18:32 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 19:18:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 19:18:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 19:18:32 clusterSetup.go:229: Failing over: [127.0.0.1:19001]
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 19:18:42 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 19:18:42 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 19:18:42 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/03 19:18:47 Rebalance progress: 12.5
2021/12/03 19:18:52 Rebalance progress: 12.5
2021/12/03 19:18:57 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 19:18:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 19:18:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 19:18:57 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/03 19:18:58 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/03 19:19:03 Rebalance progress: 12.5
2021/12/03 19:19:08 Rebalance progress: 12.5
2021/12/03 19:19:13 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 19:19:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 19:19:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 19:19:13 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/03 19:19:18 Rebalance progress: 12.5
2021/12/03 19:19:23 Rebalance progress: 12.5
2021/12/03 19:19:28 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 19:19:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 19:19:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 19:19:28 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 19:19:33 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/03 19:19:38 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/03 19:19:43 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/03 19:19:48 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/03 19:19:53 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/03 19:19:58 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/03 19:19:58 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/03 19:19:58 eventingSetup.go:46: No eventing-con process running
2021/12/03 19:19:58 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/03 19:20:00 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 19:20:05 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 19:20:10 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 68
2021/12/03 19:20:15 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 68
2021/12/03 19:20:15 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 19:20:17 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 19:20:22 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 19:20:27 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 8577
2021/12/03 19:20:27 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVFailoverOnUpdateBucketOp (179.76s)
=== RUN   TestBootstrapAfterKVHardFailover
2021/12/03 19:20:32 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/03 19:20:32 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/03 19:20:37 Rebalance progress: 100
2021/12/03 19:20:37 Failing over: [127.0.0.1:19001]
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 19:21:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 19:21:07 Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/03 19:21:08 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/03 19:21:13 Removing node(s): [] from the cluster
2021/12/03 19:21:13 Starting bucket ops to source bucket
2021/12/03 19:21:18 Rebalance failed. See logs for detailed reason. You can try again.
2021/12/03 19:21:23 Waiting for app: bucket_op_on_update to get deployed
2021/12/03 19:21:28 Waiting for app: bucket_op_on_update to get deployed
2021/12/03 19:21:28 App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 19:21:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 19:21:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/03 19:21:29 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/03 19:21:34 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/03 19:21:39 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/03 19:21:44 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/03 19:21:49 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/03 19:21:54 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/03 19:21:59 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/03 19:21:59 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/03 19:21:59 eventingSetup.go:46: No eventing-con process running
2021/12/03 19:21:59 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/03 19:22:01 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 19:22:06 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 19:22:11 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 996
2021/12/03 19:22:16 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 996
2021/12/03 19:22:16 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/03 19:22:17 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/03 19:22:22 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/03 19:22:27 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1512
2021/12/03 19:22:27 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestBootstrapAfterKVHardFailover (120.72s)
PASS
ok  	github.com/couchbase/eventing/tests/functional_tests	12136.470s

Rebalance tests took 3 hours, 22 minutes and 19 seconds.

Testing Succeeded


Version: versions-03.12.2021-11.39.cfg
Build Log: make-03.12.2021-11.39.log
Server Log: logs-03.12.2021-11.39.tar.gz

Finished