Building

Started building at 2021/12/17 02:11:04
Calculating base
Updating mirror
Basing run on 7.1.0-1930 20f953064
Updating tree for run 17.12.2021-02.11
gocb is at 635fa6c, changes since last good build: none
libcouchbase is at f6822fc, changes since last good build: none
indexing is at f0d7a1e, changes since last good build: none
kv_engine is at f0adfd6, changes since last good build: 
 f0adfd6c3 Change the cert_auth_config in testapp to the way ns_server provides it
 7f8f27128 Remove leak from jemalloc_allocator_stats
ns_server is at 4146c32, changes since last good build: 
 4146c3289 MB-48561 Upgrade ns_config from 7.0 to 7.1 after chronicle.
 6df0a163e Revert "MB-49945: add importMap plugin for esbuild"
 983fb63c7 MB-49945: enable mistakenly disabled minification
 2b49582f7 MB-49945: add importMap plugin for esbuild
 9743b2b7b MB-49945: add printErrorAndExit func
 b2bba6ff7 MB-49945: use type=importmap-shim instead of importmap
 cc8a522ff MB-49945: upgrade es-module-shims and use it's minified version
 85c2bc0f3 MB-49086 Add inline spinners to angular pages
 71f65437a MB-49681 Disable 'install sample buckets' in mixed mode
couchstore is at 70053f3, changes since last good build: none
couchdb is at fca2e6b, changes since last good build: none
testrunner is at a657fc6, changes since last good build: 
 a657fc6ba Optimise base setup and teardown
 eeff10873 XDCR system event logs
Switching eventing to unstable
eventing is at 7f52754, changes since last good build: none
Switching eventing-ee to unstable
eventing-ee is at 3f826e0, changes since last good build: none
Pulling in uncommitted change 160397 at refs/changes/97/160397/3
Total 23 (delta 15), reused 17 (delta 15)
[unstable a23dd04a] MB-100 : Do not fail n1ql timeout test if we are catching timeout exceptions successfully
 Author: abhijpes 
 Date: Mon Aug 30 08:24:38 2021 +0530
 1 file changed, 2 insertions(+), 2 deletions(-)
Pulling in uncommitted change 163449 at refs/changes/49/163449/1
Total 48 (delta 27), reused 39 (delta 27)
[unstable 6c87b3ed] MB-100: collect data when test fails
 Author: AnkitPrabhu 
 Date: Tue Oct 12 14:02:31 2021 +0530
 1 file changed, 2 insertions(+)
Pulling in uncommitted change 165976 at refs/changes/76/165976/18
Total 4 (delta 3), reused 4 (delta 3)
[unstable 5093cfe4] MB-48538: Add new audit ids for different events
 Author: AnkitPrabhu 
 Date: Tue Nov 23 14:29:10 2021 +0530
 1 file changed, 337 insertions(+), 61 deletions(-)
Pulling in uncommitted change 166698 at refs/changes/98/166698/3
Total 4 (delta 3), reused 4 (delta 3)
[unstable a260a42e] MB-48334 : Fail rebalance unless all functions deployed on current encryption level
 Author: abhijpes 
 Date: Wed Dec 1 14:51:29 2021 +0530
 1 file changed, 9 insertions(+)
Pulling in uncommitted change 167196 at refs/changes/96/167196/8
Total 19 (delta 15), reused 18 (delta 15)
[unstable 938c1555] MB-49720: Store failover log in checkpoint blob
 Author: AnkitPrabhu 
 Date: Wed Dec 8 15:16:22 2021 +0530
 5 files changed, 158 insertions(+), 72 deletions(-)
Pulling in uncommitted change 167542 at refs/changes/42/167542/5
Total 44 (delta 38), reused 44 (delta 38)
[unstable 9424a43e] MB-49570: Start dcp stream for a single collection id
 Author: AnkitPrabhu 
 Date: Tue Dec 14 14:27:25 2021 +0530
 7 files changed, 52 insertions(+), 29 deletions(-)
Pulling in uncommitted change 167544 at refs/changes/44/167544/4
Total 52 (delta 44), reused 52 (delta 44)
[unstable 44a6c6cd] MB-49570: Add backlog stats on ui
 Author: AnkitPrabhu 
 Date: Tue Dec 14 14:28:01 2021 +0530
 2 files changed, 18 insertions(+), 1 deletion(-)
Building cmakefiles and deps
Building main product

Testing

Started testing at 2021/12/17 02:27:05
Testing mode: rebalance,n1ql,functional
Starting server
Starting server: attempt 1

Functional tests

2021/12/17 02:28:33 Started functional
2021/12/17 02:28:42 Adding node: https://127.0.0.1:19001 with role: kv,index,n1ql to the cluster
2021/12/17 02:28:48 Error in adding nodes : , response: {"otpNode":"n_1@127.0.0.1"}
2021/12/17 02:28:48 Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/17 02:28:53 Removing node(s): [] from the cluster
2021/12/17 02:28:58 Rebalance progress: 0
2021/12/17 02:29:03 Rebalance progress: 0
2021/12/17 02:29:08 Rebalance progress: 4.62962962962963
2021/12/17 02:29:13 Rebalance progress: 9.25925925925926
2021/12/17 02:29:18 Rebalance progress: 12.73148148148148
2021/12/17 02:29:23 Rebalance progress: 17.36111111111111
2021/12/17 02:29:28 Rebalance progress: 20.68865740740741
2021/12/17 02:29:33 Rebalance progress: 25.19169560185185
2021/12/17 02:29:38 Rebalance progress: 100
2021/12/17 02:29:41 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/17 02:29:56 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/17 02:30:11 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/17 02:30:26 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/17 02:30:41 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/17 02:30:56 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/17 02:31:11 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/17 02:31:26 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/17 02:31:41 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/17 02:31:56 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/17 02:32:11 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/17 02:32:26 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/17 02:32:41 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/17 02:32:56 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/17 02:33:11 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/17 02:33:26 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/17 02:33:41 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/17 02:33:56 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
=== RUN   TestAdvancedGetOps
2021/12/17 02:34:13 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/17 02:34:28 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/17 02:34:43 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/17 02:34:58 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/17 02:35:13 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/17 02:35:28 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/17 02:35:43 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/17 02:35:58 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/17 02:36:13 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/17 02:36:28 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/17 02:36:43 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/17 02:36:58 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/17 02:37:13 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/17 02:37:28 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/17 02:37:43 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/17 02:37:58 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/17 02:38:13 Deploying app: TestAdvancedGetOps. Item count src bucket: 0 dst bucket: 0
2021/12/17 02:38:15 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedGetOps' in metakv",
  "warnings": null
 }
}
2021/12/17 02:38:20 Waiting for app: TestAdvancedGetOps to get deployed
2021/12/17 02:38:25 Waiting for app: TestAdvancedGetOps to get deployed
2021/12/17 02:38:30 Waiting for app: TestAdvancedGetOps to get deployed
2021/12/17 02:38:35 Waiting for app: TestAdvancedGetOps to get deployed
2021/12/17 02:38:35 App: TestAdvancedGetOps got deployed
2021/12/17 02:38:35 Starting bucket ops to source bucket
2021/12/17 02:38:40 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/17 02:38:45 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/17 02:38:45 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/17 02:38:45 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 02:38:45 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/17 02:38:45 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 02:38:50 Waiting for app: TestAdvancedGetOps to get un-deployed
2021/12/17 02:38:55 Waiting for app: TestAdvancedGetOps to get un-deployed
2021/12/17 02:39:00 Waiting for app: TestAdvancedGetOps to get un-deployed
2021/12/17 02:39:05 Waiting for app: TestAdvancedGetOps to get un-deployed
2021/12/17 02:39:05 App: TestAdvancedGetOps got un-deployed
2021/12/17 02:39:05 No eventing-con process running
2021/12/17 02:39:05 Function: TestAdvancedGetOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedGetOps deleting in the background"
}
2021/12/17 02:39:07 Context: Bucket flush, response code: 200 dump: 
2021/12/17 02:39:17 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/17 02:39:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 02:39:19 Context: Bucket flush, response code: 200 dump: 
2021/12/17 02:39:29 Waiting for dst bucket item count to get to: 0 curr count: 82
2021/12/17 02:39:34 Waiting for dst bucket item count to get to: 0 curr count: 32
2021/12/17 02:39:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedGetOps (320.18s)
=== RUN   TestAdvancedInsertOps
2021/12/17 02:39:34 Deploying app: TestAdvancedInsertOps. Item count src bucket: 0 dst bucket: 0
2021/12/17 02:39:35 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedInsertOps' in metakv",
  "warnings": null
 }
}
2021/12/17 02:39:40 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/17 02:39:45 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/17 02:39:50 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/17 02:39:55 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/17 02:39:55 App: TestAdvancedInsertOps got deployed
2021/12/17 02:39:55 Starting bucket ops to source bucket
2021/12/17 02:40:01 Waiting for dst bucket item count to get to: 1024 curr count: 0
2021/12/17 02:40:06 Waiting for dst bucket item count to get to: 1024 curr count: 512
2021/12/17 02:40:06 src & dst bucket item count matched up. src bucket count: 1024 dst bucket count: 1024
2021/12/17 02:40:06 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 02:40:06 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/17 02:40:06 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 02:40:11 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/17 02:40:16 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/17 02:40:21 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/17 02:40:26 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/17 02:40:26 App: TestAdvancedInsertOps got un-deployed
2021/12/17 02:40:26 No eventing-con process running
2021/12/17 02:40:26 Function: TestAdvancedInsertOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedInsertOps deleting in the background"
}
2021/12/17 02:40:28 Context: Bucket flush, response code: 200 dump: 
2021/12/17 02:40:38 Waiting for dst bucket item count to get to: 0 curr count: 512
2021/12/17 02:40:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 02:40:39 Context: Bucket flush, response code: 200 dump: 
2021/12/17 02:40:49 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/17 02:40:54 Waiting for dst bucket item count to get to: 0 curr count: 512
2021/12/17 02:40:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 02:40:54 Testing insert operation on source bucket
2021/12/17 02:40:54 Deploying app: TestAdvancedInsertOps. Item count src bucket: 0 dst bucket: 0
2021/12/17 02:40:56 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedInsertOps' in metakv",
  "warnings": null
 }
}
2021/12/17 02:41:01 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/17 02:41:06 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/17 02:41:11 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/17 02:41:16 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/17 02:41:16 App: TestAdvancedInsertOps got deployed
2021/12/17 02:41:16 Starting bucket ops to source bucket
2021/12/17 02:41:22 Waiting for dst bucket item count to get to: 2048 curr count: 0
2021/12/17 02:41:22 src & dst bucket item count matched up. src bucket count: 2048 dst bucket count: 2048
2021/12/17 02:41:22 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 02:41:22 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/17 02:41:22 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 02:41:27 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/17 02:41:32 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/17 02:41:37 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/17 02:41:42 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/17 02:41:42 App: TestAdvancedInsertOps got un-deployed
2021/12/17 02:41:42 No eventing-con process running
2021/12/17 02:41:42 Function: TestAdvancedInsertOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedInsertOps deleting in the background"
}
2021/12/17 02:41:43 Context: Bucket flush, response code: 200 dump: 
2021/12/17 02:41:53 Waiting for dst bucket item count to get to: 0 curr count: 2048
2021/12/17 02:41:58 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/17 02:41:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 02:42:00 Context: Bucket flush, response code: 200 dump: 
2021/12/17 02:42:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedInsertOps (151.76s)
=== RUN   TestAdvancedUpsertOps
2021/12/17 02:42:05 Starting bucket ops to source bucket
2021/12/17 02:42:06 Deploying app: TestAdvancedUpsertOps. Item count src bucket: 0 dst bucket: 0
2021/12/17 02:42:07 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedUpsertOps' in metakv",
  "warnings": null
 }
}
2021/12/17 02:42:12 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/17 02:42:17 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/17 02:42:22 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/17 02:42:27 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/17 02:42:27 App: TestAdvancedUpsertOps got deployed
2021/12/17 02:42:27 Starting bucket ops to source bucket
2021/12/17 02:42:33 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/17 02:42:38 Waiting for dst bucket item count to get to: 0 curr count: 512
2021/12/17 02:42:43 Waiting for dst bucket item count to get to: 0 curr count: 512
2021/12/17 02:42:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 02:42:43 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 02:42:43 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/17 02:42:43 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 02:42:48 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/17 02:42:53 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/17 02:42:58 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/17 02:43:03 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/17 02:43:03 App: TestAdvancedUpsertOps got un-deployed
2021/12/17 02:43:03 No eventing-con process running
2021/12/17 02:43:03 Function: TestAdvancedUpsertOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedUpsertOps deleting in the background"
}
2021/12/17 02:43:05 Context: Bucket flush, response code: 200 dump: 
2021/12/17 02:43:15 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/17 02:43:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 02:43:17 Context: Bucket flush, response code: 200 dump: 
2021/12/17 02:43:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 02:43:22 Testing upsert operation on source bucket
2021/12/17 02:43:22 Deploying app: TestAdvancedUpsertOps. Item count src bucket: 0 dst bucket: 0
2021/12/17 02:43:23 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedUpsertOps' in metakv",
  "warnings": null
 }
}
2021/12/17 02:43:28 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/17 02:43:33 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/17 02:43:38 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/17 02:43:43 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/17 02:43:43 App: TestAdvancedUpsertOps got deployed
2021/12/17 02:43:43 Starting bucket ops to source bucket
2021/12/17 02:43:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 02:43:44 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 02:43:44 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/17 02:43:44 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 02:43:49 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/17 02:43:54 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/17 02:43:59 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/17 02:44:04 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/17 02:44:04 App: TestAdvancedUpsertOps got un-deployed
2021/12/17 02:44:04 No eventing-con process running
2021/12/17 02:44:04 Function: TestAdvancedUpsertOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedUpsertOps deleting in the background"
}
2021/12/17 02:44:06 Context: Bucket flush, response code: 200 dump: 
2021/12/17 02:44:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 02:44:13 Context: Bucket flush, response code: 200 dump: 
2021/12/17 02:44:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedUpsertOps (133.10s)
=== RUN   TestAdvancedReplaceOps
2021/12/17 02:44:18 Starting bucket ops to source bucket
2021/12/17 02:44:19 Deploying app: TestAdvancedReplaceOps. Item count src bucket: 0 dst bucket: 0
2021/12/17 02:44:20 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedReplaceOps' in metakv",
  "warnings": null
 }
}
2021/12/17 02:44:25 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/17 02:44:30 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/17 02:44:35 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/17 02:44:40 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/17 02:44:40 App: TestAdvancedReplaceOps got deployed
2021/12/17 02:44:40 Starting bucket ops to source bucket
2021/12/17 02:44:47 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/17 02:44:52 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/17 02:44:57 Waiting for dst bucket item count to get to: 0 curr count: 512
2021/12/17 02:44:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 02:44:57 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 02:44:57 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/17 02:44:57 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 02:45:02 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/17 02:45:07 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/17 02:45:12 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/17 02:45:17 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/17 02:45:17 App: TestAdvancedReplaceOps got un-deployed
2021/12/17 02:45:17 No eventing-con process running
2021/12/17 02:45:17 Function: TestAdvancedReplaceOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedReplaceOps deleting in the background"
}
2021/12/17 02:45:18 Context: Bucket flush, response code: 200 dump: 
2021/12/17 02:45:28 Waiting for dst bucket item count to get to: 0 curr count: 875
2021/12/17 02:45:33 Waiting for dst bucket item count to get to: 0 curr count: 363
2021/12/17 02:45:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 02:45:35 Context: Bucket flush, response code: 200 dump: 
2021/12/17 02:45:40 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 02:45:40 Testing upsert operation on source bucket
2021/12/17 02:45:40 Deploying app: TestAdvancedReplaceOps. Item count src bucket: 0 dst bucket: 0
2021/12/17 02:45:41 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedReplaceOps' in metakv",
  "warnings": null
 }
}
2021/12/17 02:45:46 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/17 02:45:51 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/17 02:45:57 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/17 02:46:02 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/17 02:46:02 App: TestAdvancedReplaceOps got deployed
2021/12/17 02:46:02 Starting bucket ops to source bucket
2021/12/17 02:46:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 02:46:03 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 02:46:03 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/17 02:46:03 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 02:46:08 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/17 02:46:13 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/17 02:46:18 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/17 02:46:23 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/17 02:46:23 App: TestAdvancedReplaceOps got un-deployed
2021/12/17 02:46:23 No eventing-con process running
2021/12/17 02:46:23 Function: TestAdvancedReplaceOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedReplaceOps deleting in the background"
}
2021/12/17 02:46:25 Context: Bucket flush, response code: 200 dump: 
2021/12/17 02:46:35 Waiting for dst bucket item count to get to: 0 curr count: 190
2021/12/17 02:46:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 02:46:36 Context: Bucket flush, response code: 200 dump: 
2021/12/17 02:46:41 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedReplaceOps (143.03s)
=== RUN   TestAdvancedDeleteOps
2021/12/17 02:46:41 Starting bucket ops to source bucket
2021/12/17 02:46:42 Deploying app: TestAdvancedDeleteOps. Item count src bucket: 0 dst bucket: 0
2021/12/17 02:46:43 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedDeleteOps' in metakv",
  "warnings": null
 }
}
2021/12/17 02:46:48 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/17 02:46:53 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/17 02:46:58 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/17 02:47:03 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/17 02:47:03 App: TestAdvancedDeleteOps got deployed
2021/12/17 02:47:03 Starting bucket ops to source bucket
2021/12/17 02:47:08 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/17 02:47:13 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/17 02:47:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 02:47:13 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 02:47:13 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/17 02:47:13 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 02:47:18 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/17 02:47:23 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/17 02:47:28 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/17 02:47:33 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/17 02:47:33 App: TestAdvancedDeleteOps got un-deployed
2021/12/17 02:47:33 No eventing-con process running
2021/12/17 02:47:33 Function: TestAdvancedDeleteOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedDeleteOps deleting in the background"
}
2021/12/17 02:47:35 Context: Bucket flush, response code: 200 dump: 
2021/12/17 02:47:45 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/17 02:47:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 02:47:47 Context: Bucket flush, response code: 200 dump: 
2021/12/17 02:47:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 02:47:52 Testing delete operation on source bucket
2021/12/17 02:47:52 Deploying app: TestAdvancedDeleteOps. Item count src bucket: 0 dst bucket: 0
2021/12/17 02:47:53 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedDeleteOps' in metakv",
  "warnings": null
 }
}
2021/12/17 02:47:58 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/17 02:48:03 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/17 02:48:08 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/17 02:48:13 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/17 02:48:13 App: TestAdvancedDeleteOps got deployed
2021/12/17 02:48:13 Starting bucket ops to source bucket
2021/12/17 02:48:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 02:48:13 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 02:48:13 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/17 02:48:13 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 02:48:18 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/17 02:48:23 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/17 02:48:28 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/17 02:48:33 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/17 02:48:33 App: TestAdvancedDeleteOps got un-deployed
2021/12/17 02:48:33 No eventing-con process running
2021/12/17 02:48:33 Function: TestAdvancedDeleteOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedDeleteOps deleting in the background"
}
2021/12/17 02:48:35 Context: Bucket flush, response code: 200 dump: 
2021/12/17 02:48:40 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 02:48:42 Context: Bucket flush, response code: 200 dump: 
2021/12/17 02:48:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedDeleteOps (125.62s)
=== RUN   TestEnoentAdvancedGet
2021/12/17 02:48:47 Deploying app: TestEnoentAdvancedGet. Item count src bucket: 0 dst bucket: 0
2021/12/17 02:48:48 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedGet' in metakv",
  "warnings": null
 }
}
2021/12/17 02:48:53 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/17 02:48:58 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/17 02:49:03 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/17 02:49:08 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/17 02:49:08 App: TestEnoentAdvancedGet got deployed
2021/12/17 02:49:08 Starting bucket ops to source bucket
2021/12/17 02:49:13 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/17 02:49:18 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/17 02:49:18 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/17 02:49:18 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 02:49:18 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/17 02:49:18 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 02:49:23 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/17 02:49:28 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/17 02:49:33 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/17 02:49:38 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/17 02:49:38 App: TestEnoentAdvancedGet got un-deployed
2021/12/17 02:49:39 No eventing-con process running
2021/12/17 02:49:39 Function: TestEnoentAdvancedGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedGet deleting in the background"
}
2021/12/17 02:49:40 Context: Bucket flush, response code: 200 dump: 
2021/12/17 02:49:50 Waiting for dst bucket item count to get to: 0 curr count: 72
2021/12/17 02:49:55 Waiting for dst bucket item count to get to: 0 curr count: 72
2021/12/17 02:49:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 02:49:57 Context: Bucket flush, response code: 200 dump: 
2021/12/17 02:50:07 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/17 02:50:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 02:50:07 Testing get enoent operation on source bucket
2021/12/17 02:50:07 Deploying app: TestEnoentAdvancedGet. Item count src bucket: 0 dst bucket: 0
2021/12/17 02:50:08 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedGet' in metakv",
  "warnings": null
 }
}
2021/12/17 02:50:13 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/17 02:50:18 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/17 02:50:23 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/17 02:50:28 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/17 02:50:28 App: TestEnoentAdvancedGet got deployed
2021/12/17 02:50:28 Starting bucket ops to source bucket
2021/12/17 02:50:33 Waiting for dst bucket item count to get to: 200 curr count: 0
2021/12/17 02:50:38 Waiting for dst bucket item count to get to: 200 curr count: 100
2021/12/17 02:50:38 src & dst bucket item count matched up. src bucket count: 200 dst bucket count: 200
2021/12/17 02:50:38 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 02:50:38 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/17 02:50:38 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 02:50:43 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/17 02:50:48 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/17 02:50:53 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/17 02:50:59 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/17 02:50:59 App: TestEnoentAdvancedGet got un-deployed
2021/12/17 02:50:59 No eventing-con process running
2021/12/17 02:50:59 Function: TestEnoentAdvancedGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedGet deleting in the background"
}
2021/12/17 02:51:00 Context: Bucket flush, response code: 200 dump: 
2021/12/17 02:51:10 Waiting for dst bucket item count to get to: 0 curr count: 147
2021/12/17 02:51:15 Waiting for dst bucket item count to get to: 0 curr count: 147
2021/12/17 02:51:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 02:51:17 Context: Bucket flush, response code: 200 dump: 
2021/12/17 02:51:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEnoentAdvancedGet (154.97s)
=== RUN   TestEnoentAdvancedReplace
2021/12/17 02:51:22 Deploying app: TestEnoentAdvancedReplace. Item count src bucket: 0 dst bucket: 0
2021/12/17 02:51:23 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedReplace' in metakv",
  "warnings": null
 }
}
2021/12/17 02:51:28 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/17 02:51:33 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/17 02:51:38 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/17 02:51:43 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/17 02:51:43 App: TestEnoentAdvancedReplace got deployed
2021/12/17 02:51:43 Starting bucket ops to source bucket
2021/12/17 02:51:48 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/17 02:51:53 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/17 02:51:53 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/17 02:51:53 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 02:51:53 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/17 02:51:53 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 02:51:58 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/17 02:52:03 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/17 02:52:08 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/17 02:52:13 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/17 02:52:13 App: TestEnoentAdvancedReplace got un-deployed
2021/12/17 02:52:13 No eventing-con process running
2021/12/17 02:52:13 Function: TestEnoentAdvancedReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedReplace deleting in the background"
}
2021/12/17 02:52:15 Context: Bucket flush, response code: 200 dump: 
2021/12/17 02:52:25 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/17 02:52:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 02:52:27 Context: Bucket flush, response code: 200 dump: 
2021/12/17 02:52:37 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/17 02:52:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 02:52:37 Testing get enoent operation on source bucket
2021/12/17 02:52:37 Deploying app: TestEnoentAdvancedReplace. Item count src bucket: 0 dst bucket: 0
2021/12/17 02:52:38 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedReplace' in metakv",
  "warnings": null
 }
}
2021/12/17 02:52:43 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/17 02:52:48 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/17 02:52:53 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/17 02:52:58 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/17 02:52:58 App: TestEnoentAdvancedReplace got deployed
2021/12/17 02:52:58 Starting bucket ops to source bucket
2021/12/17 02:53:03 Waiting for dst bucket item count to get to: 200 curr count: 0
2021/12/17 02:53:08 Waiting for dst bucket item count to get to: 200 curr count: 100
2021/12/17 02:53:08 src & dst bucket item count matched up. src bucket count: 200 dst bucket count: 200
2021/12/17 02:53:08 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 02:53:08 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/17 02:53:08 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 02:53:13 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/17 02:53:18 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/17 02:53:24 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/17 02:53:29 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/17 02:53:29 App: TestEnoentAdvancedReplace got un-deployed
2021/12/17 02:53:29 No eventing-con process running
2021/12/17 02:53:29 Function: TestEnoentAdvancedReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedReplace deleting in the background"
}
2021/12/17 02:53:30 Context: Bucket flush, response code: 200 dump: 
2021/12/17 02:53:40 Waiting for dst bucket item count to get to: 0 curr count: 145
2021/12/17 02:53:45 Waiting for dst bucket item count to get to: 0 curr count: 145
2021/12/17 02:53:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 02:53:47 Context: Bucket flush, response code: 200 dump: 
2021/12/17 02:53:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEnoentAdvancedReplace (150.03s)
=== RUN   TestEnoentAdvancedDelete
2021/12/17 02:53:52 Deploying app: TestEnoentAdvancedDelete. Item count src bucket: 0 dst bucket: 0
2021/12/17 02:53:53 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedDelete' in metakv",
  "warnings": null
 }
}
2021/12/17 02:53:58 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/17 02:54:03 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/17 02:54:08 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/17 02:54:13 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/17 02:54:13 App: TestEnoentAdvancedDelete got deployed
2021/12/17 02:54:13 Starting bucket ops to source bucket
2021/12/17 02:54:18 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/17 02:54:23 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/17 02:54:23 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/17 02:54:23 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 02:54:23 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/17 02:54:23 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 02:54:28 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/17 02:54:33 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/17 02:54:38 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/17 02:54:43 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/17 02:54:43 App: TestEnoentAdvancedDelete got un-deployed
2021/12/17 02:54:43 No eventing-con process running
2021/12/17 02:54:43 Function: TestEnoentAdvancedDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedDelete deleting in the background"
}
2021/12/17 02:54:45 Context: Bucket flush, response code: 200 dump: 
2021/12/17 02:54:55 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/17 02:54:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 02:54:57 Context: Bucket flush, response code: 200 dump: 
2021/12/17 02:55:07 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/17 02:55:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 02:55:07 Testing get enoent operation on source bucket
2021/12/17 02:55:07 Deploying app: TestEnoentAdvancedDelete. Item count src bucket: 0 dst bucket: 0
2021/12/17 02:55:08 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedDelete' in metakv",
  "warnings": null
 }
}
2021/12/17 02:55:13 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/17 02:55:18 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/17 02:55:23 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/17 02:55:28 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/17 02:55:28 App: TestEnoentAdvancedDelete got deployed
2021/12/17 02:55:28 Starting bucket ops to source bucket
2021/12/17 02:55:33 Waiting for dst bucket item count to get to: 200 curr count: 0
2021/12/17 02:55:38 Waiting for dst bucket item count to get to: 200 curr count: 100
2021/12/17 02:55:38 src & dst bucket item count matched up. src bucket count: 200 dst bucket count: 200
2021/12/17 02:55:38 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 02:55:38 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/17 02:55:39 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 02:55:44 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/17 02:55:49 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/17 02:55:54 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/17 02:55:59 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/17 02:55:59 App: TestEnoentAdvancedDelete got un-deployed
2021/12/17 02:55:59 No eventing-con process running
2021/12/17 02:55:59 Function: TestEnoentAdvancedDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedDelete deleting in the background"
}
2021/12/17 02:56:00 Context: Bucket flush, response code: 200 dump: 
2021/12/17 02:56:10 Waiting for dst bucket item count to get to: 0 curr count: 153
2021/12/17 02:56:15 Waiting for dst bucket item count to get to: 0 curr count: 153
2021/12/17 02:56:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 02:56:17 Context: Bucket flush, response code: 200 dump: 
2021/12/17 02:56:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEnoentAdvancedDelete (150.07s)
=== RUN   TestAdvancedInsertKeyExist
2021/12/17 02:56:22 Deploying app: TestAdvancedInsertKeyExist. Item count src bucket: 0 dst bucket: 0
2021/12/17 02:56:23 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedInsertKeyExist' in metakv",
  "warnings": null
 }
}
2021/12/17 02:56:28 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/17 02:56:33 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/17 02:56:38 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/17 02:56:43 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/17 02:56:43 App: TestAdvancedInsertKeyExist got deployed
2021/12/17 02:56:43 Starting bucket ops to source bucket
2021/12/17 02:56:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 02:56:44 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 02:56:44 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/17 02:56:44 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 02:56:49 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/17 02:56:54 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/17 02:56:59 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/17 02:57:04 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/17 02:57:04 App: TestAdvancedInsertKeyExist got un-deployed
2021/12/17 02:57:04 No eventing-con process running
2021/12/17 02:57:04 Function: TestAdvancedInsertKeyExist delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedInsertKeyExist deleting in the background"
}
2021/12/17 02:57:05 Context: Bucket flush, response code: 200 dump: 
2021/12/17 02:57:10 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 02:57:13 Context: Bucket flush, response code: 200 dump: 
2021/12/17 02:57:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 02:57:18 Pumping document in the destination bucket
2021/12/17 02:57:18 Starting bucket ops to source bucket
2021/12/17 02:57:18 Deploying app: TestAdvancedInsertKeyExist. Item count src bucket: 0 dst bucket: 0
2021/12/17 02:57:19 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedInsertKeyExist' in metakv",
  "warnings": null
 }
}
2021/12/17 02:57:24 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/17 02:57:29 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/17 02:57:34 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/17 02:57:39 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/17 02:57:39 App: TestAdvancedInsertKeyExist got deployed
2021/12/17 02:57:39 Starting bucket ops to source bucket
2021/12/17 02:57:44 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/17 02:57:49 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/17 02:57:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 02:57:49 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 02:57:49 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/17 02:57:49 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 02:57:54 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/17 02:57:59 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/17 02:58:04 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/17 02:58:09 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/17 02:58:09 App: TestAdvancedInsertKeyExist got un-deployed
2021/12/17 02:58:09 No eventing-con process running
2021/12/17 02:58:09 Function: TestAdvancedInsertKeyExist delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedInsertKeyExist deleting in the background"
}
2021/12/17 02:58:11 Context: Bucket flush, response code: 200 dump: 
2021/12/17 02:58:21 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/17 02:58:26 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/17 02:58:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 02:58:28 Context: Bucket flush, response code: 200 dump: 
2021/12/17 02:58:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedInsertKeyExist (130.74s)
=== RUN   TestExipryGet
2021/12/17 02:58:33 Deploying app: TestExipryGet. Item count src bucket: 0 dst bucket: 0
2021/12/17 02:58:34 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryGet' in metakv",
  "warnings": null
 }
}
2021/12/17 02:58:39 Waiting for app: TestExipryGet to get deployed
2021/12/17 02:58:44 Waiting for app: TestExipryGet to get deployed
2021/12/17 02:58:49 Waiting for app: TestExipryGet to get deployed
2021/12/17 02:58:54 Waiting for app: TestExipryGet to get deployed
2021/12/17 02:58:54 App: TestExipryGet got deployed
2021/12/17 02:58:54 Starting bucket ops to source bucket
2021/12/17 02:59:40 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/17 02:59:45 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/17 02:59:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 02:59:45 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 02:59:45 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/17 02:59:45 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 02:59:50 Waiting for app: TestExipryGet to get un-deployed
2021/12/17 02:59:55 Waiting for app: TestExipryGet to get un-deployed
2021/12/17 03:00:00 Waiting for app: TestExipryGet to get un-deployed
2021/12/17 03:00:05 Waiting for app: TestExipryGet to get un-deployed
2021/12/17 03:00:05 App: TestExipryGet got un-deployed
2021/12/17 03:00:05 No eventing-con process running
2021/12/17 03:00:05 Function: TestExipryGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryGet deleting in the background"
}
2021/12/17 03:00:06 Context: Bucket flush, response code: 200 dump: 
2021/12/17 03:00:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 03:00:14 Context: Bucket flush, response code: 200 dump: 
2021/12/17 03:00:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 03:00:19 Pumping document in the destination bucket for expiry get
2021/12/17 03:00:19 Starting bucket ops to source bucket
2021/12/17 03:00:19 Deploying app: TestExipryGet. Item count src bucket: 0 dst bucket: 0
2021/12/17 03:00:20 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryGet' in metakv",
  "warnings": null
 }
}
2021/12/17 03:00:25 Waiting for app: TestExipryGet to get deployed
2021/12/17 03:00:30 Waiting for app: TestExipryGet to get deployed
2021/12/17 03:00:35 Waiting for app: TestExipryGet to get deployed
2021/12/17 03:00:40 Waiting for app: TestExipryGet to get deployed
2021/12/17 03:00:40 App: TestExipryGet got deployed
2021/12/17 03:00:40 Starting bucket ops to source bucket
2021/12/17 03:01:25 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/17 03:01:30 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/17 03:01:30 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 03:01:30 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 03:01:30 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/17 03:01:30 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 03:01:35 Waiting for app: TestExipryGet to get un-deployed
2021/12/17 03:01:40 Waiting for app: TestExipryGet to get un-deployed
2021/12/17 03:01:45 Waiting for app: TestExipryGet to get un-deployed
2021/12/17 03:01:50 Waiting for app: TestExipryGet to get un-deployed
2021/12/17 03:01:50 App: TestExipryGet got un-deployed
2021/12/17 03:01:50 No eventing-con process running
2021/12/17 03:01:50 Function: TestExipryGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryGet deleting in the background"
}
2021/12/17 03:01:52 Context: Bucket flush, response code: 200 dump: 
2021/12/17 03:02:02 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/17 03:02:07 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/17 03:02:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 03:02:09 Context: Bucket flush, response code: 200 dump: 
2021/12/17 03:02:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestExipryGet (221.00s)
=== RUN   TestExipryInsert
2021/12/17 03:02:14 Deploying app: TestExipryInsert. Item count src bucket: 0 dst bucket: 0
2021/12/17 03:02:15 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryInsert' in metakv",
  "warnings": null
 }
}
2021/12/17 03:02:20 Waiting for app: TestExipryInsert to get deployed
2021/12/17 03:02:25 Waiting for app: TestExipryInsert to get deployed
2021/12/17 03:02:30 Waiting for app: TestExipryInsert to get deployed
2021/12/17 03:02:35 Waiting for app: TestExipryInsert to get deployed
2021/12/17 03:02:35 App: TestExipryInsert got deployed
2021/12/17 03:02:35 Starting bucket ops to source bucket
2021/12/17 03:03:20 Waiting for dst bucket item count to get to: 0 curr count: 200
2021/12/17 03:03:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 03:03:20 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 03:03:20 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/17 03:03:20 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 03:03:25 Waiting for app: TestExipryInsert to get un-deployed
2021/12/17 03:03:30 Waiting for app: TestExipryInsert to get un-deployed
2021/12/17 03:03:35 Waiting for app: TestExipryInsert to get un-deployed
2021/12/17 03:03:40 Waiting for app: TestExipryInsert to get un-deployed
2021/12/17 03:03:40 App: TestExipryInsert got un-deployed
2021/12/17 03:03:40 No eventing-con process running
2021/12/17 03:03:41 Function: TestExipryInsert delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryInsert deleting in the background"
}
2021/12/17 03:03:42 Context: Bucket flush, response code: 200 dump: 
2021/12/17 03:03:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 03:03:50 Context: Bucket flush, response code: 200 dump: 
2021/12/17 03:03:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 03:03:55 Testing on the destination bucket for expiry insert
2021/12/17 03:03:55 Deploying app: TestExipryInsert. Item count src bucket: 0 dst bucket: 0
2021/12/17 03:03:56 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryInsert' in metakv",
  "warnings": null
 }
}
2021/12/17 03:04:01 Waiting for app: TestExipryInsert to get deployed
2021/12/17 03:04:06 Waiting for app: TestExipryInsert to get deployed
2021/12/17 03:04:11 Waiting for app: TestExipryInsert to get deployed
2021/12/17 03:04:16 Waiting for app: TestExipryInsert to get deployed
2021/12/17 03:04:16 App: TestExipryInsert got deployed
2021/12/17 03:04:16 Starting bucket ops to source bucket
2021/12/17 03:05:01 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/17 03:05:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 03:05:01 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 03:05:01 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/17 03:05:01 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 03:05:06 Waiting for app: TestExipryInsert to get un-deployed
2021/12/17 03:05:11 Waiting for app: TestExipryInsert to get un-deployed
2021/12/17 03:05:16 Waiting for app: TestExipryInsert to get un-deployed
2021/12/17 03:05:21 Waiting for app: TestExipryInsert to get un-deployed
2021/12/17 03:05:21 App: TestExipryInsert got un-deployed
2021/12/17 03:05:21 No eventing-con process running
2021/12/17 03:05:21 Function: TestExipryInsert delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryInsert deleting in the background"
}
2021/12/17 03:05:23 Context: Bucket flush, response code: 200 dump: 
2021/12/17 03:05:33 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/17 03:05:38 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/17 03:05:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 03:05:40 Context: Bucket flush, response code: 200 dump: 
2021/12/17 03:05:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestExipryInsert (210.64s)
=== RUN   TestExipryReplace
2021/12/17 03:05:45 Deploying app: TestExipryReplace. Item count src bucket: 0 dst bucket: 0
2021/12/17 03:05:46 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryReplace' in metakv",
  "warnings": null
 }
}
2021/12/17 03:05:51 Waiting for app: TestExipryReplace to get deployed
2021/12/17 03:05:56 Waiting for app: TestExipryReplace to get deployed
2021/12/17 03:06:01 Waiting for app: TestExipryReplace to get deployed
2021/12/17 03:06:06 Waiting for app: TestExipryReplace to get deployed
2021/12/17 03:06:06 App: TestExipryReplace got deployed
2021/12/17 03:06:06 Starting bucket ops to source bucket
2021/12/17 03:06:51 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/17 03:06:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 03:06:51 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 03:06:51 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/17 03:06:51 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 03:06:56 Waiting for app: TestExipryReplace to get un-deployed
2021/12/17 03:07:01 Waiting for app: TestExipryReplace to get un-deployed
2021/12/17 03:07:06 Waiting for app: TestExipryReplace to get un-deployed
2021/12/17 03:07:11 Waiting for app: TestExipryReplace to get un-deployed
2021/12/17 03:07:11 App: TestExipryReplace got un-deployed
2021/12/17 03:07:11 No eventing-con process running
2021/12/17 03:07:11 Function: TestExipryReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryReplace deleting in the background"
}
2021/12/17 03:07:13 Context: Bucket flush, response code: 200 dump: 
2021/12/17 03:07:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 03:07:20 Context: Bucket flush, response code: 200 dump: 
2021/12/17 03:07:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 03:07:25 Testing on the destination bucket for expiry insert
2021/12/17 03:07:25 Deploying app: TestExipryReplace. Item count src bucket: 0 dst bucket: 0
2021/12/17 03:07:26 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryReplace' in metakv",
  "warnings": null
 }
}
2021/12/17 03:07:31 Waiting for app: TestExipryReplace to get deployed
2021/12/17 03:07:36 Waiting for app: TestExipryReplace to get deployed
2021/12/17 03:07:41 Waiting for app: TestExipryReplace to get deployed
2021/12/17 03:07:46 Waiting for app: TestExipryReplace to get deployed
2021/12/17 03:07:46 App: TestExipryReplace got deployed
2021/12/17 03:07:46 Starting bucket ops to source bucket
2021/12/17 03:08:32 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/17 03:08:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 03:08:32 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 03:08:32 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/17 03:08:32 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 03:08:37 Waiting for app: TestExipryReplace to get un-deployed
2021/12/17 03:08:42 Waiting for app: TestExipryReplace to get un-deployed
2021/12/17 03:08:47 Waiting for app: TestExipryReplace to get un-deployed
2021/12/17 03:08:52 Waiting for app: TestExipryReplace to get un-deployed
2021/12/17 03:08:52 App: TestExipryReplace got un-deployed
2021/12/17 03:08:52 No eventing-con process running
2021/12/17 03:08:52 Function: TestExipryReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryReplace deleting in the background"
}
2021/12/17 03:08:53 Context: Bucket flush, response code: 200 dump: 
2021/12/17 03:09:04 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/17 03:09:09 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/17 03:09:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 03:09:10 Context: Bucket flush, response code: 200 dump: 
2021/12/17 03:09:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestExipryReplace (210.76s)
=== RUN   TestExipryUpsert
2021/12/17 03:09:15 Deploying app: TestExipryUpsert. Item count src bucket: 0 dst bucket: 0
2021/12/17 03:09:16 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryUpsert' in metakv",
  "warnings": null
 }
}
2021/12/17 03:09:21 Waiting for app: TestExipryUpsert to get deployed
2021/12/17 03:09:27 Waiting for app: TestExipryUpsert to get deployed
2021/12/17 03:09:32 Waiting for app: TestExipryUpsert to get deployed
2021/12/17 03:09:37 Waiting for app: TestExipryUpsert to get deployed
2021/12/17 03:09:37 App: TestExipryUpsert got deployed
2021/12/17 03:09:37 Starting bucket ops to source bucket
2021/12/17 03:10:22 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/17 03:10:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 03:10:22 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 03:10:22 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/17 03:10:22 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 03:10:27 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/17 03:10:32 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/17 03:10:37 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/17 03:10:42 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/17 03:10:42 App: TestExipryUpsert got un-deployed
2021/12/17 03:10:42 No eventing-con process running
2021/12/17 03:10:42 Function: TestExipryUpsert delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryUpsert deleting in the background"
}
2021/12/17 03:10:44 Context: Bucket flush, response code: 200 dump: 
2021/12/17 03:10:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 03:10:51 Context: Bucket flush, response code: 200 dump: 
2021/12/17 03:10:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 03:10:56 Testing on the destination bucket for expiry insert
2021/12/17 03:10:56 Deploying app: TestExipryUpsert. Item count src bucket: 0 dst bucket: 0
2021/12/17 03:10:57 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryUpsert' in metakv",
  "warnings": null
 }
}
2021/12/17 03:11:02 Waiting for app: TestExipryUpsert to get deployed
2021/12/17 03:11:07 Waiting for app: TestExipryUpsert to get deployed
2021/12/17 03:11:12 Waiting for app: TestExipryUpsert to get deployed
2021/12/17 03:11:17 Waiting for app: TestExipryUpsert to get deployed
2021/12/17 03:11:17 App: TestExipryUpsert got deployed
2021/12/17 03:11:17 Starting bucket ops to source bucket
2021/12/17 03:12:02 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/17 03:12:07 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/17 03:12:12 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/17 03:12:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 03:12:12 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 03:12:12 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/17 03:12:12 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 03:12:17 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/17 03:12:22 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/17 03:12:28 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/17 03:12:33 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/17 03:12:33 App: TestExipryUpsert got un-deployed
2021/12/17 03:12:33 No eventing-con process running
2021/12/17 03:12:33 Function: TestExipryUpsert delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryUpsert deleting in the background"
}
2021/12/17 03:12:34 Context: Bucket flush, response code: 200 dump: 
2021/12/17 03:12:44 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/17 03:12:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 03:12:46 Context: Bucket flush, response code: 200 dump: 
2021/12/17 03:12:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestExipryUpsert (215.79s)
=== RUN   TestCasReplace
2021/12/17 03:12:51 Deploying app: TestCasReplace. Item count src bucket: 0 dst bucket: 0
2021/12/17 03:12:52 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCasReplace' in metakv",
  "warnings": null
 }
}
2021/12/17 03:12:57 Waiting for app: TestCasReplace to get deployed
2021/12/17 03:13:02 Waiting for app: TestCasReplace to get deployed
2021/12/17 03:13:07 Waiting for app: TestCasReplace to get deployed
2021/12/17 03:13:12 Waiting for app: TestCasReplace to get deployed
2021/12/17 03:13:12 App: TestCasReplace got deployed
2021/12/17 03:13:12 Starting bucket ops to source bucket
2021/12/17 03:13:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 03:13:12 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 03:13:12 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/17 03:13:13 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 03:13:18 Waiting for app: TestCasReplace to get un-deployed
2021/12/17 03:13:23 Waiting for app: TestCasReplace to get un-deployed
2021/12/17 03:13:28 Waiting for app: TestCasReplace to get un-deployed
2021/12/17 03:13:33 Waiting for app: TestCasReplace to get un-deployed
2021/12/17 03:13:33 App: TestCasReplace got un-deployed
2021/12/17 03:13:33 No eventing-con process running
2021/12/17 03:13:33 Function: TestCasReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCasReplace deleting in the background"
}
2021/12/17 03:13:34 Context: Bucket flush, response code: 200 dump: 
2021/12/17 03:13:44 Waiting for dst bucket item count to get to: 0 curr count: 10
2021/12/17 03:13:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 03:13:46 Context: Bucket flush, response code: 200 dump: 
2021/12/17 03:13:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 03:13:51 Testing on the destination bucket for upsert with cas
2021/12/17 03:13:51 Starting bucket ops to source bucket
2021/12/17 03:13:51 Deploying app: TestCasReplace. Item count src bucket: 0 dst bucket: 0
2021/12/17 03:13:52 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCasReplace' in metakv",
  "warnings": null
 }
}
2021/12/17 03:13:57 Waiting for app: TestCasReplace to get deployed
2021/12/17 03:14:02 Waiting for app: TestCasReplace to get deployed
2021/12/17 03:14:07 Waiting for app: TestCasReplace to get deployed
2021/12/17 03:14:12 Waiting for app: TestCasReplace to get deployed
2021/12/17 03:14:12 App: TestCasReplace got deployed
2021/12/17 03:14:12 Starting bucket ops to source bucket
2021/12/17 03:14:18 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/17 03:14:23 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/17 03:14:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 03:14:23 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 03:14:23 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/17 03:14:23 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 03:14:28 Waiting for app: TestCasReplace to get un-deployed
2021/12/17 03:14:33 Waiting for app: TestCasReplace to get un-deployed
2021/12/17 03:14:38 Waiting for app: TestCasReplace to get un-deployed
2021/12/17 03:14:43 Waiting for app: TestCasReplace to get un-deployed
2021/12/17 03:14:43 App: TestCasReplace got un-deployed
2021/12/17 03:14:43 No eventing-con process running
2021/12/17 03:14:43 Function: TestCasReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCasReplace deleting in the background"
}
2021/12/17 03:14:44 Context: Bucket flush, response code: 200 dump: 
2021/12/17 03:14:54 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/17 03:14:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 03:14:56 Context: Bucket flush, response code: 200 dump: 
2021/12/17 03:15:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCasReplace (130.15s)
=== RUN   TestCasDelete
2021/12/17 03:15:01 Deploying app: TestCasDelete. Item count src bucket: 0 dst bucket: 0
2021/12/17 03:15:03 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCasDelete' in metakv",
  "warnings": null
 }
}
2021/12/17 03:15:08 Waiting for app: TestCasDelete to get deployed
2021/12/17 03:15:13 Waiting for app: TestCasDelete to get deployed
2021/12/17 03:15:18 Waiting for app: TestCasDelete to get deployed
2021/12/17 03:15:23 Waiting for app: TestCasDelete to get deployed
2021/12/17 03:15:23 App: TestCasDelete got deployed
2021/12/17 03:15:23 Starting bucket ops to source bucket
2021/12/17 03:15:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 03:15:23 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 03:15:23 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/17 03:15:23 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 03:15:28 Waiting for app: TestCasDelete to get un-deployed
2021/12/17 03:15:33 Waiting for app: TestCasDelete to get un-deployed
2021/12/17 03:15:38 Waiting for app: TestCasDelete to get un-deployed
2021/12/17 03:15:43 Waiting for app: TestCasDelete to get un-deployed
2021/12/17 03:15:43 App: TestCasDelete got un-deployed
2021/12/17 03:15:43 No eventing-con process running
2021/12/17 03:15:43 Function: TestCasDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCasDelete deleting in the background"
}
2021/12/17 03:15:45 Context: Bucket flush, response code: 200 dump: 
2021/12/17 03:15:55 Waiting for dst bucket item count to get to: 0 curr count: 1
2021/12/17 03:15:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 03:15:56 Context: Bucket flush, response code: 200 dump: 
2021/12/17 03:16:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 03:16:01 Testing on the destination bucket for delete with cas
2021/12/17 03:16:01 Starting bucket ops to source bucket
2021/12/17 03:16:02 Deploying app: TestCasDelete. Item count src bucket: 0 dst bucket: 0
2021/12/17 03:16:03 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCasDelete' in metakv",
  "warnings": null
 }
}
2021/12/17 03:16:08 Waiting for app: TestCasDelete to get deployed
2021/12/17 03:16:13 Waiting for app: TestCasDelete to get deployed
2021/12/17 03:16:18 Waiting for app: TestCasDelete to get deployed
2021/12/17 03:16:23 Waiting for app: TestCasDelete to get deployed
2021/12/17 03:16:23 App: TestCasDelete got deployed
2021/12/17 03:16:23 Starting bucket ops to source bucket
2021/12/17 03:16:28 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/17 03:16:33 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/17 03:16:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 03:16:33 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 03:16:33 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/17 03:16:33 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 03:16:38 Waiting for app: TestCasDelete to get un-deployed
2021/12/17 03:16:43 Waiting for app: TestCasDelete to get un-deployed
2021/12/17 03:16:48 Waiting for app: TestCasDelete to get un-deployed
2021/12/17 03:16:53 Waiting for app: TestCasDelete to get un-deployed
2021/12/17 03:16:53 App: TestCasDelete got un-deployed
2021/12/17 03:16:53 No eventing-con process running
2021/12/17 03:16:53 Function: TestCasDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCasDelete deleting in the background"
}
2021/12/17 03:16:55 Context: Bucket flush, response code: 200 dump: 
2021/12/17 03:17:05 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/17 03:17:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 03:17:06 Context: Bucket flush, response code: 200 dump: 
2021/12/17 03:17:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCasDelete (130.23s)
=== RUN   TestCountersIncrement
2021/12/17 03:17:11 Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/17 03:17:17 Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/17 03:17:18 Removing node(s): [] from the cluster
2021/12/17 03:17:23 Rebalance progress: 100
2021/12/17 03:17:23 Deploying app: TestCountersIncrement. Item count src bucket: 0 dst bucket: 0
2021/12/17 03:17:24 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCountersIncrement' in metakv",
  "warnings": null
 }
}
2021/12/17 03:17:29 Waiting for app: TestCountersIncrement to get deployed
2021/12/17 03:17:34 Waiting for app: TestCountersIncrement to get deployed
2021/12/17 03:17:39 Waiting for app: TestCountersIncrement to get deployed
2021/12/17 03:17:44 Waiting for app: TestCountersIncrement to get deployed
2021/12/17 03:17:44 App: TestCountersIncrement got deployed
2021/12/17 03:17:44 Starting bucket ops to source bucket
2021/12/17 03:17:50 Waiting for dst bucket item count to get to: 1024 curr count: 0
2021/12/17 03:17:55 Waiting for dst bucket item count to get to: 1024 curr count: 0
2021/12/17 03:17:55 src & dst bucket item count matched up. src bucket count: 1024 dst bucket count: 1024
2021/12/17 03:17:55 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 03:17:55 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 03:18:00 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/17 03:18:05 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/17 03:18:10 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/17 03:18:15 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/17 03:18:15 App: TestCountersIncrement got un-deployed
2021/12/17 03:18:15 No eventing-con process running
2021/12/17 03:18:15 Function: TestCountersIncrement delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCountersIncrement deleting in the background"
}
2021/12/17 03:18:17 Context: Bucket flush, response code: 200 dump: 
2021/12/17 03:18:27 Waiting for dst bucket item count to get to: 0 curr count: 1025
2021/12/17 03:18:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 03:18:29 Context: Bucket flush, response code: 200 dump: 
2021/12/17 03:18:39 Waiting for dst bucket item count to get to: 0 curr count: 1020
2021/12/17 03:18:44 Waiting for dst bucket item count to get to: 0 curr count: 508
2021/12/17 03:18:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 03:18:44 Deploying app: TestCountersIncrement. Item count src bucket: 0 dst bucket: 0
2021/12/17 03:18:45 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCountersIncrement' in metakv",
  "warnings": null
 }
}
2021/12/17 03:18:50 Waiting for app: TestCountersIncrement to get deployed
2021/12/17 03:18:55 Waiting for app: TestCountersIncrement to get deployed
2021/12/17 03:19:00 Waiting for app: TestCountersIncrement to get deployed
2021/12/17 03:19:05 Waiting for app: TestCountersIncrement to get deployed
2021/12/17 03:19:05 App: TestCountersIncrement got deployed
2021/12/17 03:19:05 Starting bucket ops to source bucket
2021/12/17 03:19:11 Waiting for dst bucket item count to get to: 2048 curr count: 0
2021/12/17 03:19:11 src & dst bucket item count matched up. src bucket count: 2048 dst bucket count: 2048
2021/12/17 03:19:11 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 03:19:11 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 03:19:16 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/17 03:19:21 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/17 03:19:26 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/17 03:19:31 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/17 03:19:31 App: TestCountersIncrement got un-deployed
2021/12/17 03:19:31 No eventing-con process running
2021/12/17 03:19:31 Function: TestCountersIncrement delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCountersIncrement deleting in the background"
}
2021/12/17 03:19:33 Context: Bucket flush, response code: 200 dump: 
2021/12/17 03:19:43 Waiting for dst bucket item count to get to: 0 curr count: 2048
2021/12/17 03:19:48 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/17 03:19:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 03:19:50 Context: Bucket flush, response code: 200 dump: 
2021/12/17 03:19:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 03:19:55 Removing node(s): [http://127.0.0.1:19002 http://127.0.0.1:19003] from the cluster
2021/12/17 03:20:00 Rebalance progress: 100
--- PASS: TestCountersIncrement (168.59s)
=== RUN   TestCountersDecrement
2021/12/17 03:20:00 Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/17 03:20:00 Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/17 03:20:01 Removing node(s): [] from the cluster
2021/12/17 03:20:06 Rebalance progress: 100
2021/12/17 03:20:06 Deploying app: TestCountersDecrement. Item count src bucket: 0 dst bucket: 0
2021/12/17 03:20:07 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCountersDecrement' in metakv",
  "warnings": null
 }
}
2021/12/17 03:20:12 Waiting for app: TestCountersDecrement to get deployed
2021/12/17 03:20:17 Waiting for app: TestCountersDecrement to get deployed
2021/12/17 03:20:22 Waiting for app: TestCountersDecrement to get deployed
2021/12/17 03:20:27 Waiting for app: TestCountersDecrement to get deployed
2021/12/17 03:20:27 App: TestCountersDecrement got deployed
2021/12/17 03:20:27 Starting bucket ops to source bucket
2021/12/17 03:20:33 Waiting for dst bucket item count to get to: 1024 curr count: 0
2021/12/17 03:20:38 Waiting for dst bucket item count to get to: 1024 curr count: 512
2021/12/17 03:20:43 Waiting for dst bucket item count to get to: 1024 curr count: 512
2021/12/17 03:20:43 src & dst bucket item count matched up. src bucket count: 1024 dst bucket count: 1024
2021/12/17 03:20:43 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 03:20:43 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 03:20:48 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/17 03:20:53 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/17 03:20:58 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/17 03:21:03 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/17 03:21:03 App: TestCountersDecrement got un-deployed
2021/12/17 03:21:03 No eventing-con process running
2021/12/17 03:21:03 Function: TestCountersDecrement delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCountersDecrement deleting in the background"
}
2021/12/17 03:21:05 Context: Bucket flush, response code: 200 dump: 
2021/12/17 03:21:15 Waiting for dst bucket item count to get to: 0 curr count: 1025
2021/12/17 03:21:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 03:21:17 Context: Bucket flush, response code: 200 dump: 
2021/12/17 03:21:27 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/17 03:21:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 03:21:27 Deploying app: TestCountersDecrement. Item count src bucket: 0 dst bucket: 0
2021/12/17 03:21:28 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCountersDecrement' in metakv",
  "warnings": null
 }
}
2021/12/17 03:21:33 Waiting for app: TestCountersDecrement to get deployed
2021/12/17 03:21:38 Waiting for app: TestCountersDecrement to get deployed
2021/12/17 03:21:43 Waiting for app: TestCountersDecrement to get deployed
2021/12/17 03:21:48 Waiting for app: TestCountersDecrement to get deployed
2021/12/17 03:21:48 App: TestCountersDecrement got deployed
2021/12/17 03:21:48 Starting bucket ops to source bucket
2021/12/17 03:21:54 Waiting for dst bucket item count to get to: 2048 curr count: 0
2021/12/17 03:21:59 Waiting for dst bucket item count to get to: 2048 curr count: 894
2021/12/17 03:22:04 Waiting for dst bucket item count to get to: 2048 curr count: 1918
2021/12/17 03:22:04 src & dst bucket item count matched up. src bucket count: 2048 dst bucket count: 2048
2021/12/17 03:22:04 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 03:22:05 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 03:22:10 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/17 03:22:15 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/17 03:22:20 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/17 03:22:25 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/17 03:22:25 App: TestCountersDecrement got un-deployed
2021/12/17 03:22:25 No eventing-con process running
2021/12/17 03:22:25 Function: TestCountersDecrement delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCountersDecrement deleting in the background"
}
2021/12/17 03:22:27 Context: Bucket flush, response code: 200 dump: 
2021/12/17 03:22:37 Waiting for dst bucket item count to get to: 0 curr count: 2048
2021/12/17 03:22:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 03:22:38 Context: Bucket flush, response code: 200 dump: 
2021/12/17 03:22:48 Waiting for dst bucket item count to get to: 0 curr count: 1
2021/12/17 03:22:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 03:22:48 Removing node(s): [http://127.0.0.1:19002 http://127.0.0.1:19003] from the cluster
2021/12/17 03:22:53 Rebalance progress: 100
--- PASS: TestCountersDecrement (173.37s)
=== RUN   TestCacheGet
2021/12/17 03:22:53 Deploying app: TestCacheGet. Item count src bucket: 0 dst bucket: 0
2021/12/17 03:22:55 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCacheGet' in metakv",
  "warnings": null
 }
}
2021/12/17 03:23:00 Waiting for app: TestCacheGet to get deployed
2021/12/17 03:23:05 Waiting for app: TestCacheGet to get deployed
2021/12/17 03:23:10 Waiting for app: TestCacheGet to get deployed
2021/12/17 03:23:15 Waiting for app: TestCacheGet to get deployed
2021/12/17 03:23:15 App: TestCacheGet got deployed
2021/12/17 03:23:15 Starting bucket ops to source bucket
2021/12/17 03:23:20 Waiting for dst bucket item count to get to: 20 curr count: 0
2021/12/17 03:23:25 Waiting for dst bucket item count to get to: 20 curr count: 0
2021/12/17 03:23:30 Waiting for dst bucket item count to get to: 20 curr count: 0
2021/12/17 03:23:35 Waiting for dst bucket item count to get to: 20 curr count: 4
2021/12/17 03:23:40 Waiting for dst bucket item count to get to: 20 curr count: 12
2021/12/17 03:23:45 Waiting for dst bucket item count to get to: 20 curr count: 18
2021/12/17 03:23:45 src & dst bucket item count matched up. src bucket count: 20 dst bucket count: 20
2021/12/17 03:23:45 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 03:23:45 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 03:23:50 Waiting for app: TestCacheGet to get un-deployed
2021/12/17 03:23:55 Waiting for app: TestCacheGet to get un-deployed
2021/12/17 03:24:00 Waiting for app: TestCacheGet to get un-deployed
2021/12/17 03:24:05 Waiting for app: TestCacheGet to get un-deployed
2021/12/17 03:24:05 App: TestCacheGet got un-deployed
2021/12/17 03:24:05 No eventing-con process running
2021/12/17 03:24:05 Function: TestCacheGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCacheGet deleting in the background"
}
2021/12/17 03:24:07 Context: Bucket flush, response code: 200 dump: 
2021/12/17 03:24:17 Waiting for dst bucket item count to get to: 0 curr count: 20
2021/12/17 03:24:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 03:24:19 Context: Bucket flush, response code: 200 dump: 
2021/12/17 03:24:29 Waiting for dst bucket item count to get to: 0 curr count: 20
2021/12/17 03:24:34 Waiting for dst bucket item count to get to: 0 curr count: 10
2021/12/17 03:24:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCacheGet (100.37s)
=== RUN   TestCacheRYOW
2021/12/17 03:24:34 Deploying app: TestCacheRYOW. Item count src bucket: 0 dst bucket: 0
2021/12/17 03:24:35 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCacheRYOW' in metakv",
  "warnings": null
 }
}
2021/12/17 03:24:40 Waiting for app: TestCacheRYOW to get deployed
2021/12/17 03:24:45 Waiting for app: TestCacheRYOW to get deployed
2021/12/17 03:24:50 Waiting for app: TestCacheRYOW to get deployed
2021/12/17 03:24:55 Waiting for app: TestCacheRYOW to get deployed
2021/12/17 03:24:55 App: TestCacheRYOW got deployed
2021/12/17 03:24:55 Starting bucket ops to source bucket
2021/12/17 03:25:00 Waiting for dst bucket item count to get to: 50 curr count: 0
2021/12/17 03:25:05 Waiting for dst bucket item count to get to: 50 curr count: 25
2021/12/17 03:25:05 src & dst bucket item count matched up. src bucket count: 50 dst bucket count: 50
2021/12/17 03:25:05 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 03:25:05 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 03:25:10 Waiting for app: TestCacheRYOW to get un-deployed
2021/12/17 03:25:15 Waiting for app: TestCacheRYOW to get un-deployed
2021/12/17 03:25:20 Waiting for app: TestCacheRYOW to get un-deployed
2021/12/17 03:25:25 Waiting for app: TestCacheRYOW to get un-deployed
2021/12/17 03:25:25 App: TestCacheRYOW got un-deployed
2021/12/17 03:25:25 No eventing-con process running
2021/12/17 03:25:25 Function: TestCacheRYOW delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCacheRYOW deleting in the background"
}
2021/12/17 03:25:27 Context: Bucket flush, response code: 200 dump: 
2021/12/17 03:25:37 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/17 03:25:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 03:25:39 Context: Bucket flush, response code: 200 dump: 
2021/12/17 03:25:49 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/17 03:25:54 Waiting for dst bucket item count to get to: 0 curr count: 25
2021/12/17 03:25:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCacheRYOW (80.19s)
=== RUN   TestCacheExpiry
2021/12/17 03:25:54 Deploying app: TestCacheExpiry. Item count src bucket: 0 dst bucket: 0
2021/12/17 03:25:55 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCacheExpiry' in metakv",
  "warnings": null
 }
}
2021/12/17 03:26:00 Waiting for app: TestCacheExpiry to get deployed
2021/12/17 03:26:05 Waiting for app: TestCacheExpiry to get deployed
2021/12/17 03:26:10 Waiting for app: TestCacheExpiry to get deployed
2021/12/17 03:26:15 Waiting for app: TestCacheExpiry to get deployed
2021/12/17 03:26:15 App: TestCacheExpiry got deployed
2021/12/17 03:26:15 Starting bucket ops to source bucket
2021/12/17 03:26:20 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/17 03:26:25 Waiting for dst bucket item count to get to: 100 curr count: 19
2021/12/17 03:26:30 Waiting for dst bucket item count to get to: 100 curr count: 19
2021/12/17 03:26:35 Waiting for dst bucket item count to get to: 100 curr count: 94
2021/12/17 03:26:40 Waiting for dst bucket item count to get to: 100 curr count: 94
2021/12/17 03:26:40 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/17 03:26:40 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 03:26:41 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 03:26:46 Waiting for app: TestCacheExpiry to get un-deployed
2021/12/17 03:26:51 Waiting for app: TestCacheExpiry to get un-deployed
2021/12/17 03:26:56 Waiting for app: TestCacheExpiry to get un-deployed
2021/12/17 03:27:01 Waiting for app: TestCacheExpiry to get un-deployed
2021/12/17 03:27:01 App: TestCacheExpiry got un-deployed
2021/12/17 03:27:01 No eventing-con process running
2021/12/17 03:27:01 Function: TestCacheExpiry delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCacheExpiry deleting in the background"
}
2021/12/17 03:27:02 Context: Bucket flush, response code: 200 dump: 
2021/12/17 03:27:12 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/17 03:27:17 Waiting for dst bucket item count to get to: 0 curr count: 25
2021/12/17 03:27:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 03:27:19 Context: Bucket flush, response code: 200 dump: 
2021/12/17 03:27:29 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/17 03:27:34 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/17 03:27:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCacheExpiry (100.27s)
=== RUN   TestCacheExpiryCustom
2021/12/17 03:27:34 Deploying app: TestCacheExpiryCustom. Item count src bucket: 0 dst bucket: 0
2021/12/17 03:27:35 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCacheExpiryCustom' in metakv",
  "warnings": null
 }
}
2021/12/17 03:27:40 Waiting for app: TestCacheExpiryCustom to get deployed
2021/12/17 03:27:45 Waiting for app: TestCacheExpiryCustom to get deployed
2021/12/17 03:27:50 Waiting for app: TestCacheExpiryCustom to get deployed
2021/12/17 03:27:55 Waiting for app: TestCacheExpiryCustom to get deployed
2021/12/17 03:27:55 App: TestCacheExpiryCustom got deployed
2021/12/17 03:27:55 Starting bucket ops to source bucket
2021/12/17 03:28:01 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/17 03:28:06 Waiting for dst bucket item count to get to: 100 curr count: 19
2021/12/17 03:28:11 Waiting for dst bucket item count to get to: 100 curr count: 19
2021/12/17 03:28:16 Waiting for dst bucket item count to get to: 100 curr count: 94
2021/12/17 03:28:21 Waiting for dst bucket item count to get to: 100 curr count: 94
2021/12/17 03:28:21 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/17 03:28:21 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 03:28:21 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 03:28:26 Waiting for app: TestCacheExpiryCustom to get un-deployed
2021/12/17 03:28:31 Waiting for app: TestCacheExpiryCustom to get un-deployed
2021/12/17 03:28:36 Waiting for app: TestCacheExpiryCustom to get un-deployed
2021/12/17 03:28:41 Waiting for app: TestCacheExpiryCustom to get un-deployed
2021/12/17 03:28:41 App: TestCacheExpiryCustom got un-deployed
2021/12/17 03:28:41 No eventing-con process running
2021/12/17 03:28:41 Function: TestCacheExpiryCustom delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCacheExpiryCustom deleting in the background"
}
2021/12/17 03:28:43 Context: Bucket flush, response code: 200 dump: 
2021/12/17 03:28:53 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/17 03:28:58 Waiting for dst bucket item count to get to: 0 curr count: 25
2021/12/17 03:28:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 03:28:59 Context: Bucket flush, response code: 200 dump: 
2021/12/17 03:29:09 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/17 03:29:14 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/17 03:29:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCacheExpiryCustom (100.15s)
=== RUN   TestCacheOverflowCustom
2021/12/17 03:29:14 Deploying app: TestCacheOverflowCustom. Item count src bucket: 0 dst bucket: 0
2021/12/17 03:29:16 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCacheOverflowCustom' in metakv",
  "warnings": null
 }
}
2021/12/17 03:29:21 Waiting for app: TestCacheOverflowCustom to get deployed
2021/12/17 03:29:26 Waiting for app: TestCacheOverflowCustom to get deployed
2021/12/17 03:29:31 Waiting for app: TestCacheOverflowCustom to get deployed
2021/12/17 03:29:36 Waiting for app: TestCacheOverflowCustom to get deployed
2021/12/17 03:29:36 App: TestCacheOverflowCustom got deployed
2021/12/17 03:29:36 Starting bucket ops to source bucket
2021/12/17 03:29:41 Waiting for dst bucket item count to get to: 35 curr count: 0
2021/12/17 03:29:46 Waiting for dst bucket item count to get to: 35 curr count: 13
2021/12/17 03:29:51 Waiting for dst bucket item count to get to: 35 curr count: 13
2021/12/17 03:29:56 Waiting for dst bucket item count to get to: 35 curr count: 27
2021/12/17 03:30:01 Waiting for dst bucket item count to get to: 35 curr count: 27
2021/12/17 03:30:01 src & dst bucket item count matched up. src bucket count: 35 dst bucket count: 35
2021/12/17 03:30:01 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 03:30:01 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 03:30:06 Waiting for app: TestCacheOverflowCustom to get un-deployed
2021/12/17 03:30:11 Waiting for app: TestCacheOverflowCustom to get un-deployed
2021/12/17 03:30:16 Waiting for app: TestCacheOverflowCustom to get un-deployed
2021/12/17 03:30:21 Waiting for app: TestCacheOverflowCustom to get un-deployed
2021/12/17 03:30:21 App: TestCacheOverflowCustom got un-deployed
2021/12/17 03:30:21 No eventing-con process running
2021/12/17 03:30:21 Function: TestCacheOverflowCustom delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCacheOverflowCustom deleting in the background"
}
2021/12/17 03:30:23 Context: Bucket flush, response code: 200 dump: 
2021/12/17 03:30:33 Waiting for dst bucket item count to get to: 0 curr count: 5
2021/12/17 03:30:38 Waiting for dst bucket item count to get to: 0 curr count: 3
2021/12/17 03:30:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 03:30:40 Context: Bucket flush, response code: 200 dump: 
2021/12/17 03:30:50 Waiting for dst bucket item count to get to: 0 curr count: 17
2021/12/17 03:30:55 Waiting for dst bucket item count to get to: 0 curr count: 17
2021/12/17 03:30:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCacheOverflowCustom (100.17s)
=== RUN   TestStrictMode
2021/12/17 03:30:55 Deploying app: TestStrictMode. Item count src bucket: 0 dst bucket: 0
2021/12/17 03:30:56 Context: Post to main store, response code: 422 dump: {
 "name": "ERR_HANDLER_COMPILATION",
 "code": 27,
 "description": "Function compilation failed",
 "attributes": null,
 "runtime_info": {
  "code": 27,
  "info": {
   "area": "handlerCode",
   "column_number": 63,
   "compile_success": false,
   "description": "        destination[meta.id] = 00000000000000000000000000000101;\n                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nSyntaxError: Octal literals are not allowed in strict mode.\n",
   "index": 132,
   "language": "JavaScript",
   "line_number": 4
  }
 }
}
--- PASS: TestStrictMode (1.06s)
=== RUN   TestDataTypes
2021/12/17 03:30:56 Deploying app: TestDataTypes. Item count src bucket: 0 dst bucket: 0
2021/12/17 03:30:57 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDataTypes' in metakv",
  "warnings": null
 }
}
2021/12/17 03:31:02 Waiting for app: TestDataTypes to get deployed
2021/12/17 03:31:07 Waiting for app: TestDataTypes to get deployed
2021/12/17 03:31:12 Waiting for app: TestDataTypes to get deployed
2021/12/17 03:31:17 Waiting for app: TestDataTypes to get deployed
2021/12/17 03:31:17 App: TestDataTypes got deployed
2021/12/17 03:31:17 Starting bucket ops to source bucket
2021/12/17 03:31:22 Waiting for dst bucket item count to get to: 18 curr count: 0
2021/12/17 03:31:22 src & dst bucket item count matched up. src bucket count: 18 dst bucket count: 18
2021/12/17 03:31:22 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 03:31:22 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 03:31:27 Waiting for app: TestDataTypes to get un-deployed
2021/12/17 03:31:32 Waiting for app: TestDataTypes to get un-deployed
2021/12/17 03:31:37 Waiting for app: TestDataTypes to get un-deployed
2021/12/17 03:31:42 Waiting for app: TestDataTypes to get un-deployed
2021/12/17 03:31:42 App: TestDataTypes got un-deployed
2021/12/17 03:31:42 No eventing-con process running
2021/12/17 03:31:42 Function: TestDataTypes delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDataTypes deleting in the background"
}
2021/12/17 03:31:44 Context: Bucket flush, response code: 200 dump: 
2021/12/17 03:31:54 Waiting for dst bucket item count to get to: 0 curr count: 1
2021/12/17 03:31:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 03:31:56 Context: Bucket flush, response code: 200 dump: 
2021/12/17 03:32:06 Waiting for dst bucket item count to get to: 0 curr count: 18
2021/12/17 03:32:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDataTypes (70.09s)
=== RUN   TestEnoentGet
2021/12/17 03:32:06 Deploying app: TestEnoentGet. Item count src bucket: 0 dst bucket: 0
2021/12/17 03:32:07 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentGet' in metakv",
  "warnings": null
 }
}
2021/12/17 03:32:12 Waiting for app: TestEnoentGet to get deployed
2021/12/17 03:32:17 Waiting for app: TestEnoentGet to get deployed
2021/12/17 03:32:22 Waiting for app: TestEnoentGet to get deployed
2021/12/17 03:32:27 Waiting for app: TestEnoentGet to get deployed
2021/12/17 03:32:27 App: TestEnoentGet got deployed
2021/12/17 03:32:27 Starting bucket ops to source bucket
2021/12/17 03:32:32 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/17 03:32:32 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/17 03:32:32 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 03:32:32 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 03:32:37 Waiting for app: TestEnoentGet to get un-deployed
2021/12/17 03:32:42 Waiting for app: TestEnoentGet to get un-deployed
2021/12/17 03:32:47 Waiting for app: TestEnoentGet to get un-deployed
2021/12/17 03:32:52 Waiting for app: TestEnoentGet to get un-deployed
2021/12/17 03:32:52 App: TestEnoentGet got un-deployed
2021/12/17 03:32:52 No eventing-con process running
2021/12/17 03:32:52 Function: TestEnoentGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentGet deleting in the background"
}
2021/12/17 03:32:54 Context: Bucket flush, response code: 200 dump: 
2021/12/17 03:33:04 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/17 03:33:09 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/17 03:33:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 03:33:11 Context: Bucket flush, response code: 200 dump: 
2021/12/17 03:33:21 Waiting for dst bucket item count to get to: 0 curr count: 91
2021/12/17 03:33:26 Waiting for dst bucket item count to get to: 0 curr count: 91
2021/12/17 03:33:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 03:33:26 Deploying app: TestEnoentGet. Item count src bucket: 0 dst bucket: 0
2021/12/17 03:33:27 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentGet' in metakv",
  "warnings": null
 }
}
2021/12/17 03:33:32 Waiting for app: TestEnoentGet to get deployed
2021/12/17 03:33:37 Waiting for app: TestEnoentGet to get deployed
2021/12/17 03:33:42 Waiting for app: TestEnoentGet to get deployed
2021/12/17 03:33:47 Waiting for app: TestEnoentGet to get deployed
2021/12/17 03:33:47 App: TestEnoentGet got deployed
2021/12/17 03:33:47 Starting bucket ops to source bucket
2021/12/17 03:33:52 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/17 03:33:52 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/17 03:33:52 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 03:33:52 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 03:33:57 Waiting for app: TestEnoentGet to get un-deployed
2021/12/17 03:34:02 Waiting for app: TestEnoentGet to get un-deployed
2021/12/17 03:34:07 Waiting for app: TestEnoentGet to get un-deployed
2021/12/17 03:34:12 Waiting for app: TestEnoentGet to get un-deployed
2021/12/17 03:34:12 App: TestEnoentGet got un-deployed
2021/12/17 03:34:12 No eventing-con process running
2021/12/17 03:34:12 Function: TestEnoentGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentGet deleting in the background"
}
2021/12/17 03:34:14 Context: Bucket flush, response code: 200 dump: 
2021/12/17 03:34:24 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/17 03:34:29 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/17 03:34:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 03:34:31 Context: Bucket flush, response code: 200 dump: 
2021/12/17 03:34:41 Waiting for dst bucket item count to get to: 0 curr count: 94
2021/12/17 03:34:46 Waiting for dst bucket item count to get to: 0 curr count: 94
2021/12/17 03:34:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEnoentGet (160.24s)
=== RUN   TestEnoentDelete
2021/12/17 03:34:46 Deploying app: TestEnoentDelete. Item count src bucket: 0 dst bucket: 0
2021/12/17 03:34:47 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentDelete' in metakv",
  "warnings": null
 }
}
2021/12/17 03:34:52 Waiting for app: TestEnoentDelete to get deployed
2021/12/17 03:34:57 Waiting for app: TestEnoentDelete to get deployed
2021/12/17 03:35:02 Waiting for app: TestEnoentDelete to get deployed
2021/12/17 03:35:07 Waiting for app: TestEnoentDelete to get deployed
2021/12/17 03:35:07 App: TestEnoentDelete got deployed
2021/12/17 03:35:07 Starting bucket ops to source bucket
2021/12/17 03:35:12 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/17 03:35:17 Waiting for dst bucket item count to get to: 100 curr count: 60
2021/12/17 03:35:22 Waiting for dst bucket item count to get to: 100 curr count: 60
2021/12/17 03:35:22 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/17 03:35:22 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 03:35:22 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 03:35:27 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/17 03:35:32 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/17 03:35:37 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/17 03:35:42 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/17 03:35:42 App: TestEnoentDelete got un-deployed
2021/12/17 03:35:42 No eventing-con process running
2021/12/17 03:35:42 Function: TestEnoentDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentDelete deleting in the background"
}
2021/12/17 03:35:44 Context: Bucket flush, response code: 200 dump: 
2021/12/17 03:35:54 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/17 03:35:59 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/17 03:35:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 03:36:01 Context: Bucket flush, response code: 200 dump: 
2021/12/17 03:36:11 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/17 03:36:16 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/17 03:36:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 03:36:16 Deploying app: TestEnoentDelete. Item count src bucket: 0 dst bucket: 0
2021/12/17 03:36:17 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentDelete' in metakv",
  "warnings": null
 }
}
2021/12/17 03:36:22 Waiting for app: TestEnoentDelete to get deployed
2021/12/17 03:36:27 Waiting for app: TestEnoentDelete to get deployed
2021/12/17 03:36:32 Waiting for app: TestEnoentDelete to get deployed
2021/12/17 03:36:37 Waiting for app: TestEnoentDelete to get deployed
2021/12/17 03:36:37 App: TestEnoentDelete got deployed
2021/12/17 03:36:37 Starting bucket ops to source bucket
2021/12/17 03:36:42 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/17 03:36:47 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/17 03:36:52 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/17 03:36:52 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/17 03:36:52 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 03:36:53 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 03:36:58 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/17 03:37:03 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/17 03:37:08 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/17 03:37:13 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/17 03:37:13 App: TestEnoentDelete got un-deployed
2021/12/17 03:37:13 No eventing-con process running
2021/12/17 03:37:13 Function: TestEnoentDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentDelete deleting in the background"
}
2021/12/17 03:37:14 Context: Bucket flush, response code: 200 dump: 
2021/12/17 03:37:24 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/17 03:37:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 03:37:26 Context: Bucket flush, response code: 200 dump: 
2021/12/17 03:37:36 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/17 03:37:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEnoentDelete (170.29s)
=== RUN   TestError
2021/12/17 03:37:46 Waiting for app: TestError to get un-deployed
2021/12/17 03:37:46 App: TestError got un-deployed
2021/12/17 03:37:46 No eventing-con process running
2021/12/17 03:37:46 Function: TestError delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestError not found"
 }
}
2021/12/17 03:37:48 Context: Bucket flush, response code: 200 dump: 
2021/12/17 03:37:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 03:37:55 Context: Bucket flush, response code: 200 dump: 
2021/12/17 03:38:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 03:38:00 Deploying app: TestError. Item count src bucket: 0 dst bucket: 0
2021/12/17 03:38:01 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestError' in metakv",
  "warnings": null
 }
}
2021/12/17 03:38:06 Waiting for app: TestError to get deployed
2021/12/17 03:38:11 Waiting for app: TestError to get deployed
2021/12/17 03:38:16 Waiting for app: TestError to get deployed
2021/12/17 03:38:21 Waiting for app: TestError to get deployed
2021/12/17 03:38:21 App: TestError got deployed
2021/12/17 03:38:21 Starting bucket ops to source bucket
2021/12/17 03:38:26 Waiting for dst bucket item count to get to: 300 curr count: 0
2021/12/17 03:38:31 Waiting for dst bucket item count to get to: 300 curr count: 0
2021/12/17 03:38:31 src & dst bucket item count matched up. src bucket count: 300 dst bucket count: 300
2021/12/17 03:38:31 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 03:38:31 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 03:38:36 Waiting for app: TestError to get un-deployed
2021/12/17 03:38:41 Waiting for app: TestError to get un-deployed
2021/12/17 03:38:46 Waiting for app: TestError to get un-deployed
2021/12/17 03:38:51 Waiting for app: TestError to get un-deployed
2021/12/17 03:38:51 App: TestError got un-deployed
2021/12/17 03:38:51 No eventing-con process running
2021/12/17 03:38:52 Function: TestError delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestError deleting in the background"
}
2021/12/17 03:38:53 Context: Bucket flush, response code: 200 dump: 
2021/12/17 03:39:03 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/17 03:39:08 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/17 03:39:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 03:39:10 Context: Bucket flush, response code: 200 dump: 
2021/12/17 03:39:20 Waiting for dst bucket item count to get to: 0 curr count: 203
2021/12/17 03:39:25 Waiting for dst bucket item count to get to: 0 curr count: 203
2021/12/17 03:39:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestError (108.88s)
=== RUN   TestCRLF
2021/12/17 03:39:35 Waiting for app: TestCRLF to get un-deployed
2021/12/17 03:39:35 App: TestCRLF got un-deployed
2021/12/17 03:39:35 No eventing-con process running
2021/12/17 03:39:35 Function: TestCRLF delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestCRLF not found"
 }
}
2021/12/17 03:39:37 Context: Bucket flush, response code: 200 dump: 
2021/12/17 03:39:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 03:39:44 Context: Bucket flush, response code: 200 dump: 
2021/12/17 03:39:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 03:39:49 Deploying app: TestCRLF. Item count src bucket: 0 dst bucket: 0
2021/12/17 03:39:50 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCRLF' in metakv",
  "warnings": null
 }
}
2021/12/17 03:39:55 Waiting for app: TestCRLF to get deployed
2021/12/17 03:40:00 Waiting for app: TestCRLF to get deployed
2021/12/17 03:40:05 Waiting for app: TestCRLF to get deployed
2021/12/17 03:40:10 Waiting for app: TestCRLF to get deployed
2021/12/17 03:40:10 App: TestCRLF got deployed
2021/12/17 03:40:10 Starting bucket ops to source bucket
2021/12/17 03:40:15 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/17 03:40:20 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/17 03:40:25 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/17 03:40:25 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/17 03:40:25 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 03:40:25 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 03:40:30 Waiting for app: TestCRLF to get un-deployed
2021/12/17 03:40:35 Waiting for app: TestCRLF to get un-deployed
2021/12/17 03:40:40 Waiting for app: TestCRLF to get un-deployed
2021/12/17 03:40:45 Waiting for app: TestCRLF to get un-deployed
2021/12/17 03:40:45 App: TestCRLF got un-deployed
2021/12/17 03:40:45 No eventing-con process running
2021/12/17 03:40:45 Function: TestCRLF delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCRLF deleting in the background"
}
2021/12/17 03:40:47 Context: Bucket flush, response code: 200 dump: 
2021/12/17 03:40:57 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/17 03:40:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 03:40:59 Context: Bucket flush, response code: 200 dump: 
2021/12/17 03:41:09 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/17 03:41:14 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/17 03:41:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCRLF (108.85s)
=== RUN   TestImportExport
2021/12/17 03:41:24 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/17 03:41:24 App: TestImportExport_n1ql_insert_on_update got un-deployed
2021/12/17 03:41:24 No eventing-con process running
2021/12/17 03:41:24 Function: TestImportExport_n1ql_insert_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestImportExport_n1ql_insert_on_update not found"
 }
}
2021/12/17 03:41:26 Context: Bucket flush, response code: 200 dump: 
2021/12/17 03:41:31 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 03:41:33 Context: Bucket flush, response code: 200 dump: 
2021/12/17 03:41:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 03:41:43 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/17 03:41:43 App: TestImportExport_bucket_op_on_update got un-deployed
2021/12/17 03:41:43 No eventing-con process running
2021/12/17 03:41:43 Function: TestImportExport_bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestImportExport_bucket_op_on_update not found"
 }
}
2021/12/17 03:41:44 Context: Bucket flush, response code: 200 dump: 
2021/12/17 03:41:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 03:41:51 Context: Bucket flush, response code: 200 dump: 
2021/12/17 03:41:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 03:41:56 Deploying app: TestImportExport_n1ql_insert_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/17 03:41:57 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestImportExport_n1ql_insert_on_update' in metakv",
  "warnings": null
 }
}
2021/12/17 03:41:57 Deploying app: TestImportExport_bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/17 03:41:59 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestImportExport_bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/17 03:42:04 Waiting for app: TestImportExport_n1ql_insert_on_update to get deployed
2021/12/17 03:42:09 Waiting for app: TestImportExport_n1ql_insert_on_update to get deployed
2021/12/17 03:42:14 Waiting for app: TestImportExport_n1ql_insert_on_update to get deployed
2021/12/17 03:42:19 Waiting for app: TestImportExport_n1ql_insert_on_update to get deployed
2021/12/17 03:42:19 App: TestImportExport_n1ql_insert_on_update got deployed
2021/12/17 03:42:24 Waiting for app: TestImportExport_bucket_op_on_update to get deployed
2021/12/17 03:42:29 Waiting for app: TestImportExport_bucket_op_on_update to get deployed
2021/12/17 03:42:34 Waiting for app: TestImportExport_bucket_op_on_update to get deployed
2021/12/17 03:42:34 App: TestImportExport_bucket_op_on_update got deployed
2021/12/17 03:42:34 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 03:42:39 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/17 03:42:44 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/17 03:42:49 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/17 03:42:54 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/17 03:42:54 App: TestImportExport_n1ql_insert_on_update got un-deployed
2021/12/17 03:42:54 No eventing-con process running
2021/12/17 03:42:54 Function: TestImportExport_n1ql_insert_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestImportExport_n1ql_insert_on_update deleting in the background"
}
2021/12/17 03:42:54 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 03:42:59 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/17 03:43:04 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/17 03:43:09 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/17 03:43:14 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/17 03:43:14 App: TestImportExport_bucket_op_on_update got un-deployed
2021/12/17 03:43:14 No eventing-con process running
2021/12/17 03:43:14 Function: TestImportExport_bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestImportExport_bucket_op_on_update deleting in the background"
}
2021/12/17 03:43:31 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/17 03:43:31 App: TestImportExport_bucket_op_on_update got un-deployed
2021/12/17 03:43:31 No eventing-con process running
2021/12/17 03:43:31 Function: TestImportExport_bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestImportExport_bucket_op_on_update deleting in the background"
}
2021/12/17 03:43:33 Context: Bucket flush, response code: 200 dump: 
2021/12/17 03:43:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 03:43:40 Context: Bucket flush, response code: 200 dump: 
2021/12/17 03:43:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 03:43:50 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/17 03:43:50 App: TestImportExport_n1ql_insert_on_update got un-deployed
2021/12/17 03:43:50 No eventing-con process running
2021/12/17 03:43:50 Function: TestImportExport_n1ql_insert_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestImportExport_n1ql_insert_on_update deleting in the background"
}
2021/12/17 03:43:52 Context: Bucket flush, response code: 200 dump: 
2021/12/17 03:43:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 03:43:59 Context: Bucket flush, response code: 200 dump: 
2021/12/17 03:44:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestImportExport (169.70s)
=== RUN   TestDeployUndeployLoopNonDefaultSettings
2021/12/17 03:44:14 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/17 03:44:14 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/17 03:44:14 No eventing-con process running
2021/12/17 03:44:14 Function: TestDeployUndeployLoopNonDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDeployUndeployLoopNonDefaultSettings not found"
 }
}
2021/12/17 03:44:15 Context: Bucket flush, response code: 200 dump: 
2021/12/17 03:44:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 03:44:22 Context: Bucket flush, response code: 200 dump: 
2021/12/17 03:44:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 03:44:27 Deploying app: TestDeployUndeployLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/17 03:44:28 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/17 03:44:28 Starting bucket ops to source bucket
2021/12/17 03:44:37 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 03:44:42 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 03:44:47 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 03:44:52 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 03:44:57 Waiting for dst bucket item count to get to: 5000 curr count: 2500
2021/12/17 03:44:57 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/17 03:45:02 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get deployed
2021/12/17 03:45:02 App: TestDeployUndeployLoopNonDefaultSettings got deployed
2021/12/17 03:45:02 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 03:45:02 Undeploying app: bucket_op_on_update
2021/12/17 03:45:02 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 03:45:07 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/17 03:45:12 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/17 03:45:17 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/17 03:45:22 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/17 03:45:22 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/17 03:45:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 03:45:22 No eventing-con process running
2021/12/17 03:45:24 Context: Bucket flush, response code: 200 dump: 
2021/12/17 03:45:31 Context: Bucket flush, response code: 200 dump: 
2021/12/17 03:46:06 Deploying app: TestDeployUndeployLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/17 03:46:07 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/17 03:46:07 Starting bucket ops to source bucket
2021/12/17 03:46:15 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 03:46:20 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 03:46:25 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 03:46:30 Waiting for dst bucket item count to get to: 5000 curr count: 844
2021/12/17 03:46:35 Waiting for dst bucket item count to get to: 5000 curr count: 844
2021/12/17 03:46:35 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/17 03:46:40 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get deployed
2021/12/17 03:46:40 App: TestDeployUndeployLoopNonDefaultSettings got deployed
2021/12/17 03:46:40 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 03:46:40 Undeploying app: bucket_op_on_update
2021/12/17 03:46:40 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 03:46:45 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/17 03:46:50 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/17 03:46:55 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/17 03:47:00 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/17 03:47:00 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/17 03:47:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 03:47:00 No eventing-con process running
2021/12/17 03:47:02 Context: Bucket flush, response code: 200 dump: 
2021/12/17 03:47:09 Context: Bucket flush, response code: 200 dump: 
2021/12/17 03:47:44 Deploying app: TestDeployUndeployLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/17 03:47:45 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/17 03:47:45 Starting bucket ops to source bucket
2021/12/17 03:47:53 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 03:47:58 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 03:48:03 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 03:48:08 Waiting for dst bucket item count to get to: 5000 curr count: 2500
2021/12/17 03:48:13 Waiting for dst bucket item count to get to: 5000 curr count: 2500
2021/12/17 03:48:13 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/17 03:48:18 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get deployed
2021/12/17 03:48:18 App: TestDeployUndeployLoopNonDefaultSettings got deployed
2021/12/17 03:48:18 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 03:48:18 Undeploying app: bucket_op_on_update
2021/12/17 03:48:19 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 03:48:24 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/17 03:48:29 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/17 03:48:34 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/17 03:48:39 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/17 03:48:39 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/17 03:48:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 03:48:39 No eventing-con process running
2021/12/17 03:48:40 Context: Bucket flush, response code: 200 dump: 
2021/12/17 03:48:48 Context: Bucket flush, response code: 200 dump: 
2021/12/17 03:49:23 Deploying app: TestDeployUndeployLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/17 03:49:24 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/17 03:49:24 Starting bucket ops to source bucket
2021/12/17 03:49:32 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 03:49:37 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 03:49:42 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 03:49:47 Waiting for dst bucket item count to get to: 5000 curr count: 1843
2021/12/17 03:49:52 Waiting for dst bucket item count to get to: 5000 curr count: 4343
2021/12/17 03:49:52 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/17 03:49:57 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get deployed
2021/12/17 03:49:57 App: TestDeployUndeployLoopNonDefaultSettings got deployed
2021/12/17 03:49:57 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 03:49:57 Undeploying app: bucket_op_on_update
2021/12/17 03:49:57 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 03:50:02 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/17 03:50:07 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/17 03:50:12 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/17 03:50:17 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/17 03:50:17 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/17 03:50:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 03:50:17 No eventing-con process running
2021/12/17 03:50:19 Context: Bucket flush, response code: 200 dump: 
2021/12/17 03:50:26 Context: Bucket flush, response code: 200 dump: 
2021/12/17 03:51:01 Deploying app: TestDeployUndeployLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/17 03:51:02 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/17 03:51:02 Starting bucket ops to source bucket
2021/12/17 03:51:10 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 03:51:15 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 03:51:20 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 03:51:25 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 03:51:25 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/17 03:51:30 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get deployed
2021/12/17 03:51:30 App: TestDeployUndeployLoopNonDefaultSettings got deployed
2021/12/17 03:51:30 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 03:51:30 Undeploying app: bucket_op_on_update
2021/12/17 03:51:30 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 03:51:35 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/17 03:51:40 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/17 03:51:45 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/17 03:51:50 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/17 03:51:50 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/17 03:51:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 03:51:50 No eventing-con process running
2021/12/17 03:51:52 Context: Bucket flush, response code: 200 dump: 
2021/12/17 03:51:59 Context: Bucket flush, response code: 200 dump: 
2021/12/17 03:52:35 Function: TestDeployUndeployLoopNonDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDeployUndeployLoopNonDefaultSettings deleting in the background"
}
--- PASS: TestDeployUndeployLoopNonDefaultSettings (510.92s)
=== RUN   TestOnUpdateN1QLOp
2021/12/17 03:52:45 Waiting for app: TestOnUpdateN1QLOp to get un-deployed
2021/12/17 03:52:45 App: TestOnUpdateN1QLOp got un-deployed
2021/12/17 03:52:45 No eventing-con process running
2021/12/17 03:52:45 Function: TestOnUpdateN1QLOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateN1QLOp not found"
 }
}
2021/12/17 03:52:46 Context: Bucket flush, response code: 200 dump: 
2021/12/17 03:52:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 03:52:53 Context: Bucket flush, response code: 200 dump: 
2021/12/17 03:52:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 03:52:58 Deploying app: TestOnUpdateN1QLOp. Item count src bucket: 0 dst bucket: 0
2021/12/17 03:52:59 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateN1QLOp' in metakv",
  "warnings": null
 }
}
2021/12/17 03:52:59 Starting bucket ops to source bucket
2021/12/17 03:53:07 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 03:53:12 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 03:53:17 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 03:53:22 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 03:53:27 Waiting for dst bucket item count to get to: 5000 curr count: 2556
2021/12/17 03:53:27 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/17 03:53:27 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 03:53:27 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 03:53:32 Waiting for app: TestOnUpdateN1QLOp to get un-deployed
2021/12/17 03:53:37 Waiting for app: TestOnUpdateN1QLOp to get un-deployed
2021/12/17 03:53:42 Waiting for app: TestOnUpdateN1QLOp to get un-deployed
2021/12/17 03:53:47 Waiting for app: TestOnUpdateN1QLOp to get un-deployed
2021/12/17 03:53:48 App: TestOnUpdateN1QLOp got un-deployed
2021/12/17 03:53:48 No eventing-con process running
2021/12/17 03:53:48 Function: TestOnUpdateN1QLOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateN1QLOp deleting in the background"
}
2021/12/17 03:53:49 Context: Bucket flush, response code: 200 dump: 
2021/12/17 03:53:59 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/17 03:54:04 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/17 03:54:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 03:54:06 Context: Bucket flush, response code: 200 dump: 
2021/12/17 03:54:16 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/17 03:54:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateN1QLOp (101.68s)
=== RUN   TestOnUpdateBucketOpDefaultSettings
2021/12/17 03:54:26 Waiting for app: TestOnUpdateBucketOpDefaultSettings to get un-deployed
2021/12/17 03:54:26 App: TestOnUpdateBucketOpDefaultSettings got un-deployed
2021/12/17 03:54:26 No eventing-con process running
2021/12/17 03:54:26 Function: TestOnUpdateBucketOpDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateBucketOpDefaultSettings not found"
 }
}
2021/12/17 03:54:28 Context: Bucket flush, response code: 200 dump: 
2021/12/17 03:54:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 03:54:35 Context: Bucket flush, response code: 200 dump: 
2021/12/17 03:54:40 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 03:54:40 Deploying app: TestOnUpdateBucketOpDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/17 03:54:41 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateBucketOpDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/17 03:54:41 Starting bucket ops to source bucket
2021/12/17 03:54:49 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 03:54:54 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 03:54:59 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 03:55:04 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 03:55:04 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/17 03:55:04 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 03:55:04 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 03:55:09 Waiting for app: TestOnUpdateBucketOpDefaultSettings to get un-deployed
2021/12/17 03:55:14 Waiting for app: TestOnUpdateBucketOpDefaultSettings to get un-deployed
2021/12/17 03:55:19 Waiting for app: TestOnUpdateBucketOpDefaultSettings to get un-deployed
2021/12/17 03:55:24 Waiting for app: TestOnUpdateBucketOpDefaultSettings to get un-deployed
2021/12/17 03:55:24 App: TestOnUpdateBucketOpDefaultSettings got un-deployed
2021/12/17 03:55:24 No eventing-con process running
2021/12/17 03:55:24 Function: TestOnUpdateBucketOpDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateBucketOpDefaultSettings deleting in the background"
}
2021/12/17 03:55:26 Context: Bucket flush, response code: 200 dump: 
2021/12/17 03:55:36 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/17 03:55:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 03:55:38 Context: Bucket flush, response code: 200 dump: 
2021/12/17 03:55:48 Waiting for dst bucket item count to get to: 0 curr count: 3225
2021/12/17 03:55:53 Waiting for dst bucket item count to get to: 0 curr count: 725
2021/12/17 03:55:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateBucketOpDefaultSettings (96.63s)
=== RUN   TestOnUpdateBucketOpNonDefaultSettings
2021/12/17 03:56:03 Waiting for app: TestOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/17 03:56:03 App: TestOnUpdateBucketOpNonDefaultSettings got un-deployed
2021/12/17 03:56:03 No eventing-con process running
2021/12/17 03:56:03 Function: TestOnUpdateBucketOpNonDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateBucketOpNonDefaultSettings not found"
 }
}
2021/12/17 03:56:05 Context: Bucket flush, response code: 200 dump: 
2021/12/17 03:56:10 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 03:56:11 Context: Bucket flush, response code: 200 dump: 
2021/12/17 03:56:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 03:56:17 Deploying app: TestOnUpdateBucketOpNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/17 03:56:18 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateBucketOpNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/17 03:56:18 Starting bucket ops to source bucket
2021/12/17 03:56:26 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 03:56:31 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 03:56:36 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 03:56:41 Waiting for dst bucket item count to get to: 5000 curr count: 284
2021/12/17 03:56:46 Waiting for dst bucket item count to get to: 5000 curr count: 284
2021/12/17 03:56:46 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/17 03:56:46 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 03:56:46 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 03:56:51 Waiting for app: TestOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/17 03:56:56 Waiting for app: TestOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/17 03:57:01 Waiting for app: TestOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/17 03:57:06 Waiting for app: TestOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/17 03:57:06 App: TestOnUpdateBucketOpNonDefaultSettings got un-deployed
2021/12/17 03:57:06 No eventing-con process running
2021/12/17 03:57:06 Function: TestOnUpdateBucketOpNonDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateBucketOpNonDefaultSettings deleting in the background"
}
2021/12/17 03:57:08 Context: Bucket flush, response code: 200 dump: 
2021/12/17 03:57:18 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/17 03:57:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 03:57:19 Context: Bucket flush, response code: 200 dump: 
2021/12/17 03:57:29 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/17 03:57:34 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/17 03:57:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateBucketOpNonDefaultSettings (101.64s)
=== RUN   TestOnUpdateBucketOpDefaultSettings10K
2021/12/17 03:57:45 Waiting for app: TestOnUpdateBucketOpDefaultSettings10K to get un-deployed
2021/12/17 03:57:45 App: TestOnUpdateBucketOpDefaultSettings10K got un-deployed
2021/12/17 03:57:45 No eventing-con process running
2021/12/17 03:57:45 Function: TestOnUpdateBucketOpDefaultSettings10K delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateBucketOpDefaultSettings10K not found"
 }
}
2021/12/17 03:57:46 Context: Bucket flush, response code: 200 dump: 
2021/12/17 03:57:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 03:57:53 Context: Bucket flush, response code: 200 dump: 
2021/12/17 03:57:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 03:57:58 Deploying app: TestOnUpdateBucketOpDefaultSettings10K. Item count src bucket: 0 dst bucket: 0
2021/12/17 03:57:59 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateBucketOpDefaultSettings10K' in metakv",
  "warnings": null
 }
}
2021/12/17 03:57:59 Starting bucket ops to source bucket
2021/12/17 03:58:07 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 03:58:12 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 03:58:17 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 03:58:22 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 03:58:27 Waiting for dst bucket item count to get to: 5000 curr count: 2500
2021/12/17 03:58:27 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/17 03:58:27 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 03:58:27 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 03:58:32 Waiting for app: TestOnUpdateBucketOpDefaultSettings10K to get un-deployed
2021/12/17 03:58:38 Waiting for app: TestOnUpdateBucketOpDefaultSettings10K to get un-deployed
2021/12/17 03:58:43 Waiting for app: TestOnUpdateBucketOpDefaultSettings10K to get un-deployed
2021/12/17 03:58:48 Waiting for app: TestOnUpdateBucketOpDefaultSettings10K to get un-deployed
2021/12/17 03:58:48 App: TestOnUpdateBucketOpDefaultSettings10K got un-deployed
2021/12/17 03:58:48 No eventing-con process running
2021/12/17 03:58:48 Function: TestOnUpdateBucketOpDefaultSettings10K delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateBucketOpDefaultSettings10K deleting in the background"
}
2021/12/17 03:58:49 Context: Bucket flush, response code: 200 dump: 
2021/12/17 03:58:59 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/17 03:59:04 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/17 03:59:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 03:59:06 Context: Bucket flush, response code: 200 dump: 
2021/12/17 03:59:16 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/17 03:59:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateBucketOpDefaultSettings10K (101.75s)
=== RUN   TestOnUpdateBucketOpDefaultSettings100K
2021/12/17 03:59:26 Waiting for app: TestOnUpdateBucketOpDefaultSettings100K to get un-deployed
2021/12/17 03:59:26 App: TestOnUpdateBucketOpDefaultSettings100K got un-deployed
2021/12/17 03:59:26 No eventing-con process running
2021/12/17 03:59:26 Function: TestOnUpdateBucketOpDefaultSettings100K delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateBucketOpDefaultSettings100K not found"
 }
}
2021/12/17 03:59:28 Context: Bucket flush, response code: 200 dump: 
2021/12/17 03:59:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 03:59:35 Context: Bucket flush, response code: 200 dump: 
2021/12/17 03:59:40 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 03:59:40 Deploying app: TestOnUpdateBucketOpDefaultSettings100K. Item count src bucket: 0 dst bucket: 0
2021/12/17 03:59:41 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateBucketOpDefaultSettings100K' in metakv",
  "warnings": null
 }
}
2021/12/17 03:59:41 Starting bucket ops to source bucket
2021/12/17 03:59:50 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 03:59:55 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 04:00:00 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 04:00:05 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 04:00:05 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/17 04:00:05 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 04:00:05 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 04:00:10 Waiting for app: TestOnUpdateBucketOpDefaultSettings100K to get un-deployed
2021/12/17 04:00:15 Waiting for app: TestOnUpdateBucketOpDefaultSettings100K to get un-deployed
2021/12/17 04:00:20 Waiting for app: TestOnUpdateBucketOpDefaultSettings100K to get un-deployed
2021/12/17 04:00:25 Waiting for app: TestOnUpdateBucketOpDefaultSettings100K to get un-deployed
2021/12/17 04:00:25 App: TestOnUpdateBucketOpDefaultSettings100K got un-deployed
2021/12/17 04:00:25 No eventing-con process running
2021/12/17 04:00:25 Function: TestOnUpdateBucketOpDefaultSettings100K delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateBucketOpDefaultSettings100K deleting in the background"
}
2021/12/17 04:00:27 Context: Bucket flush, response code: 200 dump: 
2021/12/17 04:00:37 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/17 04:00:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 04:00:38 Context: Bucket flush, response code: 200 dump: 
2021/12/17 04:00:48 Waiting for dst bucket item count to get to: 0 curr count: 4318
2021/12/17 04:00:53 Waiting for dst bucket item count to get to: 0 curr count: 1818
2021/12/17 04:00:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateBucketOpDefaultSettings100K (97.24s)
=== RUN   TestOnDeleteBucketOp
2021/12/17 04:01:03 Waiting for app: TestOnDeleteBucketOp to get un-deployed
2021/12/17 04:01:04 App: TestOnDeleteBucketOp got un-deployed
2021/12/17 04:01:04 No eventing-con process running
2021/12/17 04:01:04 Function: TestOnDeleteBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnDeleteBucketOp not found"
 }
}
2021/12/17 04:01:05 Context: Bucket flush, response code: 200 dump: 
2021/12/17 04:01:10 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 04:01:12 Context: Bucket flush, response code: 200 dump: 
2021/12/17 04:01:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 04:01:17 Deploying app: TestOnDeleteBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/17 04:01:18 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnDeleteBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/17 04:01:18 Starting bucket ops to source bucket
2021/12/17 04:01:29 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 04:01:34 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 04:01:39 Waiting for dst bucket item count to get to: 5000 curr count: 111
2021/12/17 04:01:44 Waiting for dst bucket item count to get to: 5000 curr count: 2615
2021/12/17 04:01:44 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/17 04:01:44 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 04:01:44 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 04:01:49 Waiting for app: TestOnDeleteBucketOp to get un-deployed
2021/12/17 04:01:54 Waiting for app: TestOnDeleteBucketOp to get un-deployed
2021/12/17 04:01:59 Waiting for app: TestOnDeleteBucketOp to get un-deployed
2021/12/17 04:02:04 Waiting for app: TestOnDeleteBucketOp to get un-deployed
2021/12/17 04:02:04 App: TestOnDeleteBucketOp got un-deployed
2021/12/17 04:02:04 No eventing-con process running
2021/12/17 04:02:04 Function: TestOnDeleteBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnDeleteBucketOp deleting in the background"
}
2021/12/17 04:02:06 Context: Bucket flush, response code: 200 dump: 
2021/12/17 04:02:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 04:02:13 Context: Bucket flush, response code: 200 dump: 
2021/12/17 04:02:23 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/17 04:02:28 Waiting for dst bucket item count to get to: 0 curr count: 2496
2021/12/17 04:02:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnDeleteBucketOp (94.95s)
=== RUN   TestOnDeleteBucketOp5K
2021/12/17 04:02:38 Waiting for app: TestOnDeleteBucketOp5K to get un-deployed
2021/12/17 04:02:38 App: TestOnDeleteBucketOp5K got un-deployed
2021/12/17 04:02:38 No eventing-con process running
2021/12/17 04:02:38 Function: TestOnDeleteBucketOp5K delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnDeleteBucketOp5K not found"
 }
}
2021/12/17 04:02:40 Context: Bucket flush, response code: 200 dump: 
2021/12/17 04:02:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 04:02:47 Context: Bucket flush, response code: 200 dump: 
2021/12/17 04:02:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 04:02:52 Deploying app: TestOnDeleteBucketOp5K. Item count src bucket: 0 dst bucket: 0
2021/12/17 04:02:53 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnDeleteBucketOp5K' in metakv",
  "warnings": null
 }
}
2021/12/17 04:02:53 Starting bucket ops to source bucket
2021/12/17 04:03:04 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 04:03:09 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 04:03:14 Waiting for dst bucket item count to get to: 5000 curr count: 2504
2021/12/17 04:03:14 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/17 04:03:14 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 04:03:14 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 04:03:19 Waiting for app: TestOnDeleteBucketOp5K to get un-deployed
2021/12/17 04:03:24 Waiting for app: TestOnDeleteBucketOp5K to get un-deployed
2021/12/17 04:03:29 Waiting for app: TestOnDeleteBucketOp5K to get un-deployed
2021/12/17 04:03:34 Waiting for app: TestOnDeleteBucketOp5K to get un-deployed
2021/12/17 04:03:34 App: TestOnDeleteBucketOp5K got un-deployed
2021/12/17 04:03:34 No eventing-con process running
2021/12/17 04:03:34 Function: TestOnDeleteBucketOp5K delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnDeleteBucketOp5K deleting in the background"
}
2021/12/17 04:03:36 Context: Bucket flush, response code: 200 dump: 
2021/12/17 04:03:41 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 04:03:43 Context: Bucket flush, response code: 200 dump: 
2021/12/17 04:03:53 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/17 04:03:58 Waiting for dst bucket item count to get to: 0 curr count: 2496
2021/12/17 04:03:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnDeleteBucketOp5K (89.95s)
=== RUN   TestTimerBucketOp
2021/12/17 04:04:08 Waiting for app: TestTimerBucketOp to get un-deployed
2021/12/17 04:04:08 App: TestTimerBucketOp got un-deployed
2021/12/17 04:04:08 No eventing-con process running
2021/12/17 04:04:08 Function: TestTimerBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestTimerBucketOp not found"
 }
}
2021/12/17 04:04:10 Context: Bucket flush, response code: 200 dump: 
2021/12/17 04:04:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 04:04:17 Context: Bucket flush, response code: 200 dump: 
2021/12/17 04:04:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 04:04:22 Deploying app: TestTimerBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/17 04:04:23 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestTimerBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/17 04:04:23 Starting bucket ops to source bucket
2021/12/17 04:04:31 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 04:04:36 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 04:04:41 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 04:04:46 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 04:04:51 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 04:04:56 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 04:05:01 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 04:05:06 Waiting for dst bucket item count to get to: 5000 curr count: 2500
2021/12/17 04:05:06 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/17 04:05:06 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 04:05:06 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 04:05:11 Waiting for app: TestTimerBucketOp to get un-deployed
2021/12/17 04:05:16 Waiting for app: TestTimerBucketOp to get un-deployed
2021/12/17 04:05:21 Waiting for app: TestTimerBucketOp to get un-deployed
2021/12/17 04:05:26 Waiting for app: TestTimerBucketOp to get un-deployed
2021/12/17 04:05:26 App: TestTimerBucketOp got un-deployed
2021/12/17 04:05:26 No eventing-con process running
2021/12/17 04:05:26 Function: TestTimerBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestTimerBucketOp deleting in the background"
}
2021/12/17 04:05:28 Context: Bucket flush, response code: 200 dump: 
2021/12/17 04:05:38 Waiting for dst bucket item count to get to: 0 curr count: 3333
2021/12/17 04:05:43 Waiting for dst bucket item count to get to: 0 curr count: 833
2021/12/17 04:05:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 04:05:45 Context: Bucket flush, response code: 200 dump: 
2021/12/17 04:05:55 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/17 04:05:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestTimerBucketOp (116.42s)
=== RUN   TestTimerInPastBucketOp
2021/12/17 04:06:05 Waiting for app: TestTimerInPastBucketOp to get un-deployed
2021/12/17 04:06:05 App: TestTimerInPastBucketOp got un-deployed
2021/12/17 04:06:05 No eventing-con process running
2021/12/17 04:06:05 Function: TestTimerInPastBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestTimerInPastBucketOp not found"
 }
}
2021/12/17 04:06:07 Context: Bucket flush, response code: 200 dump: 
2021/12/17 04:06:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 04:06:13 Context: Bucket flush, response code: 200 dump: 
2021/12/17 04:06:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 04:06:18 Deploying app: TestTimerInPastBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/17 04:06:20 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestTimerInPastBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/17 04:06:20 Starting bucket ops to source bucket
2021/12/17 04:06:28 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 04:06:33 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 04:06:38 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 04:06:43 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 04:06:48 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 04:06:53 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 04:06:58 Waiting for dst bucket item count to get to: 5000 curr count: 1287
2021/12/17 04:07:03 Waiting for dst bucket item count to get to: 5000 curr count: 3787
2021/12/17 04:07:03 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/17 04:07:03 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 04:07:03 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 04:07:08 Waiting for app: TestTimerInPastBucketOp to get un-deployed
2021/12/17 04:07:13 Waiting for app: TestTimerInPastBucketOp to get un-deployed
2021/12/17 04:07:18 Waiting for app: TestTimerInPastBucketOp to get un-deployed
2021/12/17 04:07:23 Waiting for app: TestTimerInPastBucketOp to get un-deployed
2021/12/17 04:07:23 App: TestTimerInPastBucketOp got un-deployed
2021/12/17 04:07:23 No eventing-con process running
2021/12/17 04:07:23 Function: TestTimerInPastBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestTimerInPastBucketOp deleting in the background"
}
2021/12/17 04:07:25 Context: Bucket flush, response code: 200 dump: 
2021/12/17 04:07:35 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/17 04:07:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 04:07:36 Context: Bucket flush, response code: 200 dump: 
2021/12/17 04:07:46 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/17 04:07:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestTimerInPastBucketOp (111.55s)
=== RUN   TestDeployUndeployLoopTimer
2021/12/17 04:07:56 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/17 04:07:56 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/17 04:07:56 No eventing-con process running
2021/12/17 04:07:56 Function: TestDeployUndeployLoopTimer delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDeployUndeployLoopTimer not found"
 }
}
2021/12/17 04:07:58 Context: Bucket flush, response code: 200 dump: 
2021/12/17 04:08:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 04:08:05 Context: Bucket flush, response code: 200 dump: 
2021/12/17 04:08:10 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 04:08:10 Deploying app: TestDeployUndeployLoopTimer. Item count src bucket: 0 dst bucket: 0
2021/12/17 04:08:11 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopTimer' in metakv",
  "warnings": null
 }
}
2021/12/17 04:08:11 Starting bucket ops to source bucket
2021/12/17 04:08:19 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 04:08:24 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 04:08:29 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 04:08:34 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 04:08:39 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 04:08:44 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 04:08:49 Waiting for dst bucket item count to get to: 5000 curr count: 1948
2021/12/17 04:08:54 Waiting for dst bucket item count to get to: 5000 curr count: 1948
2021/12/17 04:08:54 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/17 04:08:59 Waiting for app: TestDeployUndeployLoopTimer to get deployed
2021/12/17 04:08:59 App: TestDeployUndeployLoopTimer got deployed
2021/12/17 04:08:59 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 04:08:59 Undeploying app: bucket_op_with_timer
2021/12/17 04:08:59 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 04:09:04 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/17 04:09:09 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/17 04:09:14 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/17 04:09:19 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/17 04:09:19 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/17 04:09:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 04:09:20 No eventing-con process running
2021/12/17 04:09:21 Context: Bucket flush, response code: 200 dump: 
2021/12/17 04:09:29 Context: Bucket flush, response code: 200 dump: 
2021/12/17 04:10:04 Deploying app: TestDeployUndeployLoopTimer. Item count src bucket: 0 dst bucket: 0
2021/12/17 04:10:05 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopTimer' in metakv",
  "warnings": null
 }
}
2021/12/17 04:10:05 Starting bucket ops to source bucket
2021/12/17 04:10:13 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 04:10:18 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 04:10:23 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 04:10:28 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 04:10:33 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 04:10:38 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 04:10:43 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 04:10:43 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/17 04:10:48 Waiting for app: TestDeployUndeployLoopTimer to get deployed
2021/12/17 04:10:48 App: TestDeployUndeployLoopTimer got deployed
2021/12/17 04:10:48 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 04:10:48 Undeploying app: bucket_op_with_timer
2021/12/17 04:10:48 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 04:10:53 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/17 04:10:58 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/17 04:11:03 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/17 04:11:08 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/17 04:11:08 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/17 04:11:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 04:11:08 No eventing-con process running
2021/12/17 04:11:10 Context: Bucket flush, response code: 200 dump: 
2021/12/17 04:11:17 Context: Bucket flush, response code: 200 dump: 
2021/12/17 04:11:52 Deploying app: TestDeployUndeployLoopTimer. Item count src bucket: 0 dst bucket: 0
2021/12/17 04:11:53 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopTimer' in metakv",
  "warnings": null
 }
}
2021/12/17 04:11:53 Starting bucket ops to source bucket
2021/12/17 04:12:01 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 04:12:06 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 04:12:11 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 04:12:16 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 04:12:21 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 04:12:26 Waiting for dst bucket item count to get to: 5000 curr count: 391
2021/12/17 04:12:31 Waiting for dst bucket item count to get to: 5000 curr count: 769
2021/12/17 04:12:36 Waiting for dst bucket item count to get to: 5000 curr count: 2878
2021/12/17 04:12:36 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/17 04:12:41 Waiting for app: TestDeployUndeployLoopTimer to get deployed
2021/12/17 04:12:41 App: TestDeployUndeployLoopTimer got deployed
2021/12/17 04:12:41 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 04:12:41 Undeploying app: bucket_op_with_timer
2021/12/17 04:12:41 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 04:12:46 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/17 04:12:51 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/17 04:12:56 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/17 04:13:01 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/17 04:13:01 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/17 04:13:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 04:13:01 No eventing-con process running
2021/12/17 04:13:03 Context: Bucket flush, response code: 200 dump: 
2021/12/17 04:13:10 Context: Bucket flush, response code: 200 dump: 
2021/12/17 04:13:45 Deploying app: TestDeployUndeployLoopTimer. Item count src bucket: 0 dst bucket: 0
2021/12/17 04:13:47 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopTimer' in metakv",
  "warnings": null
 }
}
2021/12/17 04:13:47 Starting bucket ops to source bucket
2021/12/17 04:13:55 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 04:14:00 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 04:14:05 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 04:14:10 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 04:14:15 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 04:14:20 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 04:14:25 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 04:14:25 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/17 04:14:30 Waiting for app: TestDeployUndeployLoopTimer to get deployed
2021/12/17 04:14:30 App: TestDeployUndeployLoopTimer got deployed
2021/12/17 04:14:30 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 04:14:30 Undeploying app: bucket_op_with_timer
2021/12/17 04:14:30 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 04:14:35 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/17 04:14:40 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/17 04:14:45 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/17 04:14:50 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/17 04:14:50 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/17 04:14:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 04:14:50 No eventing-con process running
2021/12/17 04:14:52 Context: Bucket flush, response code: 200 dump: 
2021/12/17 04:14:59 Context: Bucket flush, response code: 200 dump: 
2021/12/17 04:15:34 Deploying app: TestDeployUndeployLoopTimer. Item count src bucket: 0 dst bucket: 0
2021/12/17 04:15:35 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopTimer' in metakv",
  "warnings": null
 }
}
2021/12/17 04:15:35 Starting bucket ops to source bucket
2021/12/17 04:15:43 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 04:15:48 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 04:15:53 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 04:15:58 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 04:16:03 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 04:16:08 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 04:16:13 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 04:16:13 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/17 04:16:18 Waiting for app: TestDeployUndeployLoopTimer to get deployed
2021/12/17 04:16:18 App: TestDeployUndeployLoopTimer got deployed
2021/12/17 04:16:18 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 04:16:18 Undeploying app: bucket_op_with_timer
2021/12/17 04:16:18 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 04:16:23 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/17 04:16:28 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/17 04:16:33 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/17 04:16:38 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/17 04:16:38 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/17 04:16:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 04:16:38 No eventing-con process running
2021/12/17 04:16:40 Context: Bucket flush, response code: 200 dump: 
2021/12/17 04:16:47 Context: Bucket flush, response code: 200 dump: 
2021/12/17 04:17:22 Function: TestDeployUndeployLoopTimer delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDeployUndeployLoopTimer deleting in the background"
}
--- PASS: TestDeployUndeployLoopTimer (576.03s)
=== RUN   TestMultipleHandlers
2021/12/17 04:17:32 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get un-deployed
2021/12/17 04:17:32 App: TestMultipleHandlers_bucket_op_on_update got un-deployed
2021/12/17 04:17:32 No eventing-con process running
2021/12/17 04:17:32 Function: TestMultipleHandlers_bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMultipleHandlers_bucket_op_on_update not found"
 }
}
2021/12/17 04:17:34 Context: Bucket flush, response code: 200 dump: 
2021/12/17 04:17:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 04:17:41 Context: Bucket flush, response code: 200 dump: 
2021/12/17 04:17:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 04:17:51 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get un-deployed
2021/12/17 04:17:51 App: TestMultipleHandlers_bucket_op_on_delete got un-deployed
2021/12/17 04:17:51 No eventing-con process running
2021/12/17 04:17:51 Function: TestMultipleHandlers_bucket_op_on_delete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMultipleHandlers_bucket_op_on_delete not found"
 }
}
2021/12/17 04:17:53 Context: Bucket flush, response code: 200 dump: 
2021/12/17 04:17:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 04:18:00 Context: Bucket flush, response code: 200 dump: 
2021/12/17 04:18:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 04:18:05 Deploying app: TestMultipleHandlers_bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/17 04:18:06 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMultipleHandlers_bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/17 04:18:06 Deploying app: TestMultipleHandlers_bucket_op_on_delete. Item count src bucket: 0 dst bucket: 0
2021/12/17 04:18:07 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMultipleHandlers_bucket_op_on_delete' in metakv",
  "warnings": null
 }
}
2021/12/17 04:18:12 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get deployed
2021/12/17 04:18:17 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get deployed
2021/12/17 04:18:22 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get deployed
2021/12/17 04:18:27 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get deployed
2021/12/17 04:18:27 App: TestMultipleHandlers_bucket_op_on_update got deployed
2021/12/17 04:18:32 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get deployed
2021/12/17 04:18:37 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get deployed
2021/12/17 04:18:42 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get deployed
2021/12/17 04:18:42 App: TestMultipleHandlers_bucket_op_on_delete got deployed
2021/12/17 04:18:42 Starting bucket ops to source bucket
2021/12/17 04:18:52 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 04:18:52 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/17 04:18:52 Starting bucket ops to source bucket
2021/12/17 04:19:07 Waiting for dst bucket item count to get to: 10000 curr count: 6044
2021/12/17 04:19:12 Waiting for dst bucket item count to get to: 10000 curr count: 6044
2021/12/17 04:19:12 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/17 04:19:12 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 04:19:12 Function: TestMultipleHandlers_bucket_op_on_update update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/17 04:19:12 Function: TestMultipleHandlers_bucket_op_on_delete update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/17 04:19:13 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 04:19:18 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get un-deployed
2021/12/17 04:19:23 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get un-deployed
2021/12/17 04:19:28 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get un-deployed
2021/12/17 04:19:33 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get un-deployed
2021/12/17 04:19:33 App: TestMultipleHandlers_bucket_op_on_update got un-deployed
2021/12/17 04:19:33 No eventing-con process running
2021/12/17 04:19:33 Function: TestMultipleHandlers_bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMultipleHandlers_bucket_op_on_update deleting in the background"
}
2021/12/17 04:19:34 Context: Bucket flush, response code: 200 dump: 
2021/12/17 04:19:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 04:19:42 Context: Bucket flush, response code: 200 dump: 
2021/12/17 04:19:52 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/17 04:19:57 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/17 04:19:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 04:19:57 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 04:20:02 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get un-deployed
2021/12/17 04:20:07 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get un-deployed
2021/12/17 04:20:12 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get un-deployed
2021/12/17 04:20:17 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get un-deployed
2021/12/17 04:20:17 App: TestMultipleHandlers_bucket_op_on_delete got un-deployed
2021/12/17 04:20:17 No eventing-con process running
2021/12/17 04:20:17 Function: TestMultipleHandlers_bucket_op_on_delete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMultipleHandlers_bucket_op_on_delete deleting in the background"
}
2021/12/17 04:20:19 Context: Bucket flush, response code: 200 dump: 
2021/12/17 04:20:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 04:20:25 Context: Bucket flush, response code: 200 dump: 
2021/12/17 04:20:30 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestMultipleHandlers (188.02s)
=== RUN   TestPauseResumeLoopDefaultSettings
2021/12/17 04:20:40 Waiting for app: TestPauseResumeLoopDefaultSettings to get un-deployed
2021/12/17 04:20:40 App: TestPauseResumeLoopDefaultSettings got un-deployed
2021/12/17 04:20:40 No eventing-con process running
2021/12/17 04:20:40 Function: TestPauseResumeLoopDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestPauseResumeLoopDefaultSettings not found"
 }
}
2021/12/17 04:20:42 Context: Bucket flush, response code: 200 dump: 
2021/12/17 04:20:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 04:20:49 Context: Bucket flush, response code: 200 dump: 
2021/12/17 04:20:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 04:20:54 Deploying app: TestPauseResumeLoopDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/17 04:20:55 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestPauseResumeLoopDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/17 04:20:55 Starting bucket ops to source bucket
2021/12/17 04:21:03 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 04:21:08 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 04:21:13 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 04:21:18 Waiting for dst bucket item count to get to: 5000 curr count: 67
2021/12/17 04:21:23 Waiting for dst bucket item count to get to: 5000 curr count: 2567
2021/12/17 04:21:23 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/17 04:21:23 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 04:21:28 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to deployed
2021/12/17 04:21:28 Function: TestPauseResumeLoopDefaultSettings status changed to deployed
2021/12/17 04:21:28 Pausing app: bucket_op_on_update
2021/12/17 04:21:28 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/17 04:21:33 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to paused
2021/12/17 04:21:33 Function: TestPauseResumeLoopDefaultSettings status changed to paused
2021/12/17 04:21:33 Resuming app: bucket_op_on_update
2021/12/17 04:21:34 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/17 04:21:34 Starting bucket ops to source bucket
2021/12/17 04:21:42 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/17 04:21:47 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/17 04:21:52 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/17 04:21:52 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/17 04:21:52 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 04:21:57 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to deployed
2021/12/17 04:21:57 Function: TestPauseResumeLoopDefaultSettings status changed to deployed
2021/12/17 04:21:57 Pausing app: bucket_op_on_update
2021/12/17 04:21:58 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/17 04:22:03 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to paused
2021/12/17 04:22:03 Function: TestPauseResumeLoopDefaultSettings status changed to paused
2021/12/17 04:22:03 Resuming app: bucket_op_on_update
2021/12/17 04:22:04 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/17 04:22:04 Starting bucket ops to source bucket
2021/12/17 04:22:12 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/17 04:22:17 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/17 04:22:22 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/17 04:22:22 src & dst bucket item count matched up. src bucket count: 15000 dst bucket count: 15000
2021/12/17 04:22:22 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 04:22:27 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to deployed
2021/12/17 04:22:27 Function: TestPauseResumeLoopDefaultSettings status changed to deployed
2021/12/17 04:22:27 Pausing app: bucket_op_on_update
2021/12/17 04:22:27 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/17 04:22:32 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to paused
2021/12/17 04:22:32 Function: TestPauseResumeLoopDefaultSettings status changed to paused
2021/12/17 04:22:32 Resuming app: bucket_op_on_update
2021/12/17 04:22:33 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/17 04:22:33 Starting bucket ops to source bucket
2021/12/17 04:22:41 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/17 04:22:46 Waiting for dst bucket item count to get to: 20000 curr count: 15097
2021/12/17 04:22:51 Waiting for dst bucket item count to get to: 20000 curr count: 15097
2021/12/17 04:22:51 src & dst bucket item count matched up. src bucket count: 20000 dst bucket count: 20000
2021/12/17 04:22:51 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 04:22:56 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to deployed
2021/12/17 04:22:56 Function: TestPauseResumeLoopDefaultSettings status changed to deployed
2021/12/17 04:22:56 Pausing app: bucket_op_on_update
2021/12/17 04:22:56 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/17 04:23:01 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to paused
2021/12/17 04:23:01 Function: TestPauseResumeLoopDefaultSettings status changed to paused
2021/12/17 04:23:01 Resuming app: bucket_op_on_update
2021/12/17 04:23:02 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/17 04:23:02 Starting bucket ops to source bucket
2021/12/17 04:23:10 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/17 04:23:15 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/17 04:23:20 Waiting for dst bucket item count to get to: 25000 curr count: 20350
2021/12/17 04:23:25 Waiting for dst bucket item count to get to: 25000 curr count: 20350
2021/12/17 04:23:25 src & dst bucket item count matched up. src bucket count: 25000 dst bucket count: 25000
2021/12/17 04:23:25 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 04:23:30 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to deployed
2021/12/17 04:23:30 Function: TestPauseResumeLoopDefaultSettings status changed to deployed
2021/12/17 04:23:30 Pausing app: bucket_op_on_update
2021/12/17 04:23:31 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/17 04:23:36 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to paused
2021/12/17 04:23:36 Function: TestPauseResumeLoopDefaultSettings status changed to paused
2021/12/17 04:23:36 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 04:23:41 Waiting for app: TestPauseResumeLoopDefaultSettings to get un-deployed
2021/12/17 04:23:46 Waiting for app: TestPauseResumeLoopDefaultSettings to get un-deployed
2021/12/17 04:23:51 Waiting for app: TestPauseResumeLoopDefaultSettings to get un-deployed
2021/12/17 04:23:56 Waiting for app: TestPauseResumeLoopDefaultSettings to get un-deployed
2021/12/17 04:23:56 App: TestPauseResumeLoopDefaultSettings got un-deployed
2021/12/17 04:23:56 No eventing-con process running
2021/12/17 04:23:56 Function: TestPauseResumeLoopDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestPauseResumeLoopDefaultSettings deleting in the background"
}
2021/12/17 04:23:58 Context: Bucket flush, response code: 200 dump: 
2021/12/17 04:24:08 Waiting for dst bucket item count to get to: 0 curr count: 12500
2021/12/17 04:24:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 04:24:09 Context: Bucket flush, response code: 200 dump: 
2021/12/17 04:24:19 Waiting for dst bucket item count to get to: 0 curr count: 12500
2021/12/17 04:24:24 Waiting for dst bucket item count to get to: 0 curr count: 12500
2021/12/17 04:24:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestPauseResumeLoopDefaultSettings (233.94s)
=== RUN   TestPauseResumeLoopNonDefaultSettings
2021/12/17 04:24:34 Waiting for app: TestPauseResumeLoopNonDefaultSettings to get un-deployed
2021/12/17 04:24:34 App: TestPauseResumeLoopNonDefaultSettings got un-deployed
2021/12/17 04:24:34 No eventing-con process running
2021/12/17 04:24:34 Function: TestPauseResumeLoopNonDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestPauseResumeLoopNonDefaultSettings not found"
 }
}
2021/12/17 04:24:36 Context: Bucket flush, response code: 200 dump: 
2021/12/17 04:24:41 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 04:24:43 Context: Bucket flush, response code: 200 dump: 
2021/12/17 04:24:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 04:24:48 Deploying app: TestPauseResumeLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/17 04:24:49 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestPauseResumeLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/17 04:24:49 Starting bucket ops to source bucket
2021/12/17 04:24:57 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 04:25:02 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 04:25:07 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 04:25:12 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 04:25:17 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 04:25:22 Waiting for dst bucket item count to get to: 5000 curr count: 988
2021/12/17 04:25:27 Waiting for dst bucket item count to get to: 5000 curr count: 3111
2021/12/17 04:25:32 Waiting for dst bucket item count to get to: 5000 curr count: 4313
2021/12/17 04:25:37 Waiting for dst bucket item count to get to: 5000 curr count: 4690
2021/12/17 04:25:37 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/17 04:25:37 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 04:25:42 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to deployed
2021/12/17 04:25:42 Function: TestPauseResumeLoopNonDefaultSettings status changed to deployed
2021/12/17 04:25:42 Pausing app: bucket_op_with_timer
2021/12/17 04:25:42 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/17 04:25:47 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to paused
2021/12/17 04:25:47 Function: TestPauseResumeLoopNonDefaultSettings status changed to paused
2021/12/17 04:25:47 Resuming app: bucket_op_with_timer
2021/12/17 04:25:49 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/17 04:25:49 Starting bucket ops to source bucket
2021/12/17 04:25:57 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/17 04:26:02 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/17 04:26:07 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/17 04:26:12 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/17 04:26:17 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/17 04:26:22 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/17 04:26:27 Waiting for dst bucket item count to get to: 10000 curr count: 7500
2021/12/17 04:26:27 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/17 04:26:27 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 04:26:32 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to deployed
2021/12/17 04:26:32 Function: TestPauseResumeLoopNonDefaultSettings status changed to deployed
2021/12/17 04:26:32 Pausing app: bucket_op_with_timer
2021/12/17 04:26:32 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/17 04:26:37 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to paused
2021/12/17 04:26:37 Function: TestPauseResumeLoopNonDefaultSettings status changed to paused
2021/12/17 04:26:37 Resuming app: bucket_op_with_timer
2021/12/17 04:26:38 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/17 04:26:38 Starting bucket ops to source bucket
2021/12/17 04:26:46 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/17 04:26:51 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/17 04:26:56 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/17 04:27:01 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/17 04:27:06 Waiting for dst bucket item count to get to: 15000 curr count: 10278
2021/12/17 04:27:11 Waiting for dst bucket item count to get to: 15000 curr count: 10278
2021/12/17 04:27:11 src & dst bucket item count matched up. src bucket count: 15000 dst bucket count: 15000
2021/12/17 04:27:11 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 04:27:16 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to deployed
2021/12/17 04:27:16 Function: TestPauseResumeLoopNonDefaultSettings status changed to deployed
2021/12/17 04:27:16 Pausing app: bucket_op_with_timer
2021/12/17 04:27:16 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/17 04:27:21 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to paused
2021/12/17 04:27:21 Function: TestPauseResumeLoopNonDefaultSettings status changed to paused
2021/12/17 04:27:21 Resuming app: bucket_op_with_timer
2021/12/17 04:27:22 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/17 04:27:22 Starting bucket ops to source bucket
2021/12/17 04:27:30 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/17 04:27:35 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/17 04:27:40 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/17 04:27:45 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/17 04:27:50 Waiting for dst bucket item count to get to: 20000 curr count: 15845
2021/12/17 04:27:55 Waiting for dst bucket item count to get to: 20000 curr count: 15845
2021/12/17 04:27:55 src & dst bucket item count matched up. src bucket count: 20000 dst bucket count: 20000
2021/12/17 04:27:55 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 04:28:00 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to deployed
2021/12/17 04:28:00 Function: TestPauseResumeLoopNonDefaultSettings status changed to deployed
2021/12/17 04:28:00 Pausing app: bucket_op_with_timer
2021/12/17 04:28:01 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/17 04:28:06 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to paused
2021/12/17 04:28:06 Function: TestPauseResumeLoopNonDefaultSettings status changed to paused
2021/12/17 04:28:06 Resuming app: bucket_op_with_timer
2021/12/17 04:28:07 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/17 04:28:07 Starting bucket ops to source bucket
2021/12/17 04:28:15 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/17 04:28:20 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/17 04:28:25 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/17 04:28:30 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/17 04:28:35 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/17 04:28:40 Waiting for dst bucket item count to get to: 25000 curr count: 22086
2021/12/17 04:28:45 Waiting for dst bucket item count to get to: 25000 curr count: 22086
2021/12/17 04:28:45 src & dst bucket item count matched up. src bucket count: 25000 dst bucket count: 25000
2021/12/17 04:28:45 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 04:28:50 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to deployed
2021/12/17 04:28:50 Function: TestPauseResumeLoopNonDefaultSettings status changed to deployed
2021/12/17 04:28:50 Pausing app: bucket_op_with_timer
2021/12/17 04:28:50 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/17 04:28:55 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to paused
2021/12/17 04:28:55 Function: TestPauseResumeLoopNonDefaultSettings status changed to paused
2021/12/17 04:28:55 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 04:29:00 Waiting for app: TestPauseResumeLoopNonDefaultSettings to get un-deployed
2021/12/17 04:29:05 Waiting for app: TestPauseResumeLoopNonDefaultSettings to get un-deployed
2021/12/17 04:29:10 Waiting for app: TestPauseResumeLoopNonDefaultSettings to get un-deployed
2021/12/17 04:29:15 Waiting for app: TestPauseResumeLoopNonDefaultSettings to get un-deployed
2021/12/17 04:29:15 App: TestPauseResumeLoopNonDefaultSettings got un-deployed
2021/12/17 04:29:15 No eventing-con process running
2021/12/17 04:29:15 Function: TestPauseResumeLoopNonDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestPauseResumeLoopNonDefaultSettings deleting in the background"
}
2021/12/17 04:29:17 Context: Bucket flush, response code: 200 dump: 
2021/12/17 04:29:27 Waiting for dst bucket item count to get to: 0 curr count: 25000
2021/12/17 04:29:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 04:29:29 Context: Bucket flush, response code: 200 dump: 
2021/12/17 04:29:39 Waiting for dst bucket item count to get to: 0 curr count: 21867
2021/12/17 04:29:44 Waiting for dst bucket item count to get to: 0 curr count: 9367
2021/12/17 04:29:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestPauseResumeLoopNonDefaultSettings (319.20s)
=== RUN   TestPauseAndResumeWithWorkerCountChange
2021/12/17 04:29:54 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get un-deployed
2021/12/17 04:29:54 App: TestPauseAndResumeWithWorkerCountChange got un-deployed
2021/12/17 04:29:54 No eventing-con process running
2021/12/17 04:29:54 Function: TestPauseAndResumeWithWorkerCountChange delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestPauseAndResumeWithWorkerCountChange not found"
 }
}
2021/12/17 04:29:55 Context: Bucket flush, response code: 200 dump: 
2021/12/17 04:30:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 04:30:02 Context: Bucket flush, response code: 200 dump: 
2021/12/17 04:30:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 04:30:07 Deploying app: TestPauseAndResumeWithWorkerCountChange. Item count src bucket: 0 dst bucket: 0
2021/12/17 04:30:08 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestPauseAndResumeWithWorkerCountChange' in metakv",
  "warnings": null
 }
}
2021/12/17 04:30:13 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get deployed
2021/12/17 04:30:18 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get deployed
2021/12/17 04:30:23 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get deployed
2021/12/17 04:30:28 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get deployed
2021/12/17 04:30:28 App: TestPauseAndResumeWithWorkerCountChange got deployed
2021/12/17 04:30:28 Starting bucket ops to source bucket
2021/12/17 04:30:38 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 04:30:43 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 04:30:48 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 04:30:53 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 04:30:58 Waiting for dst bucket item count to get to: 5000 curr count: 1841
2021/12/17 04:31:03 Waiting for dst bucket item count to get to: 5000 curr count: 4330
2021/12/17 04:31:08 Waiting for dst bucket item count to get to: 5000 curr count: 4989
2021/12/17 04:31:08 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/17 04:31:08 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 04:31:08 Pausing app: bucket_op_with_timer
2021/12/17 04:31:08 Function: TestPauseAndResumeWithWorkerCountChange update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/17 04:31:13 Waiting for function: TestPauseAndResumeWithWorkerCountChange status to change to paused
2021/12/17 04:31:13 Function: TestPauseAndResumeWithWorkerCountChange status changed to paused
2021/12/17 04:31:13 Resuming app: bucket_op_with_timer
2021/12/17 04:31:14 Function: TestPauseAndResumeWithWorkerCountChange update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:6] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/17 04:31:14 Starting bucket ops to source bucket
2021/12/17 04:31:26 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/17 04:31:31 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/17 04:31:36 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/17 04:31:41 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/17 04:31:46 Waiting for dst bucket item count to get to: 10000 curr count: 6248
2021/12/17 04:31:51 Waiting for dst bucket item count to get to: 10000 curr count: 8727
2021/12/17 04:31:56 Waiting for dst bucket item count to get to: 10000 curr count: 9979
2021/12/17 04:31:56 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/17 04:31:56 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 04:32:01 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get un-deployed
2021/12/17 04:32:06 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get un-deployed
2021/12/17 04:32:11 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get un-deployed
2021/12/17 04:32:16 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get un-deployed
2021/12/17 04:32:16 App: TestPauseAndResumeWithWorkerCountChange got un-deployed
2021/12/17 04:32:16 No eventing-con process running
2021/12/17 04:32:16 Function: TestPauseAndResumeWithWorkerCountChange delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestPauseAndResumeWithWorkerCountChange deleting in the background"
}
2021/12/17 04:32:18 Context: Bucket flush, response code: 200 dump: 
2021/12/17 04:32:28 Waiting for dst bucket item count to get to: 0 curr count: 7070
2021/12/17 04:32:33 Waiting for dst bucket item count to get to: 0 curr count: 2070
2021/12/17 04:32:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 04:32:35 Context: Bucket flush, response code: 200 dump: 
2021/12/17 04:32:45 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/17 04:32:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestPauseAndResumeWithWorkerCountChange (181.34s)
=== RUN   TestPauseResumeWithEventingReb
2021/12/17 04:32:55 Waiting for app: TestPauseResumeWithEventingReb to get un-deployed
2021/12/17 04:32:55 App: TestPauseResumeWithEventingReb got un-deployed
2021/12/17 04:32:55 No eventing-con process running
2021/12/17 04:32:55 Function: TestPauseResumeWithEventingReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestPauseResumeWithEventingReb not found"
 }
}
2021/12/17 04:32:57 Context: Bucket flush, response code: 200 dump: 
2021/12/17 04:33:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 04:33:03 Context: Bucket flush, response code: 200 dump: 
2021/12/17 04:33:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 04:33:08 Deploying app: TestPauseResumeWithEventingReb. Item count src bucket: 0 dst bucket: 0
2021/12/17 04:33:10 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestPauseResumeWithEventingReb' in metakv",
  "warnings": null
 }
}
2021/12/17 04:33:15 Waiting for app: TestPauseResumeWithEventingReb to get deployed
2021/12/17 04:33:20 Waiting for app: TestPauseResumeWithEventingReb to get deployed
2021/12/17 04:33:25 Waiting for app: TestPauseResumeWithEventingReb to get deployed
2021/12/17 04:33:30 Waiting for app: TestPauseResumeWithEventingReb to get deployed
2021/12/17 04:33:30 App: TestPauseResumeWithEventingReb got deployed
2021/12/17 04:33:30 Starting bucket ops to source bucket
2021/12/17 04:33:39 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 04:33:44 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 04:33:49 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 04:33:54 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 04:33:59 Waiting for dst bucket item count to get to: 5000 curr count: 2261
2021/12/17 04:34:04 Waiting for dst bucket item count to get to: 5000 curr count: 4520
2021/12/17 04:34:09 Waiting for dst bucket item count to get to: 5000 curr count: 4759
2021/12/17 04:34:09 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/17 04:34:09 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 04:34:09 Pausing app: bucket_op_with_timer
2021/12/17 04:34:09 Function: TestPauseResumeWithEventingReb update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/17 04:34:14 Waiting for function: TestPauseResumeWithEventingReb status to change to paused
2021/12/17 04:34:14 Function: TestPauseResumeWithEventingReb status changed to paused
2021/12/17 04:34:14 Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/17 04:34:14 Removing node(s): [] from the cluster
2021/12/17 04:34:19 Rebalance progress: 29.16666666666666
2021/12/17 04:34:24 Rebalance progress: 29.16666666666666
2021/12/17 04:34:29 Rebalance progress: 100
2021/12/17 04:34:29 Resuming app: bucket_op_with_timer
2021/12/17 04:34:31 Function: TestPauseResumeWithEventingReb update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:6] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/17 04:34:31 Starting bucket ops to source bucket
2021/12/17 04:34:42 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/17 04:34:47 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/17 04:34:52 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/17 04:34:57 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/17 04:35:02 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/17 04:35:02 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/17 04:35:02 Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/17 04:35:07 Rebalance progress: 29.16666666666666
2021/12/17 04:35:12 Rebalance progress: 29.16666666666666
2021/12/17 04:35:17 Rebalance progress: 100

VB distribution across eventing nodes and their workers::

2021/12/17 04:35:23 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:45} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/17 04:35:38 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:45} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/17 04:35:53 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:45} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/17 04:36:08 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:45} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/17 04:36:23 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:45} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/17 04:36:38 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:45} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/17 04:36:53 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:45} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/17 04:37:08 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:45} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/17 04:37:23 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:45} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/17 04:37:38 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:45} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/17 04:37:53 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:45} {Name:#primary Bucket:hello-world Completion:100}]

DCP Stream statuses:
	stream status: running
	len: 1024
	vb list dump: [0-1023]

vbucket curr owner:
Producer node: 127.0.0.1:9302	Node UUID: ac9391d01b259f92e8c0183354b22941
	workerID: worker_TestPauseResumeWithEventingReb_0
	len: 57
	v: [0-56]
	workerID: worker_TestPauseResumeWithEventingReb_1
	len: 57
	v: [57-113]
	workerID: worker_TestPauseResumeWithEventingReb_2
	len: 57
	v: [114-170]
	workerID: worker_TestPauseResumeWithEventingReb_3
	len: 57
	v: [171-227]
	workerID: worker_TestPauseResumeWithEventingReb_4
	len: 57
	v: [228-284]
	workerID: worker_TestPauseResumeWithEventingReb_5
	len: 57
	v: [285-341]
Producer node: 172.31.2.134:9300	Node UUID: dc4c94f59b06fbc7ba2da352eea789a6
	workerID: worker_TestPauseResumeWithEventingReb_2
	len: 57
	v: [797-853]
	workerID: worker_TestPauseResumeWithEventingReb_3
	len: 57
	v: [854-910]
	workerID: worker_TestPauseResumeWithEventingReb_4
	len: 57
	v: [911-967]
	workerID: worker_TestPauseResumeWithEventingReb_5
	len: 56
	v: [968-1023]
	workerID: worker_TestPauseResumeWithEventingReb_0
	len: 57
	v: [683-739]
	workerID: worker_TestPauseResumeWithEventingReb_1
	len: 57
	v: [740-796]
Producer node: 127.0.0.1:9303	Node UUID: 4225b86502e5f2f89c3dbd8b33d0b402
	workerID: worker_TestPauseResumeWithEventingReb_0
	len: 57
	v: [342-398]
	workerID: worker_TestPauseResumeWithEventingReb_1
	len: 57
	v: [399-455]
	workerID: worker_TestPauseResumeWithEventingReb_2
	len: 57
	v: [456-512]
	workerID: worker_TestPauseResumeWithEventingReb_3
	len: 57
	v: [513-569]
	workerID: worker_TestPauseResumeWithEventingReb_4
	len: 57
	v: [570-626]
	workerID: worker_TestPauseResumeWithEventingReb_5
	len: 56
	v: [627-682]


2021/12/17 04:38:08 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 04:38:08 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 04:38:13 clusterSetup.go:397: Waiting for app: TestPauseResumeWithEventingReb to get un-deployed
2021/12/17 04:38:18 clusterSetup.go:397: Waiting for app: TestPauseResumeWithEventingReb to get un-deployed
2021/12/17 04:38:23 clusterSetup.go:397: Waiting for app: TestPauseResumeWithEventingReb to get un-deployed
2021/12/17 04:38:28 clusterSetup.go:397: Waiting for app: TestPauseResumeWithEventingReb to get un-deployed
2021/12/17 04:38:28 clusterSetup.go:405: App: TestPauseResumeWithEventingReb got un-deployed
2021/12/17 04:38:28 eventingSetup.go:46: No eventing-con process running
2021/12/17 04:38:28 eventingSetup.go:418: Function: TestPauseResumeWithEventingReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestPauseResumeWithEventingReb deleting in the background"
}
2021/12/17 04:38:30 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/17 04:38:40 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/17 04:38:45 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/17 04:38:45 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 04:38:47 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/17 04:38:57 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/17 04:38:57 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestPauseResumeWithEventingReb (371.65s)
=== RUN   TestBucketDeleteAfterPause
2021/12/17 04:39:07 clusterSetup.go:397: Waiting for app: TestBucketDeleteAfterPause to get un-deployed
2021/12/17 04:39:07 clusterSetup.go:405: App: TestBucketDeleteAfterPause got un-deployed
2021/12/17 04:39:07 eventingSetup.go:46: No eventing-con process running
2021/12/17 04:39:07 eventingSetup.go:418: Function: TestBucketDeleteAfterPause delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestBucketDeleteAfterPause not found"
 }
}
2021/12/17 04:39:08 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/17 04:39:13 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 04:39:15 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/17 04:39:20 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 04:39:20 eventingSetup.go:125: Deploying app: TestBucketDeleteAfterPause. Item count src bucket: 0 dst bucket: 0
2021/12/17 04:39:21 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestBucketDeleteAfterPause' in metakv",
  "warnings": null
 }
}
2021/12/17 04:39:26 clusterSetup.go:346: Waiting for app: TestBucketDeleteAfterPause to get deployed
2021/12/17 04:39:31 clusterSetup.go:346: Waiting for app: TestBucketDeleteAfterPause to get deployed
2021/12/17 04:39:36 clusterSetup.go:346: Waiting for app: TestBucketDeleteAfterPause to get deployed
2021/12/17 04:39:41 clusterSetup.go:346: Waiting for app: TestBucketDeleteAfterPause to get deployed
2021/12/17 04:39:41 clusterSetup.go:354: App: TestBucketDeleteAfterPause got deployed
2021/12/17 04:39:41 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/17 04:39:51 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 04:39:56 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 2500
2021/12/17 04:39:56 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/17 04:39:56 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 04:39:56 handler_test.go:712: Pausing app: bucket_op_on_update
2021/12/17 04:39:56 eventingSetup.go:270: Function: TestBucketDeleteAfterPause update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/17 04:40:01 eventingSetup.go:719: Waiting for function: TestBucketDeleteAfterPause status to change to paused
2021/12/17 04:40:01 eventingSetup.go:727: Function: TestBucketDeleteAfterPause status changed to paused
2021/12/17 04:40:01 handler_test.go:716: Deleting source bucket: default
2021/12/17 04:40:07 eventingSetup.go:719: Waiting for function: TestBucketDeleteAfterPause status to change to undeployed
2021/12/17 04:40:08 eventingSetup.go:727: Function: TestBucketDeleteAfterPause status changed to undeployed
2021/12/17 04:40:13 clusterSetup.go:397: Waiting for app: TestBucketDeleteAfterPause to get un-deployed
2021/12/17 04:40:13 clusterSetup.go:405: App: TestBucketDeleteAfterPause got un-deployed
2021/12/17 04:40:13 eventingSetup.go:46: No eventing-con process running
2021/12/17 04:40:13 eventingSetup.go:418: Function: TestBucketDeleteAfterPause delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestBucketDeleteAfterPause deleting in the background"
}
2021/12/17 04:40:14 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/17 04:40:24 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/17 04:40:24 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 04:40:26 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/17 04:40:36 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/17 04:40:36 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestBucketDeleteAfterPause (99.24s)
=== RUN   TestChangeFnCodeBetweenPauseResume
2021/12/17 04:40:46 clusterSetup.go:397: Waiting for app: TestChangeFnCodeBetweenPauseResume to get un-deployed
2021/12/17 04:40:46 clusterSetup.go:405: App: TestChangeFnCodeBetweenPauseResume got un-deployed
2021/12/17 04:40:46 eventingSetup.go:46: No eventing-con process running
2021/12/17 04:40:46 eventingSetup.go:418: Function: TestChangeFnCodeBetweenPauseResume delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestChangeFnCodeBetweenPauseResume not found"
 }
}
2021/12/17 04:40:47 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/17 04:40:52 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 04:40:54 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/17 04:40:59 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 04:40:59 eventingSetup.go:125: Deploying app: TestChangeFnCodeBetweenPauseResume. Item count src bucket: 0 dst bucket: 0
2021/12/17 04:41:00 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestChangeFnCodeBetweenPauseResume' in metakv",
  "warnings": null
 }
}
2021/12/17 04:41:05 clusterSetup.go:346: Waiting for app: TestChangeFnCodeBetweenPauseResume to get deployed
2021/12/17 04:41:10 clusterSetup.go:346: Waiting for app: TestChangeFnCodeBetweenPauseResume to get deployed
2021/12/17 04:41:15 clusterSetup.go:346: Waiting for app: TestChangeFnCodeBetweenPauseResume to get deployed
2021/12/17 04:41:20 clusterSetup.go:346: Waiting for app: TestChangeFnCodeBetweenPauseResume to get deployed
2021/12/17 04:41:20 clusterSetup.go:354: App: TestChangeFnCodeBetweenPauseResume got deployed
2021/12/17 04:41:20 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/17 04:41:55 handler_test.go:739: Pausing function: TestChangeFnCodeBetweenPauseResume
2021/12/17 04:41:55 eventingSetup.go:270: Function: TestChangeFnCodeBetweenPauseResume update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/17 04:42:00 eventingSetup.go:719: Waiting for function: TestChangeFnCodeBetweenPauseResume status to change to paused
2021/12/17 04:42:00 eventingSetup.go:727: Function: TestChangeFnCodeBetweenPauseResume status changed to paused
2021/12/17 04:45:00 handler_test.go:748: Resuming function: TestChangeFnCodeBetweenPauseResume
2021/12/17 04:45:00 eventingSetup.go:125: Deploying app: TestChangeFnCodeBetweenPauseResume. Item count src bucket: 5000 dst bucket: 0
2021/12/17 04:45:02 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestChangeFnCodeBetweenPauseResume' in metakv",
  "warnings": null
 }
}
2021/12/17 04:45:07 eventingSetup.go:795: Waiting for function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter to get to 5000
2021/12/17 04:45:07 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 04:45:07 eventingSetup.go:799: Function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter got to 0 expected 5000
2021/12/17 04:45:12 eventingSetup.go:795: Waiting for function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter to get to 5000
2021/12/17 04:45:12 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 04:45:12 eventingSetup.go:799: Function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter got to 2526 expected 5000
2021/12/17 04:45:17 eventingSetup.go:795: Waiting for function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter to get to 5000
2021/12/17 04:45:17 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 04:45:17 eventingSetup.go:799: Function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter got to 3183 expected 5000
2021/12/17 04:45:22 eventingSetup.go:795: Waiting for function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter to get to 5000
2021/12/17 04:45:22 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 04:45:22 eventingSetup.go:804: Function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter got to 5000
2021/12/17 04:45:22 handler_test.go:753: Undeploying function: TestChangeFnCodeBetweenPauseResume
2021/12/17 04:45:22 eventingSetup.go:270: Function: TestChangeFnCodeBetweenPauseResume update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/17 04:45:32 clusterSetup.go:397: Waiting for app: TestChangeFnCodeBetweenPauseResume to get un-deployed
2021/12/17 04:45:37 clusterSetup.go:397: Waiting for app: TestChangeFnCodeBetweenPauseResume to get un-deployed
2021/12/17 04:45:42 clusterSetup.go:397: Waiting for app: TestChangeFnCodeBetweenPauseResume to get un-deployed
2021/12/17 04:45:42 clusterSetup.go:405: App: TestChangeFnCodeBetweenPauseResume got un-deployed
2021/12/17 04:45:42 eventingSetup.go:46: No eventing-con process running
2021/12/17 04:45:42 eventingSetup.go:418: Function: TestChangeFnCodeBetweenPauseResume delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestChangeFnCodeBetweenPauseResume deleting in the background"
}
2021/12/17 04:45:43 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/17 04:45:53 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/17 04:45:58 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/17 04:45:58 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 04:46:00 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/17 04:46:05 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestChangeFnCodeBetweenPauseResume (329.16s)
=== RUN   TestCommentUnCommentOnDelete
2021/12/17 04:46:15 clusterSetup.go:397: Waiting for app: TestCommentUnCommentOnDelete to get un-deployed
2021/12/17 04:46:15 clusterSetup.go:405: App: TestCommentUnCommentOnDelete got un-deployed
2021/12/17 04:46:15 eventingSetup.go:46: No eventing-con process running
2021/12/17 04:46:15 eventingSetup.go:418: Function: TestCommentUnCommentOnDelete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestCommentUnCommentOnDelete not found"
 }
}
2021/12/17 04:46:16 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/17 04:46:21 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 04:46:23 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/17 04:46:28 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 04:46:28 eventingSetup.go:125: Deploying app: TestCommentUnCommentOnDelete. Item count src bucket: 0 dst bucket: 0
2021/12/17 04:46:29 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCommentUnCommentOnDelete' in metakv",
  "warnings": null
 }
}
2021/12/17 04:46:34 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/17 04:46:39 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/17 04:46:44 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/17 04:46:49 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/17 04:46:49 clusterSetup.go:354: App: TestCommentUnCommentOnDelete got deployed
2021/12/17 04:46:49 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/17 04:46:58 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 04:47:03 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 04:47:08 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 2500
2021/12/17 04:47:08 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/17 04:47:08 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 04:47:08 handler_test.go:780: Undeploying app: TestCommentUnCommentOnDelete
2021/12/17 04:47:09 eventingSetup.go:270: Function: TestCommentUnCommentOnDelete update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/17 04:47:39 eventingSetup.go:125: Deploying app: TestCommentUnCommentOnDelete. Item count src bucket: 5000 dst bucket: 5000
2021/12/17 04:47:40 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCommentUnCommentOnDelete' in metakv",
  "warnings": null
 }
}
2021/12/17 04:47:45 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/17 04:47:50 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/17 04:47:55 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/17 04:48:00 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/17 04:48:00 clusterSetup.go:354: App: TestCommentUnCommentOnDelete got deployed
2021/12/17 04:48:00 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/17 04:48:13 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/17 04:48:18 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 211
2021/12/17 04:48:23 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 211
2021/12/17 04:48:23 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 04:48:23 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 04:48:23 handler_test.go:799: Undeploying app: TestCommentUnCommentOnDelete
2021/12/17 04:48:23 eventingSetup.go:270: Function: TestCommentUnCommentOnDelete update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/17 04:48:33 clusterSetup.go:397: Waiting for app: TestCommentUnCommentOnDelete to get un-deployed
2021/12/17 04:48:38 clusterSetup.go:397: Waiting for app: TestCommentUnCommentOnDelete to get un-deployed
2021/12/17 04:48:43 clusterSetup.go:397: Waiting for app: TestCommentUnCommentOnDelete to get un-deployed
2021/12/17 04:48:43 clusterSetup.go:405: App: TestCommentUnCommentOnDelete got un-deployed
2021/12/17 04:48:43 eventingSetup.go:46: No eventing-con process running
2021/12/17 04:48:43 eventingSetup.go:418: Function: TestCommentUnCommentOnDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCommentUnCommentOnDelete deleting in the background"
}
2021/12/17 04:48:44 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/17 04:48:49 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 04:48:51 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/17 04:48:56 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCommentUnCommentOnDelete (170.77s)
=== RUN   TestCPPWorkerCleanup
2021/12/17 04:49:06 clusterSetup.go:397: Waiting for app: TestCPPWorkerCleanup to get un-deployed
2021/12/17 04:49:06 clusterSetup.go:405: App: TestCPPWorkerCleanup got un-deployed
2021/12/17 04:49:06 eventingSetup.go:46: No eventing-con process running
2021/12/17 04:49:06 eventingSetup.go:418: Function: TestCPPWorkerCleanup delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestCPPWorkerCleanup not found"
 }
}
2021/12/17 04:49:07 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/17 04:49:12 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 04:49:14 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/17 04:49:19 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 04:49:19 eventingSetup.go:125: Deploying app: TestCPPWorkerCleanup. Item count src bucket: 0 dst bucket: 0
2021/12/17 04:49:20 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCPPWorkerCleanup' in metakv",
  "warnings": null
 }
}
2021/12/17 04:49:20 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/17 04:49:27 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 04:49:32 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 04:49:37 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 04:49:42 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 04:49:47 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 2368
2021/12/17 04:49:52 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 4868
2021/12/17 04:49:52 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/17 04:49:52 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 04:49:52 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 04:49:57 clusterSetup.go:397: Waiting for app: TestCPPWorkerCleanup to get un-deployed
2021/12/17 04:50:02 clusterSetup.go:397: Waiting for app: TestCPPWorkerCleanup to get un-deployed
2021/12/17 04:50:07 clusterSetup.go:397: Waiting for app: TestCPPWorkerCleanup to get un-deployed
2021/12/17 04:50:12 clusterSetup.go:397: Waiting for app: TestCPPWorkerCleanup to get un-deployed
2021/12/17 04:50:12 clusterSetup.go:405: App: TestCPPWorkerCleanup got un-deployed
2021/12/17 04:50:12 eventingSetup.go:46: No eventing-con process running
2021/12/17 04:50:12 eventingSetup.go:418: Function: TestCPPWorkerCleanup delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCPPWorkerCleanup deleting in the background"
}
2021/12/17 04:50:14 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/17 04:50:24 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/17 04:50:29 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/17 04:50:29 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 04:50:30 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/17 04:50:40 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4071
2021/12/17 04:50:45 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4071
2021/12/17 04:50:45 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCPPWorkerCleanup (139.76s)
=== RUN   TestWithUserXattrs
2021/12/17 04:51:25 clusterSetup.go:397: Waiting for app: TestWithUserXattrs to get un-deployed
2021/12/17 04:51:25 clusterSetup.go:405: App: TestWithUserXattrs got un-deployed
2021/12/17 04:51:26 eventingSetup.go:46: No eventing-con process running
2021/12/17 04:51:26 eventingSetup.go:418: Function: TestWithUserXattrs delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestWithUserXattrs not found"
 }
}
2021/12/17 04:51:27 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/17 04:51:32 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 04:51:34 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/17 04:51:39 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 04:51:39 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/17 04:51:41 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/17 04:51:43 eventingSetup.go:125: Deploying app: TestWithUserXattrs. Item count src bucket: 0 dst bucket: 0
2021/12/17 04:51:44 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestWithUserXattrs' in metakv",
  "warnings": null
 }
}
2021/12/17 04:51:49 clusterSetup.go:346: Waiting for app: TestWithUserXattrs to get deployed
2021/12/17 04:51:54 clusterSetup.go:346: Waiting for app: TestWithUserXattrs to get deployed
2021/12/17 04:51:59 clusterSetup.go:346: Waiting for app: TestWithUserXattrs to get deployed
2021/12/17 04:52:04 clusterSetup.go:346: Waiting for app: TestWithUserXattrs to get deployed
2021/12/17 04:52:04 clusterSetup.go:354: App: TestWithUserXattrs got deployed
2021/12/17 04:52:04 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/17 04:52:13 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 04:52:18 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 1889
2021/12/17 04:52:23 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 4389
2021/12/17 04:52:23 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/17 04:52:23 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 04:52:23 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 04:52:28 clusterSetup.go:397: Waiting for app: TestWithUserXattrs to get un-deployed
2021/12/17 04:52:33 clusterSetup.go:397: Waiting for app: TestWithUserXattrs to get un-deployed
2021/12/17 04:52:38 clusterSetup.go:397: Waiting for app: TestWithUserXattrs to get un-deployed
2021/12/17 04:52:43 clusterSetup.go:397: Waiting for app: TestWithUserXattrs to get un-deployed
2021/12/17 04:52:43 clusterSetup.go:405: App: TestWithUserXattrs got un-deployed
2021/12/17 04:52:43 eventingSetup.go:46: No eventing-con process running
2021/12/17 04:52:43 eventingSetup.go:418: Function: TestWithUserXattrs delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestWithUserXattrs deleting in the background"
}
2021/12/17 04:52:45 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/17 04:52:55 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/17 04:52:55 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 04:52:57 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/17 04:53:07 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/17 04:53:07 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestWithUserXattrs (111.05s)
=== RUN   TestEventProcessingPostBucketFlush
2021/12/17 04:53:17 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/17 04:53:17 clusterSetup.go:405: App: TestEventProcessingPostBucketFlush got un-deployed
2021/12/17 04:53:17 eventingSetup.go:46: No eventing-con process running
2021/12/17 04:53:17 eventingSetup.go:418: Function: TestEventProcessingPostBucketFlush delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventProcessingPostBucketFlush not found"
 }
}
2021/12/17 04:53:18 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/17 04:53:23 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 04:53:25 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/17 04:53:30 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 04:53:30 eventingSetup.go:125: Deploying app: TestEventProcessingPostBucketFlush. Item count src bucket: 0 dst bucket: 0
2021/12/17 04:53:31 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventProcessingPostBucketFlush' in metakv",
  "warnings": null
 }
}
2021/12/17 04:53:36 clusterSetup.go:346: Waiting for app: TestEventProcessingPostBucketFlush to get deployed
2021/12/17 04:53:41 clusterSetup.go:346: Waiting for app: TestEventProcessingPostBucketFlush to get deployed
2021/12/17 04:53:46 clusterSetup.go:346: Waiting for app: TestEventProcessingPostBucketFlush to get deployed
2021/12/17 04:53:51 clusterSetup.go:346: Waiting for app: TestEventProcessingPostBucketFlush to get deployed
2021/12/17 04:53:51 clusterSetup.go:354: App: TestEventProcessingPostBucketFlush got deployed
2021/12/17 04:53:51 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/17 04:54:00 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 04:54:05 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 04:54:05 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/17 04:54:05 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 04:54:09 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/17 04:54:19 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3026
2021/12/17 04:54:24 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 526
2021/12/17 04:54:24 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 04:54:26 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/17 04:54:36 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/17 04:54:36 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 04:54:36 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/17 04:55:01 eventingSetup.go:441: Waiting for dst bucket item count to get to: 25000 curr count: 15033
2021/12/17 04:55:06 eventingSetup.go:441: Waiting for dst bucket item count to get to: 25000 curr count: 15033
2021/12/17 04:55:06 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 25000 dst bucket count: 25000
2021/12/17 04:55:06 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 04:55:07 eventingSetup.go:270: Function: TestEventProcessingPostBucketFlush update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/17 04:55:12 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/17 04:55:17 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/17 04:55:22 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/17 04:55:27 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/17 04:55:27 clusterSetup.go:405: App: TestEventProcessingPostBucketFlush got un-deployed
2021/12/17 04:55:27 eventingSetup.go:46: No eventing-con process running
2021/12/17 04:55:32 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/17 04:55:32 clusterSetup.go:405: App: TestEventProcessingPostBucketFlush got un-deployed
2021/12/17 04:55:32 eventingSetup.go:46: No eventing-con process running
2021/12/17 04:55:32 eventingSetup.go:418: Function: TestEventProcessingPostBucketFlush delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventProcessingPostBucketFlush deleting in the background"
}
2021/12/17 04:55:33 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/17 04:55:43 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 25000
2021/12/17 04:55:48 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12500
2021/12/17 04:55:48 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 04:55:50 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/17 04:56:00 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12526
2021/12/17 04:56:05 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12526
2021/12/17 04:56:05 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventProcessingPostBucketFlush (178.24s)
=== RUN   TestMetaBucketDelete
2021/12/17 04:56:15 clusterSetup.go:397: Waiting for app: TestMetaBucketDelete to get un-deployed
2021/12/17 04:56:15 clusterSetup.go:405: App: TestMetaBucketDelete got un-deployed
2021/12/17 04:56:15 eventingSetup.go:46: No eventing-con process running
2021/12/17 04:56:15 eventingSetup.go:418: Function: TestMetaBucketDelete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMetaBucketDelete not found"
 }
}
2021/12/17 04:56:16 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/17 04:56:21 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 04:56:23 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/17 04:56:28 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 04:56:28 eventingSetup.go:125: Deploying app: TestMetaBucketDelete. Item count src bucket: 0 dst bucket: 0
2021/12/17 04:56:29 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMetaBucketDelete' in metakv",
  "warnings": null
 }
}
2021/12/17 04:56:34 clusterSetup.go:346: Waiting for app: TestMetaBucketDelete to get deployed
2021/12/17 04:56:39 clusterSetup.go:346: Waiting for app: TestMetaBucketDelete to get deployed
2021/12/17 04:56:44 clusterSetup.go:346: Waiting for app: TestMetaBucketDelete to get deployed
2021/12/17 04:56:49 clusterSetup.go:346: Waiting for app: TestMetaBucketDelete to get deployed
2021/12/17 04:56:49 clusterSetup.go:354: App: TestMetaBucketDelete got deployed
2021/12/17 04:56:49 handler_test.go:902: Deleting metadata bucket: eventing
2021/12/17 04:56:50 handler_test.go:904: Deleted metadata bucket: eventing
2021/12/17 04:56:55 clusterSetup.go:397: Waiting for app: TestMetaBucketDelete to get un-deployed
2021/12/17 04:56:55 clusterSetup.go:405: App: TestMetaBucketDelete got un-deployed
2021/12/17 04:57:10 clusterSetup.go:397: Waiting for app: TestMetaBucketDelete to get un-deployed
2021/12/17 04:57:10 clusterSetup.go:405: App: TestMetaBucketDelete got un-deployed
2021/12/17 04:57:10 eventingSetup.go:46: No eventing-con process running
2021/12/17 04:57:10 eventingSetup.go:418: Function: TestMetaBucketDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMetaBucketDelete deleting in the background"
}
2021/12/17 04:57:12 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/17 04:57:17 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 04:57:18 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/17 04:57:23 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestMetaBucketDelete (78.57s)
=== RUN   TestMetaBucketDeleteWithBootstrap
2021/12/17 04:57:33 clusterSetup.go:397: Waiting for app: TestMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/17 04:57:33 clusterSetup.go:405: App: TestMetaBucketDeleteWithBootstrap got un-deployed
2021/12/17 04:57:33 eventingSetup.go:46: No eventing-con process running
2021/12/17 04:57:33 eventingSetup.go:418: Function: TestMetaBucketDeleteWithBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMetaBucketDeleteWithBootstrap not found"
 }
}
2021/12/17 04:57:35 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/17 04:57:40 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 04:57:41 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/17 04:57:46 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 04:57:46 eventingSetup.go:125: Deploying app: TestMetaBucketDeleteWithBootstrap. Item count src bucket: 0 dst bucket: 0
2021/12/17 04:57:48 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMetaBucketDeleteWithBootstrap' in metakv",
  "warnings": null
 }
}
2021/12/17 04:58:06 handler_test.go:923: Deleting metadata bucket: eventing
2021/12/17 04:58:07 handler_test.go:925: Deleted metadata bucket: eventing
2021/12/17 04:58:22 clusterSetup.go:397: Waiting for app: TestMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/17 04:58:22 clusterSetup.go:405: App: TestMetaBucketDeleteWithBootstrap got un-deployed
2021/12/17 04:58:37 clusterSetup.go:397: Waiting for app: TestMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/17 04:58:37 clusterSetup.go:405: App: TestMetaBucketDeleteWithBootstrap got un-deployed
2021/12/17 04:58:37 eventingSetup.go:46: No eventing-con process running
2021/12/17 04:58:37 eventingSetup.go:418: Function: TestMetaBucketDeleteWithBootstrap delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMetaBucketDeleteWithBootstrap deleting in the background"
}
2021/12/17 04:58:38 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/17 04:58:43 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 04:58:45 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/17 04:58:50 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestMetaBucketDeleteWithBootstrap (86.80s)
=== RUN   TestSourceBucketDelete
2021/12/17 04:59:00 clusterSetup.go:397: Waiting for app: TestSourceBucketDelete to get un-deployed
2021/12/17 04:59:00 clusterSetup.go:405: App: TestSourceBucketDelete got un-deployed
2021/12/17 04:59:00 eventingSetup.go:46: No eventing-con process running
2021/12/17 04:59:00 eventingSetup.go:418: Function: TestSourceBucketDelete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestSourceBucketDelete not found"
 }
}
2021/12/17 04:59:01 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/17 04:59:06 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 04:59:08 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/17 04:59:13 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 04:59:13 eventingSetup.go:125: Deploying app: TestSourceBucketDelete. Item count src bucket: 0 dst bucket: 0
2021/12/17 04:59:14 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestSourceBucketDelete' in metakv",
  "warnings": null
 }
}
2021/12/17 04:59:19 clusterSetup.go:346: Waiting for app: TestSourceBucketDelete to get deployed
2021/12/17 04:59:24 clusterSetup.go:346: Waiting for app: TestSourceBucketDelete to get deployed
2021/12/17 04:59:29 clusterSetup.go:346: Waiting for app: TestSourceBucketDelete to get deployed
2021/12/17 04:59:34 clusterSetup.go:346: Waiting for app: TestSourceBucketDelete to get deployed
2021/12/17 04:59:34 clusterSetup.go:354: App: TestSourceBucketDelete got deployed
2021/12/17 04:59:34 handler_test.go:945: Deleting source bucket: default
2021/12/17 04:59:36 handler_test.go:947: Deleted source bucket: default
2021/12/17 04:59:41 clusterSetup.go:397: Waiting for app: TestSourceBucketDelete to get un-deployed
2021/12/17 04:59:41 clusterSetup.go:405: App: TestSourceBucketDelete got un-deployed
2021/12/17 04:59:56 clusterSetup.go:397: Waiting for app: TestSourceBucketDelete to get un-deployed
2021/12/17 04:59:56 clusterSetup.go:405: App: TestSourceBucketDelete got un-deployed
2021/12/17 04:59:56 eventingSetup.go:46: No eventing-con process running
2021/12/17 04:59:56 eventingSetup.go:418: Function: TestSourceBucketDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestSourceBucketDelete deleting in the background"
}
2021/12/17 04:59:57 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/17 05:00:02 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 05:00:04 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/17 05:00:09 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestSourceBucketDelete (79.06s)
=== RUN   TestSourceBucketDeleteWithBootstrap
2021/12/17 05:00:19 clusterSetup.go:397: Waiting for app: TestSourceBucketDeleteWithBootstrap to get un-deployed
2021/12/17 05:00:19 clusterSetup.go:405: App: TestSourceBucketDeleteWithBootstrap got un-deployed
2021/12/17 05:00:19 eventingSetup.go:46: No eventing-con process running
2021/12/17 05:00:19 eventingSetup.go:418: Function: TestSourceBucketDeleteWithBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestSourceBucketDeleteWithBootstrap not found"
 }
}
2021/12/17 05:00:21 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/17 05:00:26 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 05:00:27 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/17 05:00:32 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 05:00:32 eventingSetup.go:125: Deploying app: TestSourceBucketDeleteWithBootstrap. Item count src bucket: 0 dst bucket: 0
2021/12/17 05:00:33 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestSourceBucketDeleteWithBootstrap' in metakv",
  "warnings": null
 }
}
2021/12/17 05:00:52 handler_test.go:966: Deleting source bucket: default
2021/12/17 05:00:54 handler_test.go:968: Deleted source bucket: default
2021/12/17 05:01:09 clusterSetup.go:397: Waiting for app: TestSourceBucketDeleteWithBootstrap to get un-deployed
2021/12/17 05:01:09 clusterSetup.go:405: App: TestSourceBucketDeleteWithBootstrap got un-deployed
2021/12/17 05:01:24 clusterSetup.go:397: Waiting for app: TestSourceBucketDeleteWithBootstrap to get un-deployed
2021/12/17 05:01:24 clusterSetup.go:405: App: TestSourceBucketDeleteWithBootstrap got un-deployed
2021/12/17 05:01:24 eventingSetup.go:46: No eventing-con process running
2021/12/17 05:01:24 eventingSetup.go:418: Function: TestSourceBucketDeleteWithBootstrap delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestSourceBucketDeleteWithBootstrap deleting in the background"
}
2021/12/17 05:01:25 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/17 05:01:30 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 05:01:32 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/17 05:01:37 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestSourceBucketDeleteWithBootstrap (87.85s)
=== RUN   TestSourceAndMetaBucketDeleteWithBootstrap
2021/12/17 05:01:47 clusterSetup.go:397: Waiting for app: TestSourceAndMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/17 05:01:47 clusterSetup.go:405: App: TestSourceAndMetaBucketDeleteWithBootstrap got un-deployed
2021/12/17 05:01:47 eventingSetup.go:46: No eventing-con process running
2021/12/17 05:01:47 eventingSetup.go:418: Function: TestSourceAndMetaBucketDeleteWithBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestSourceAndMetaBucketDeleteWithBootstrap not found"
 }
}
2021/12/17 05:01:48 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/17 05:01:53 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 05:01:55 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/17 05:02:00 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 05:02:00 eventingSetup.go:125: Deploying app: TestSourceAndMetaBucketDeleteWithBootstrap. Item count src bucket: 0 dst bucket: 0
2021/12/17 05:02:01 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestSourceAndMetaBucketDeleteWithBootstrap' in metakv",
  "warnings": null
 }
}
2021/12/17 05:02:20 handler_test.go:988: Deleting source bucket: default
2021/12/17 05:02:21 handler_test.go:990: Deleted source bucket: default
2021/12/17 05:02:21 handler_test.go:992: Deleting metadata bucket: eventing
2021/12/17 05:02:22 handler_test.go:994: Deleted metadata bucket: eventing
2021/12/17 05:02:37 clusterSetup.go:397: Waiting for app: TestSourceAndMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/17 05:02:37 clusterSetup.go:405: App: TestSourceAndMetaBucketDeleteWithBootstrap got un-deployed
2021/12/17 05:02:52 clusterSetup.go:397: Waiting for app: TestSourceAndMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/17 05:02:52 clusterSetup.go:405: App: TestSourceAndMetaBucketDeleteWithBootstrap got un-deployed
2021/12/17 05:02:52 eventingSetup.go:46: No eventing-con process running
2021/12/17 05:02:52 eventingSetup.go:418: Function: TestSourceAndMetaBucketDeleteWithBootstrap delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestSourceAndMetaBucketDeleteWithBootstrap deleting in the background"
}
2021/12/17 05:02:54 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/17 05:02:59 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 05:03:00 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/17 05:03:05 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestSourceAndMetaBucketDeleteWithBootstrap (88.28s)
=== RUN   TestUndeployDuringBootstrap
2021/12/17 05:03:15 clusterSetup.go:397: Waiting for app: TestUndeployDuringBootstrap to get un-deployed
2021/12/17 05:03:15 clusterSetup.go:405: App: TestUndeployDuringBootstrap got un-deployed
2021/12/17 05:03:15 eventingSetup.go:46: No eventing-con process running
2021/12/17 05:03:15 eventingSetup.go:418: Function: TestUndeployDuringBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestUndeployDuringBootstrap not found"
 }
}
2021/12/17 05:03:17 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/17 05:03:22 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 05:03:23 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/17 05:03:28 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 05:03:28 eventingSetup.go:125: Deploying app: TestUndeployDuringBootstrap. Item count src bucket: 0 dst bucket: 0
2021/12/17 05:03:30 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestUndeployDuringBootstrap' in metakv",
  "warnings": null
 }
}
2021/12/17 05:03:30 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/17 05:03:41 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 05:03:42 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 05:03:42 clusterSetup.go:375: Apps undergoing bootstrap: true
2021/12/17 05:03:42 clusterSetup.go:378: App: TestUndeployDuringBootstrap started bootstrap
2021/12/17 05:03:42 clusterSetup.go:375: Apps undergoing bootstrap: true
2021/12/17 05:03:47 clusterSetup.go:375: Apps undergoing bootstrap: true
2021/12/17 05:03:52 clusterSetup.go:367: No apps undergoing bootstrap
2021/12/17 05:03:57 clusterSetup.go:397: Waiting for app: TestUndeployDuringBootstrap to get un-deployed
2021/12/17 05:04:02 clusterSetup.go:397: Waiting for app: TestUndeployDuringBootstrap to get un-deployed
2021/12/17 05:04:07 clusterSetup.go:397: Waiting for app: TestUndeployDuringBootstrap to get un-deployed
2021/12/17 05:04:07 clusterSetup.go:405: App: TestUndeployDuringBootstrap got un-deployed
2021/12/17 05:04:07 eventingSetup.go:46: No eventing-con process running
2021/12/17 05:04:32 clusterSetup.go:397: Waiting for app: TestUndeployDuringBootstrap to get un-deployed
2021/12/17 05:04:32 clusterSetup.go:405: App: TestUndeployDuringBootstrap got un-deployed
2021/12/17 05:04:32 eventingSetup.go:46: No eventing-con process running
2021/12/17 05:04:32 eventingSetup.go:418: Function: TestUndeployDuringBootstrap delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestUndeployDuringBootstrap deleting in the background"
}
2021/12/17 05:04:33 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/17 05:04:43 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/17 05:04:48 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/17 05:04:48 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 05:04:50 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/17 05:05:00 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2170
2021/12/17 05:05:05 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2170
2021/12/17 05:05:05 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestUndeployDuringBootstrap (119.86s)
=== RUN   TestDeleteBeforeUndeploy
2021/12/17 05:05:10 eventingSetup.go:125: Deploying app: TestDeleteBeforeUndeploy. Item count src bucket: 0 dst bucket: 0
2021/12/17 05:05:11 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeleteBeforeUndeploy' in metakv",
  "warnings": null
 }
}
2021/12/17 05:05:16 clusterSetup.go:346: Waiting for app: TestDeleteBeforeUndeploy to get deployed
2021/12/17 05:05:21 clusterSetup.go:346: Waiting for app: TestDeleteBeforeUndeploy to get deployed
2021/12/17 05:05:26 clusterSetup.go:346: Waiting for app: TestDeleteBeforeUndeploy to get deployed
2021/12/17 05:05:31 clusterSetup.go:346: Waiting for app: TestDeleteBeforeUndeploy to get deployed
2021/12/17 05:05:31 clusterSetup.go:354: App: TestDeleteBeforeUndeploy got deployed
2021/12/17 05:05:31 eventingSetup.go:418: Function: TestDeleteBeforeUndeploy delete from main store request response code: 422 dump: {
 "name": "ERR_APP_NOT_UNDEPLOYED",
 "code": 22,
 "description": "Function hasn't undeployed",
 "attributes": null,
 "runtime_info": {
  "code": 22,
  "info": "Function: TestDeleteBeforeUndeploy skipping delete request from primary store, as it hasn't been undeployed"
 }
}
2021/12/17 05:05:32 eventingSetup.go:270: Function: TestDeleteBeforeUndeploy update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/17 05:05:32 eventingSetup.go:418: Function: TestDeleteBeforeUndeploy delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDeleteBeforeUndeploy deleting in the background"
}
2021/12/17 05:05:37 clusterSetup.go:397: Waiting for app: TestDeleteBeforeUndeploy to get un-deployed
2021/12/17 05:05:37 clusterSetup.go:405: App: TestDeleteBeforeUndeploy got un-deployed
2021/12/17 05:05:37 eventingSetup.go:46: No eventing-con process running
2021/12/17 05:05:37 eventingSetup.go:418: Function: TestDeleteBeforeUndeploy delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDeleteBeforeUndeploy not found"
 }
}
2021/12/17 05:05:38 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/17 05:05:43 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 05:05:45 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/17 05:05:50 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDeleteBeforeUndeploy (44.66s)
=== RUN   TestUndeployWhenTimersAreFired
2021/12/17 05:06:00 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/17 05:06:00 clusterSetup.go:405: App: TestUndeployWhenTimersAreFired got un-deployed
2021/12/17 05:06:00 eventingSetup.go:46: No eventing-con process running
2021/12/17 05:06:00 eventingSetup.go:418: Function: TestUndeployWhenTimersAreFired delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestUndeployWhenTimersAreFired not found"
 }
}
2021/12/17 05:06:01 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/17 05:06:06 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 05:06:08 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/17 05:06:13 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 05:06:13 eventingSetup.go:125: Deploying app: TestUndeployWhenTimersAreFired. Item count src bucket: 0 dst bucket: 0
2021/12/17 05:06:15 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestUndeployWhenTimersAreFired' in metakv",
  "warnings": null
 }
}
2021/12/17 05:06:20 clusterSetup.go:346: Waiting for app: TestUndeployWhenTimersAreFired to get deployed
2021/12/17 05:06:25 clusterSetup.go:346: Waiting for app: TestUndeployWhenTimersAreFired to get deployed
2021/12/17 05:06:30 clusterSetup.go:346: Waiting for app: TestUndeployWhenTimersAreFired to get deployed
2021/12/17 05:06:35 clusterSetup.go:346: Waiting for app: TestUndeployWhenTimersAreFired to get deployed
2021/12/17 05:06:35 clusterSetup.go:354: App: TestUndeployWhenTimersAreFired got deployed
2021/12/17 05:06:35 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/17 05:07:05 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 05:07:10 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/17 05:07:15 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/17 05:07:20 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/17 05:07:25 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/17 05:07:25 clusterSetup.go:405: App: TestUndeployWhenTimersAreFired got un-deployed
2021/12/17 05:07:25 eventingSetup.go:46: No eventing-con process running
2021/12/17 05:09:10 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/17 05:09:10 clusterSetup.go:405: App: TestUndeployWhenTimersAreFired got un-deployed
2021/12/17 05:09:10 eventingSetup.go:46: No eventing-con process running
2021/12/17 05:09:10 eventingSetup.go:418: Function: TestUndeployWhenTimersAreFired delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestUndeployWhenTimersAreFired deleting in the background"
}
2021/12/17 05:09:11 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/17 05:09:21 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/17 05:09:26 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/17 05:09:26 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 05:09:28 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/17 05:09:38 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3322
2021/12/17 05:09:43 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 822
2021/12/17 05:09:43 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestUndeployWhenTimersAreFired (233.09s)
=== RUN   TestUndeployWithKVFailover
2021/12/17 05:09:53 clusterSetup.go:397: Waiting for app: TestUndeployWithKVFailover to get un-deployed
2021/12/17 05:09:53 clusterSetup.go:405: App: TestUndeployWithKVFailover got un-deployed
2021/12/17 05:09:53 eventingSetup.go:46: No eventing-con process running
2021/12/17 05:09:53 eventingSetup.go:418: Function: TestUndeployWithKVFailover delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestUndeployWithKVFailover not found"
 }
}
2021/12/17 05:09:54 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/17 05:09:59 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 05:10:01 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/17 05:10:06 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 05:10:06 eventingSetup.go:125: Deploying app: TestUndeployWithKVFailover. Item count src bucket: 0 dst bucket: 0
2021/12/17 05:10:07 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestUndeployWithKVFailover' in metakv",
  "warnings": null
 }
}
2021/12/17 05:10:07 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/17 05:10:14 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 05:10:19 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 05:10:24 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 05:10:29 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 05:10:34 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 05:10:39 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 95
2021/12/17 05:10:44 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 95
2021/12/17 05:10:44 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/17 05:10:44 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: kv to the cluster
2021/12/17 05:10:44 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/17 05:10:50 Rebalance progress: 29.16666666666666
2021/12/17 05:10:55 Rebalance progress: 29.16666666666666
2021/12/17 05:11:00 Rebalance progress: 100
2021/12/17 05:11:00 Failing over: [127.0.0.1:19003]
2021/12/17 05:11:10 Function: TestUndeployWithKVFailover update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/17 05:12:10 Removing node(s): [] from the cluster
2021/12/17 05:12:15 Rebalance progress: 29.16666666666666
2021/12/17 05:12:20 Rebalance progress: 29.16666666666666
2021/12/17 05:12:25 Rebalance progress: 100
2021/12/17 05:12:30 Waiting for app: TestUndeployWithKVFailover to get un-deployed
2021/12/17 05:12:30 App: TestUndeployWithKVFailover got un-deployed
2021/12/17 05:12:30 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 05:12:35 Waiting for app: TestUndeployWithKVFailover to get un-deployed
2021/12/17 05:12:35 App: TestUndeployWithKVFailover got un-deployed
2021/12/17 05:12:35 No eventing-con process running
2021/12/17 05:12:35 Function: TestUndeployWithKVFailover delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestUndeployWithKVFailover deleting in the background"
}
2021/12/17 05:12:36 Context: Bucket flush, response code: 200 dump: 
2021/12/17 05:12:46 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/17 05:12:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 05:12:48 Context: Bucket flush, response code: 200 dump: 
2021/12/17 05:12:58 Waiting for dst bucket item count to get to: 0 curr count: 3168
2021/12/17 05:13:03 Waiting for dst bucket item count to get to: 0 curr count: 668
2021/12/17 05:13:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestUndeployWithKVFailover (199.92s)
=== RUN   TestBucketFlushWhileFnDeployed
2021/12/17 05:13:13 Waiting for app: TestBucketFlushWhileFnDeployed to get un-deployed
2021/12/17 05:13:13 App: TestBucketFlushWhileFnDeployed got un-deployed
2021/12/17 05:13:13 No eventing-con process running
2021/12/17 05:13:13 Function: TestBucketFlushWhileFnDeployed delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestBucketFlushWhileFnDeployed not found"
 }
}
2021/12/17 05:13:14 Context: Bucket flush, response code: 200 dump: 
2021/12/17 05:13:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 05:13:21 Context: Bucket flush, response code: 200 dump: 
2021/12/17 05:13:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 05:13:26 Deploying app: TestBucketFlushWhileFnDeployed. Item count src bucket: 0 dst bucket: 0
2021/12/17 05:13:27 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestBucketFlushWhileFnDeployed' in metakv",
  "warnings": null
 }
}
2021/12/17 05:13:32 Waiting for app: TestBucketFlushWhileFnDeployed to get deployed
2021/12/17 05:13:37 Waiting for app: TestBucketFlushWhileFnDeployed to get deployed
2021/12/17 05:13:42 Waiting for app: TestBucketFlushWhileFnDeployed to get deployed
2021/12/17 05:13:47 Waiting for app: TestBucketFlushWhileFnDeployed to get deployed
2021/12/17 05:13:47 App: TestBucketFlushWhileFnDeployed got deployed
2021/12/17 05:13:47 Starting bucket ops to source bucket
2021/12/17 05:14:08 Waiting for dst bucket item count to get to: 20000 curr count: 13851
2021/12/17 05:14:13 Waiting for dst bucket item count to get to: 20000 curr count: 13851
2021/12/17 05:14:13 src & dst bucket item count matched up. src bucket count: 20000 dst bucket count: 20000
2021/12/17 05:14:13 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 05:14:16 Context: Bucket flush, response code: 200 dump: 
2021/12/17 05:14:26 Waiting for dst bucket item count to get to: 0 curr count: 20000
2021/12/17 05:14:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 05:14:28 Context: Bucket flush, response code: 200 dump: 
2021/12/17 05:14:38 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/17 05:14:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 05:14:38 Starting bucket ops to source bucket
2021/12/17 05:14:51 Waiting for dst bucket item count to get to: 10000 curr count: 993
2021/12/17 05:14:56 Waiting for dst bucket item count to get to: 10000 curr count: 5993
2021/12/17 05:14:56 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/17 05:14:56 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 05:14:56 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 05:15:01 Waiting for app: TestBucketFlushWhileFnDeployed to get un-deployed
2021/12/17 05:15:06 Waiting for app: TestBucketFlushWhileFnDeployed to get un-deployed
2021/12/17 05:15:11 Waiting for app: TestBucketFlushWhileFnDeployed to get un-deployed
2021/12/17 05:15:16 Waiting for app: TestBucketFlushWhileFnDeployed to get un-deployed
2021/12/17 05:15:16 App: TestBucketFlushWhileFnDeployed got un-deployed
2021/12/17 05:15:16 No eventing-con process running
2021/12/17 05:15:16 Function: TestBucketFlushWhileFnDeployed delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestBucketFlushWhileFnDeployed deleting in the background"
}
2021/12/17 05:15:17 Context: Bucket flush, response code: 200 dump: 
2021/12/17 05:15:27 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/17 05:15:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 05:15:29 Context: Bucket flush, response code: 200 dump: 
2021/12/17 05:15:39 Waiting for dst bucket item count to get to: 0 curr count: 9749
2021/12/17 05:15:44 Waiting for dst bucket item count to get to: 0 curr count: 4749
2021/12/17 05:15:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestBucketFlushWhileFnDeployed (160.97s)
=== RUN   TestTimerOverwrite
2021/12/17 05:15:54 Waiting for app: TestTimerOverwrite to get un-deployed
2021/12/17 05:15:54 App: TestTimerOverwrite got un-deployed
2021/12/17 05:15:54 No eventing-con process running
2021/12/17 05:15:54 Function: TestTimerOverwrite delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestTimerOverwrite not found"
 }
}
2021/12/17 05:15:55 Context: Bucket flush, response code: 200 dump: 
2021/12/17 05:16:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 05:16:02 Context: Bucket flush, response code: 200 dump: 
2021/12/17 05:16:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 05:16:07 Deploying app: TestTimerOverwrite. Item count src bucket: 0 dst bucket: 0
2021/12/17 05:16:08 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestTimerOverwrite' in metakv",
  "warnings": null
 }
}
2021/12/17 05:16:13 Waiting for app: TestTimerOverwrite to get deployed
2021/12/17 05:16:18 Waiting for app: TestTimerOverwrite to get deployed
2021/12/17 05:16:23 Waiting for app: TestTimerOverwrite to get deployed
2021/12/17 05:16:28 Waiting for app: TestTimerOverwrite to get deployed
2021/12/17 05:16:28 App: TestTimerOverwrite got deployed
2021/12/17 05:16:38 Starting bucket ops to source bucket
2021/12/17 05:16:47 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 05:16:52 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 05:16:57 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 05:17:02 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 05:17:07 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 05:17:12 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 05:17:17 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 05:17:22 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 05:17:27 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 05:17:32 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 05:17:37 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 05:17:42 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 05:17:47 Waiting for dst bucket item count to get to: 5000 curr count: 1306
2021/12/17 05:17:52 Waiting for dst bucket item count to get to: 5000 curr count: 3764
2021/12/17 05:17:52 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/17 05:18:02 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 05:18:03 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 05:18:08 Waiting for app: TestTimerOverwrite to get un-deployed
2021/12/17 05:18:13 Waiting for app: TestTimerOverwrite to get un-deployed
2021/12/17 05:18:18 Waiting for app: TestTimerOverwrite to get un-deployed
2021/12/17 05:18:23 Waiting for app: TestTimerOverwrite to get un-deployed
2021/12/17 05:18:23 App: TestTimerOverwrite got un-deployed
2021/12/17 05:18:23 No eventing-con process running
2021/12/17 05:18:23 Function: TestTimerOverwrite delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestTimerOverwrite deleting in the background"
}
2021/12/17 05:18:24 Context: Bucket flush, response code: 200 dump: 
2021/12/17 05:18:34 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/17 05:18:39 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/17 05:18:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 05:18:41 Context: Bucket flush, response code: 200 dump: 
2021/12/17 05:18:51 Waiting for dst bucket item count to get to: 0 curr count: 4464
2021/12/17 05:18:56 Waiting for dst bucket item count to get to: 0 curr count: 4464
2021/12/17 05:18:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestTimerOverwrite (191.86s)
=== RUN   TestUndeployBackdoorDuringBootstrap
2021/12/17 05:19:01 Adding node: https://127.0.0.1:19003 with role: kv to the cluster
2021/12/17 05:19:01 Removing node(s): [] from the cluster
2021/12/17 05:19:06 Rebalance progress: 100
2021/12/17 05:19:06 Failing over: [127.0.0.1:19003]
2021/12/17 05:19:11 Waiting for app: TestUndeployBackdoorDuringBootstrap to get un-deployed
2021/12/17 05:19:11 App: TestUndeployBackdoorDuringBootstrap got un-deployed
2021/12/17 05:19:11 No eventing-con process running
2021/12/17 05:19:11 Function: TestUndeployBackdoorDuringBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestUndeployBackdoorDuringBootstrap not found"
 }
}
2021/12/17 05:19:12 Context: Bucket flush, response code: 200 dump: 
2021/12/17 05:19:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 05:19:19 Context: Bucket flush, response code: 200 dump: 
2021/12/17 05:19:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 05:19:24 Deploying app: TestUndeployBackdoorDuringBootstrap. Item count src bucket: 0 dst bucket: 0
2021/12/17 05:19:25 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestUndeployBackdoorDuringBootstrap' in metakv",
  "warnings": null
 }
}
2021/12/17 05:19:25 Starting bucket ops to source bucket
2021/12/17 05:19:35 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 05:19:35 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 05:19:35 Context: Post to set retry counter, response code: 200 dump: 
2021/12/17 05:19:35 set retry counter, response &{[] }
2021/12/17 05:20:40 Waiting for app: TestUndeployBackdoorDuringBootstrap to get un-deployed
2021/12/17 05:20:40 App: TestUndeployBackdoorDuringBootstrap got un-deployed
2021/12/17 05:20:40 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 05:20:41 Function: TestUndeployBackdoorDuringBootstrap delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestUndeployBackdoorDuringBootstrap deleting in the background"
}
2021/12/17 05:20:46 Waiting for app: TestUndeployBackdoorDuringBootstrap to get un-deployed
2021/12/17 05:20:46 App: TestUndeployBackdoorDuringBootstrap got un-deployed
2021/12/17 05:20:46 No eventing-con process running
2021/12/17 05:20:46 Function: TestUndeployBackdoorDuringBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestUndeployBackdoorDuringBootstrap not found"
 }
}
2021/12/17 05:20:47 Context: Bucket flush, response code: 200 dump: 
2021/12/17 05:20:57 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/17 05:20:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 05:20:59 Context: Bucket flush, response code: 200 dump: 
2021/12/17 05:21:09 Waiting for dst bucket item count to get to: 0 curr count: 2968
2021/12/17 05:21:14 Waiting for dst bucket item count to get to: 0 curr count: 468
2021/12/17 05:21:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 05:21:14 Removing node(s): [] from the cluster
2021/12/17 05:21:19 Rebalance progress: 100
--- PASS: TestUndeployBackdoorDuringBootstrap (142.91s)
=== RUN   TestOnUpdateSrcMutation
2021/12/17 05:21:29 Waiting for app: TestOnUpdateSrcMutation to get un-deployed
2021/12/17 05:21:29 App: TestOnUpdateSrcMutation got un-deployed
2021/12/17 05:21:29 No eventing-con process running
2021/12/17 05:21:29 Function: TestOnUpdateSrcMutation delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateSrcMutation not found"
 }
}
2021/12/17 05:21:30 Context: Bucket flush, response code: 200 dump: 
2021/12/17 05:21:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 05:21:37 Context: Bucket flush, response code: 200 dump: 
2021/12/17 05:21:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 05:21:42 Deploying app: TestOnUpdateSrcMutation. Item count src bucket: 0 dst bucket: 0
2021/12/17 05:21:43 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateSrcMutation' in metakv",
  "warnings": null
 }
}
2021/12/17 05:21:43 Starting bucket ops to source bucket
2021/12/17 05:22:05 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/17 05:22:05 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 05:22:05 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 05:22:10 Waiting for app: TestOnUpdateSrcMutation to get un-deployed
2021/12/17 05:22:15 Waiting for app: TestOnUpdateSrcMutation to get un-deployed
2021/12/17 05:22:20 Waiting for app: TestOnUpdateSrcMutation to get un-deployed
2021/12/17 05:22:25 Waiting for app: TestOnUpdateSrcMutation to get un-deployed
2021/12/17 05:22:25 App: TestOnUpdateSrcMutation got un-deployed
2021/12/17 05:22:25 No eventing-con process running
2021/12/17 05:22:25 Function: TestOnUpdateSrcMutation delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateSrcMutation deleting in the background"
}
2021/12/17 05:22:26 Context: Bucket flush, response code: 200 dump: 
2021/12/17 05:22:36 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/17 05:22:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 05:22:38 Context: Bucket flush, response code: 200 dump: 
2021/12/17 05:22:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateSrcMutation (84.68s)
=== RUN   TestOnDeleteSrcMutation
2021/12/17 05:22:53 Waiting for app: TestOnDeleteSrcMutation to get un-deployed
2021/12/17 05:22:53 App: TestOnDeleteSrcMutation got un-deployed
2021/12/17 05:22:53 No eventing-con process running
2021/12/17 05:22:53 Function: TestOnDeleteSrcMutation delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnDeleteSrcMutation not found"
 }
}
2021/12/17 05:22:55 Context: Bucket flush, response code: 200 dump: 
2021/12/17 05:23:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 05:23:01 Context: Bucket flush, response code: 200 dump: 
2021/12/17 05:23:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 05:23:06 Deploying app: TestOnDeleteSrcMutation. Item count src bucket: 0 dst bucket: 0
2021/12/17 05:23:08 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnDeleteSrcMutation' in metakv",
  "warnings": null
 }
}
2021/12/17 05:23:08 Starting bucket ops to source bucket
2021/12/17 05:23:36 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/17 05:23:36 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 05:23:36 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 05:23:41 Waiting for app: TestOnDeleteSrcMutation to get un-deployed
2021/12/17 05:23:46 Waiting for app: TestOnDeleteSrcMutation to get un-deployed
2021/12/17 05:23:51 Waiting for app: TestOnDeleteSrcMutation to get un-deployed
2021/12/17 05:23:56 Waiting for app: TestOnDeleteSrcMutation to get un-deployed
2021/12/17 05:23:56 App: TestOnDeleteSrcMutation got un-deployed
2021/12/17 05:23:56 No eventing-con process running
2021/12/17 05:23:56 Function: TestOnDeleteSrcMutation delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnDeleteSrcMutation deleting in the background"
}
2021/12/17 05:23:58 Context: Bucket flush, response code: 200 dump: 
2021/12/17 05:24:08 Waiting for dst bucket item count to get to: 0 curr count: 2795
2021/12/17 05:24:13 Waiting for dst bucket item count to get to: 0 curr count: 295
2021/12/17 05:24:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 05:24:15 Context: Bucket flush, response code: 200 dump: 
2021/12/17 05:24:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnDeleteSrcMutation (96.27s)
=== RUN   TestOnUpdateSrcMutationWithTimer
2021/12/17 05:24:30 Waiting for app: TestOnUpdateSrcMutationWithTimer to get un-deployed
2021/12/17 05:24:30 App: TestOnUpdateSrcMutationWithTimer got un-deployed
2021/12/17 05:24:30 No eventing-con process running
2021/12/17 05:24:30 Function: TestOnUpdateSrcMutationWithTimer delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateSrcMutationWithTimer not found"
 }
}
2021/12/17 05:24:31 Context: Bucket flush, response code: 200 dump: 
2021/12/17 05:24:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 05:24:38 Context: Bucket flush, response code: 200 dump: 
2021/12/17 05:24:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 05:24:43 Deploying app: TestOnUpdateSrcMutationWithTimer. Item count src bucket: 0 dst bucket: 0
2021/12/17 05:24:44 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateSrcMutationWithTimer' in metakv",
  "warnings": null
 }
}
2021/12/17 05:24:44 Starting bucket ops to source bucket
2021/12/17 05:25:26 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/17 05:25:26 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 05:25:26 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 05:25:31 Waiting for app: TestOnUpdateSrcMutationWithTimer to get un-deployed
2021/12/17 05:25:36 Waiting for app: TestOnUpdateSrcMutationWithTimer to get un-deployed
2021/12/17 05:25:41 Waiting for app: TestOnUpdateSrcMutationWithTimer to get un-deployed
2021/12/17 05:25:46 Waiting for app: TestOnUpdateSrcMutationWithTimer to get un-deployed
2021/12/17 05:25:46 App: TestOnUpdateSrcMutationWithTimer got un-deployed
2021/12/17 05:25:46 No eventing-con process running
2021/12/17 05:25:46 Function: TestOnUpdateSrcMutationWithTimer delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateSrcMutationWithTimer deleting in the background"
}
2021/12/17 05:25:47 Context: Bucket flush, response code: 200 dump: 
2021/12/17 05:25:57 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/17 05:25:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 05:25:59 Context: Bucket flush, response code: 200 dump: 
2021/12/17 05:26:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateSrcMutationWithTimer (104.63s)
=== RUN   TestOnDeleteSrcMutationsWithTimer
2021/12/17 05:26:14 Waiting for app: TestOnDeleteSrcMutationsWithTimer to get un-deployed
2021/12/17 05:26:14 App: TestOnDeleteSrcMutationsWithTimer got un-deployed
2021/12/17 05:26:14 No eventing-con process running
2021/12/17 05:26:14 Function: TestOnDeleteSrcMutationsWithTimer delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnDeleteSrcMutationsWithTimer not found"
 }
}
2021/12/17 05:26:16 Context: Bucket flush, response code: 200 dump: 
2021/12/17 05:26:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 05:26:22 Context: Bucket flush, response code: 200 dump: 
2021/12/17 05:26:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 05:26:27 Deploying app: TestOnDeleteSrcMutationsWithTimer. Item count src bucket: 0 dst bucket: 0
2021/12/17 05:26:29 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnDeleteSrcMutationsWithTimer' in metakv",
  "warnings": null
 }
}
2021/12/17 05:26:29 Starting bucket ops to source bucket
2021/12/17 05:27:07 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/17 05:27:07 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 05:27:07 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 05:27:12 Waiting for app: TestOnDeleteSrcMutationsWithTimer to get un-deployed
2021/12/17 05:27:17 Waiting for app: TestOnDeleteSrcMutationsWithTimer to get un-deployed
2021/12/17 05:27:22 Waiting for app: TestOnDeleteSrcMutationsWithTimer to get un-deployed
2021/12/17 05:27:27 Waiting for app: TestOnDeleteSrcMutationsWithTimer to get un-deployed
2021/12/17 05:27:27 App: TestOnDeleteSrcMutationsWithTimer got un-deployed
2021/12/17 05:27:27 No eventing-con process running
2021/12/17 05:27:27 Function: TestOnDeleteSrcMutationsWithTimer delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnDeleteSrcMutationsWithTimer deleting in the background"
}
2021/12/17 05:27:29 Context: Bucket flush, response code: 200 dump: 
2021/12/17 05:27:39 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/17 05:27:44 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/17 05:27:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 05:27:46 Context: Bucket flush, response code: 200 dump: 
2021/12/17 05:27:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnDeleteSrcMutationsWithTimer (106.38s)
=== RUN   TestInterHandlerRecursion
2021/12/17 05:28:01 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/17 05:28:01 App: TestInterHandlerRecursion_source_bucket_update_op got un-deployed
2021/12/17 05:28:01 No eventing-con process running
2021/12/17 05:28:01 Function: TestInterHandlerRecursion_source_bucket_update_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestInterHandlerRecursion_source_bucket_update_op not found"
 }
}
2021/12/17 05:28:02 Context: Bucket flush, response code: 200 dump: 
2021/12/17 05:28:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 05:28:09 Context: Bucket flush, response code: 200 dump: 
2021/12/17 05:28:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 05:28:19 Waiting for app: TestInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/17 05:28:19 App: TestInterHandlerRecursion_src_bucket_op_on_delete got un-deployed
2021/12/17 05:28:19 No eventing-con process running
2021/12/17 05:28:19 Function: TestInterHandlerRecursion_src_bucket_op_on_delete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestInterHandlerRecursion_src_bucket_op_on_delete not found"
 }
}
2021/12/17 05:28:20 Context: Bucket flush, response code: 200 dump: 
2021/12/17 05:28:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 05:28:27 Context: Bucket flush, response code: 200 dump: 
2021/12/17 05:28:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 05:28:32 Deploying app: TestInterHandlerRecursion_source_bucket_update_op. Item count src bucket: 0 dst bucket: 0
2021/12/17 05:28:33 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterHandlerRecursion_source_bucket_update_op' in metakv",
  "warnings": null
 }
}
2021/12/17 05:28:33 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterHandlerRecursion_source_bucket_update_op' in metakv",
  "warnings": null
 }
} err 
2021/12/17 05:28:38 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/17 05:28:43 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/17 05:28:48 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/17 05:28:53 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/17 05:28:53 App: TestInterHandlerRecursion_source_bucket_update_op got deployed
2021/12/17 05:28:53 Deploying app: TestInterHandlerRecursion_src_bucket_op_on_delete. Item count src bucket: 0 dst bucket: 0
2021/12/17 05:28:53 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_INTER_FUNCTION_RECURSION",
 "code": 50,
 "description": "Inter function recursion error, only one function is allowed to do source bucket mutation/delete on a bucket",
 "attributes": null,
 "runtime_info": {
  "code": 50,
  "info": "Inter handler recursion error"
 }
}
2021/12/17 05:28:53 response body {
 "name": "ERR_INTER_FUNCTION_RECURSION",
 "code": 50,
 "description": "Inter function recursion error, only one function is allowed to do source bucket mutation/delete on a bucket",
 "attributes": null,
 "runtime_info": {
  "code": 50,
  "info": "Inter handler recursion error"
 }
} err 
2021/12/17 05:28:53 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 05:28:58 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/17 05:29:03 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/17 05:29:08 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/17 05:29:13 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/17 05:29:13 App: TestInterHandlerRecursion_source_bucket_update_op got un-deployed
2021/12/17 05:29:13 No eventing-con process running
2021/12/17 05:29:13 Function: TestInterHandlerRecursion_source_bucket_update_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestInterHandlerRecursion_source_bucket_update_op deleting in the background"
}
2021/12/17 05:29:15 Context: Bucket flush, response code: 200 dump: 
2021/12/17 05:29:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 05:29:21 Context: Bucket flush, response code: 200 dump: 
2021/12/17 05:29:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 05:29:31 Waiting for app: TestInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/17 05:29:31 App: TestInterHandlerRecursion_src_bucket_op_on_delete got un-deployed
2021/12/17 05:29:31 No eventing-con process running
2021/12/17 05:29:31 Function: TestInterHandlerRecursion_src_bucket_op_on_delete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestInterHandlerRecursion_src_bucket_op_on_delete not found"
 }
}
2021/12/17 05:29:33 Context: Bucket flush, response code: 200 dump: 
2021/12/17 05:29:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 05:29:39 Context: Bucket flush, response code: 200 dump: 
2021/12/17 05:29:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestInterHandlerRecursion (113.84s)
=== RUN   TestInterBucketRecursion
2021/12/17 05:29:49 Deploying app: TestInterBucketRecursion_function1. Item count src bucket: 0 dst bucket: 0
2021/12/17 05:29:51 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterBucketRecursion_function1' in metakv",
  "warnings": null
 }
}
2021/12/17 05:29:51 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterBucketRecursion_function1' in metakv",
  "warnings": null
 }
} err 
2021/12/17 05:29:51 Deploying app: TestInterBucketRecursion_function2. Item count src bucket: 0 dst bucket: 0
2021/12/17 05:29:51 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_INTER_BUCKET_RECURSION",
 "code": 51,
 "description": "Inter bucket recursion error, deployment of current handler will cause inter bucket recursion",
 "attributes": null,
 "runtime_info": {
  "code": 51,
  "info": "Inter bucket recursion error; function: TestInterBucketRecursion_function2 causes a cycle involving functions: [TestInterBucketRecursion_function1], hence deployment is disallowed"
 }
}
2021/12/17 05:29:56 Waiting for app: TestInterBucketRecursion_function1 to get deployed
2021/12/17 05:30:01 Waiting for app: TestInterBucketRecursion_function1 to get deployed
2021/12/17 05:30:06 Waiting for app: TestInterBucketRecursion_function1 to get deployed
2021/12/17 05:30:11 Waiting for app: TestInterBucketRecursion_function1 to get deployed
2021/12/17 05:30:11 App: TestInterBucketRecursion_function1 got deployed
2021/12/17 05:30:11 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 05:30:16 Waiting for app: TestInterBucketRecursion_function1 to get un-deployed
2021/12/17 05:30:21 Waiting for app: TestInterBucketRecursion_function1 to get un-deployed
2021/12/17 05:30:26 Waiting for app: TestInterBucketRecursion_function1 to get un-deployed
2021/12/17 05:30:31 Waiting for app: TestInterBucketRecursion_function1 to get un-deployed
2021/12/17 05:30:31 App: TestInterBucketRecursion_function1 got un-deployed
2021/12/17 05:30:31 No eventing-con process running
2021/12/17 05:30:31 Function: TestInterBucketRecursion_function1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestInterBucketRecursion_function1 deleting in the background"
}
2021/12/17 05:30:32 Context: Bucket flush, response code: 200 dump: 
2021/12/17 05:30:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 05:30:39 Context: Bucket flush, response code: 200 dump: 
2021/12/17 05:30:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 05:30:49 Waiting for app: TestInterBucketRecursion_function2 to get un-deployed
2021/12/17 05:30:49 App: TestInterBucketRecursion_function2 got un-deployed
2021/12/17 05:30:49 No eventing-con process running
2021/12/17 05:30:49 Function: TestInterBucketRecursion_function2 delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestInterBucketRecursion_function2 not found"
 }
}
2021/12/17 05:30:50 Context: Bucket flush, response code: 200 dump: 
2021/12/17 05:30:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 05:30:57 Context: Bucket flush, response code: 200 dump: 
2021/12/17 05:31:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestInterBucketRecursion (77.54s)
=== RUN   TestLargeHandler
2021/12/17 05:31:02 Changed force_compress value to false
2021/12/17 05:31:02 Deploying app: TestLargeHandler_function. Item count src bucket: 0 dst bucket: 0
2021/12/17 05:31:02 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_APPCODE_SIZE",
 "code": 39,
 "description": "Function Code size is more than the configured limit",
 "attributes": null,
 "runtime_info": {
  "code": 39,
  "info": "Function: TestLargeHandler_function handler Code size is more than 131072. Code Size: 131768"
 }
}
2021/12/17 05:31:02 Changed force_compress value to true
2021/12/17 05:31:02 Deploying app: TestLargeHandler_function. Item count src bucket: 0 dst bucket: 0
2021/12/17 05:31:04 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestLargeHandler_function' in metakv",
  "warnings": null
 }
}
2021/12/17 05:31:09 Waiting for app: TestLargeHandler_function to get deployed
2021/12/17 05:31:14 Waiting for app: TestLargeHandler_function to get deployed
2021/12/17 05:31:19 Waiting for app: TestLargeHandler_function to get deployed
2021/12/17 05:31:24 Waiting for app: TestLargeHandler_function to get deployed
2021/12/17 05:31:24 App: TestLargeHandler_function got deployed
2021/12/17 05:31:24 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 05:31:29 Waiting for app: TestLargeHandler_function to get un-deployed
2021/12/17 05:31:34 Waiting for app: TestLargeHandler_function to get un-deployed
2021/12/17 05:31:39 Waiting for app: TestLargeHandler_function to get un-deployed
2021/12/17 05:31:44 Waiting for app: TestLargeHandler_function to get un-deployed
2021/12/17 05:31:44 App: TestLargeHandler_function got un-deployed
2021/12/17 05:31:44 No eventing-con process running
2021/12/17 05:31:44 Function: TestLargeHandler_function delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestLargeHandler_function deleting in the background"
}
2021/12/17 05:31:46 Context: Bucket flush, response code: 200 dump: 
2021/12/17 05:31:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 05:31:52 Context: Bucket flush, response code: 200 dump: 
2021/12/17 05:31:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestLargeHandler (55.40s)
=== RUN   TestAllowInterHandlerRecursion
2021/12/17 05:32:07 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/17 05:32:07 App: TestAllowInterHandlerRecursion_source_bucket_update_op got un-deployed
2021/12/17 05:32:07 No eventing-con process running
2021/12/17 05:32:07 Function: TestAllowInterHandlerRecursion_source_bucket_update_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestAllowInterHandlerRecursion_source_bucket_update_op not found"
 }
}
2021/12/17 05:32:09 Context: Bucket flush, response code: 200 dump: 
2021/12/17 05:32:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 05:32:15 Context: Bucket flush, response code: 200 dump: 
2021/12/17 05:32:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 05:32:25 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/17 05:32:25 App: TestAllowInterHandlerRecursion_src_bucket_op_on_delete got un-deployed
2021/12/17 05:32:25 No eventing-con process running
2021/12/17 05:32:25 Function: TestAllowInterHandlerRecursion_src_bucket_op_on_delete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestAllowInterHandlerRecursion_src_bucket_op_on_delete not found"
 }
}
2021/12/17 05:32:27 Context: Bucket flush, response code: 200 dump: 
2021/12/17 05:32:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 05:32:34 Context: Bucket flush, response code: 200 dump: 
2021/12/17 05:32:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 05:32:39 Deploying app: TestAllowInterHandlerRecursion_source_bucket_update_op. Item count src bucket: 0 dst bucket: 0
2021/12/17 05:32:40 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterHandlerRecursion_source_bucket_update_op' in metakv",
  "warnings": null
 }
}
2021/12/17 05:32:40 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterHandlerRecursion_source_bucket_update_op' in metakv",
  "warnings": null
 }
} err 
2021/12/17 05:32:45 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/17 05:32:50 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/17 05:32:55 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/17 05:33:00 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/17 05:33:00 App: TestAllowInterHandlerRecursion_source_bucket_update_op got deployed
2021/12/17 05:33:00 Deploying app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete. Item count src bucket: 0 dst bucket: 0
2021/12/17 05:33:01 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterHandlerRecursion_src_bucket_op_on_delete' in metakv",
  "warnings": null
 }
}
2021/12/17 05:33:01 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterHandlerRecursion_src_bucket_op_on_delete' in metakv",
  "warnings": null
 }
} err 
2021/12/17 05:33:06 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get deployed
2021/12/17 05:33:11 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get deployed
2021/12/17 05:33:16 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get deployed
2021/12/17 05:33:21 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get deployed
2021/12/17 05:33:21 App: TestAllowInterHandlerRecursion_src_bucket_op_on_delete got deployed
2021/12/17 05:33:21 TestAllowInterHandlerRecursion_src_bucket_op_on_delete is deployed
2021/12/17 05:33:21 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 05:33:26 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/17 05:33:31 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/17 05:33:36 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/17 05:33:41 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/17 05:33:41 App: TestAllowInterHandlerRecursion_source_bucket_update_op got un-deployed
2021/12/17 05:33:41 No eventing-con process running
2021/12/17 05:33:42 Function: TestAllowInterHandlerRecursion_source_bucket_update_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAllowInterHandlerRecursion_source_bucket_update_op deleting in the background"
}
2021/12/17 05:33:44 Context: Bucket flush, response code: 200 dump: 
2021/12/17 05:33:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 05:33:51 Context: Bucket flush, response code: 200 dump: 
2021/12/17 05:33:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 05:33:56 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 05:34:01 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/17 05:34:06 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/17 05:34:11 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/17 05:34:16 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/17 05:34:16 App: TestAllowInterHandlerRecursion_src_bucket_op_on_delete got un-deployed
2021/12/17 05:34:16 No eventing-con process running
2021/12/17 05:34:16 Function: TestAllowInterHandlerRecursion_src_bucket_op_on_delete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAllowInterHandlerRecursion_src_bucket_op_on_delete deleting in the background"
}
2021/12/17 05:34:17 Context: Bucket flush, response code: 200 dump: 
2021/12/17 05:34:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 05:34:24 Context: Bucket flush, response code: 200 dump: 
2021/12/17 05:34:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAllowInterHandlerRecursion (151.50s)
=== RUN   TestAllowInterBucketRecursion
2021/12/17 05:34:34 Deploying app: TestAllowInterBucketRecursion_function1. Item count src bucket: 0 dst bucket: 0
2021/12/17 05:34:35 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterBucketRecursion_function1' in metakv",
  "warnings": null
 }
}
2021/12/17 05:34:35 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterBucketRecursion_function1' in metakv",
  "warnings": null
 }
} err 
2021/12/17 05:34:40 Waiting for app: TestAllowInterBucketRecursion_function1 to get deployed
2021/12/17 05:34:45 Waiting for app: TestAllowInterBucketRecursion_function1 to get deployed
2021/12/17 05:34:50 Waiting for app: TestAllowInterBucketRecursion_function1 to get deployed
2021/12/17 05:34:55 Waiting for app: TestAllowInterBucketRecursion_function1 to get deployed
2021/12/17 05:34:55 App: TestAllowInterBucketRecursion_function1 got deployed
2021/12/17 05:34:55 Deploying app: TestAllowInterBucketRecursion_function2. Item count src bucket: 0 dst bucket: 0
2021/12/17 05:34:56 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterBucketRecursion_function2' in metakv",
  "warnings": null
 }
}
2021/12/17 05:34:56 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterBucketRecursion_function2' in metakv",
  "warnings": null
 }
} err 
2021/12/17 05:35:01 Waiting for app: TestAllowInterBucketRecursion_function2 to get deployed
2021/12/17 05:35:06 Waiting for app: TestAllowInterBucketRecursion_function2 to get deployed
2021/12/17 05:35:11 Waiting for app: TestAllowInterBucketRecursion_function2 to get deployed
2021/12/17 05:35:16 Waiting for app: TestAllowInterBucketRecursion_function2 to get deployed
2021/12/17 05:35:16 App: TestAllowInterBucketRecursion_function2 got deployed
2021/12/17 05:35:16 TestAllowInterBucketRecursion_function2 is deployed
2021/12/17 05:35:16 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 05:35:21 Waiting for app: TestAllowInterBucketRecursion_function1 to get un-deployed
2021/12/17 05:35:26 Waiting for app: TestAllowInterBucketRecursion_function1 to get un-deployed
2021/12/17 05:35:31 Waiting for app: TestAllowInterBucketRecursion_function1 to get un-deployed
2021/12/17 05:35:36 Waiting for app: TestAllowInterBucketRecursion_function1 to get un-deployed
2021/12/17 05:35:36 App: TestAllowInterBucketRecursion_function1 got un-deployed
2021/12/17 05:35:36 No eventing-con process running
2021/12/17 05:35:37 Function: TestAllowInterBucketRecursion_function1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAllowInterBucketRecursion_function1 deleting in the background"
}
2021/12/17 05:35:38 Context: Bucket flush, response code: 200 dump: 
2021/12/17 05:35:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 05:35:46 Context: Bucket flush, response code: 200 dump: 
2021/12/17 05:35:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 05:35:51 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 05:35:56 Waiting for app: TestAllowInterBucketRecursion_function2 to get un-deployed
2021/12/17 05:36:01 Waiting for app: TestAllowInterBucketRecursion_function2 to get un-deployed
2021/12/17 05:36:06 Waiting for app: TestAllowInterBucketRecursion_function2 to get un-deployed
2021/12/17 05:36:11 Waiting for app: TestAllowInterBucketRecursion_function2 to get un-deployed
2021/12/17 05:36:11 App: TestAllowInterBucketRecursion_function2 got un-deployed
2021/12/17 05:36:11 No eventing-con process running
2021/12/17 05:36:11 Function: TestAllowInterBucketRecursion_function2 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAllowInterBucketRecursion_function2 deleting in the background"
}
2021/12/17 05:36:12 Context: Bucket flush, response code: 200 dump: 
2021/12/17 05:36:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 05:36:19 Context: Bucket flush, response code: 200 dump: 
2021/12/17 05:36:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAllowInterBucketRecursion (115.30s)
=== RUN   TestInterBucketRecursion2
2021/12/17 05:36:30 Context: Bucket flush, response code: 200 dump: 
2021/12/17 05:36:37 Context: Bucket flush, response code: 200 dump: 
2021/12/17 05:36:43 Context: Bucket flush, response code: 200 dump: 
2021/12/17 05:36:49 Context: Bucket flush, response code: 200 dump: 
2021/12/17 05:36:54 Deploying app: TestInterBucketRecursion2_function1. Item count src bucket: 0 dst bucket: 0
2021/12/17 05:36:55 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterBucketRecursion2_function1' in metakv",
  "warnings": null
 }
}
2021/12/17 05:37:00 Waiting for app: TestInterBucketRecursion2_function1 to get deployed
2021/12/17 05:37:05 Waiting for app: TestInterBucketRecursion2_function1 to get deployed
2021/12/17 05:37:10 Waiting for app: TestInterBucketRecursion2_function1 to get deployed
2021/12/17 05:37:15 Waiting for app: TestInterBucketRecursion2_function1 to get deployed
2021/12/17 05:37:15 App: TestInterBucketRecursion2_function1 got deployed
2021/12/17 05:37:15 Deploying app: TestInterBucketRecursion2_function2. Item count src bucket: 0 dst bucket: 0
2021/12/17 05:37:16 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterBucketRecursion2_function2' in metakv",
  "warnings": null
 }
}
2021/12/17 05:37:21 Waiting for app: TestInterBucketRecursion2_function2 to get deployed
2021/12/17 05:37:26 Waiting for app: TestInterBucketRecursion2_function2 to get deployed
2021/12/17 05:37:31 Waiting for app: TestInterBucketRecursion2_function2 to get deployed
2021/12/17 05:37:36 Waiting for app: TestInterBucketRecursion2_function2 to get deployed
2021/12/17 05:37:36 App: TestInterBucketRecursion2_function2 got deployed
2021/12/17 05:37:37 Deploying app: TestInterBucketRecursion2_function3. Item count src bucket: 0 dst bucket: 0
2021/12/17 05:37:38 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterBucketRecursion2_function3' in metakv",
  "warnings": null
 }
}
2021/12/17 05:37:43 Waiting for app: TestInterBucketRecursion2_function3 to get deployed
2021/12/17 05:37:48 Waiting for app: TestInterBucketRecursion2_function3 to get deployed
2021/12/17 05:37:53 Waiting for app: TestInterBucketRecursion2_function3 to get deployed
2021/12/17 05:37:58 Waiting for app: TestInterBucketRecursion2_function3 to get deployed
2021/12/17 05:37:58 App: TestInterBucketRecursion2_function3 got deployed
2021/12/17 05:37:58 Deploying app: TestInterBucketRecursion2_function4. Item count src bucket: 0 dst bucket: 0
2021/12/17 05:37:58 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_INTER_BUCKET_RECURSION",
 "code": 51,
 "description": "Inter bucket recursion error, deployment of current handler will cause inter bucket recursion",
 "attributes": null,
 "runtime_info": {
  "code": 51,
  "info": "Inter bucket recursion error; function: TestInterBucketRecursion2_function4 causes a cycle involving functions: [TestInterBucketRecursion2_function1 TestInterBucketRecursion2_function2 TestInterBucketRecursion2_function3], hence deployment is disallowed"
 }
}
2021/12/17 05:37:58 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 05:38:03 Waiting for app: TestInterBucketRecursion2_function1 to get un-deployed
2021/12/17 05:38:08 Waiting for app: TestInterBucketRecursion2_function1 to get un-deployed
2021/12/17 05:38:13 Waiting for app: TestInterBucketRecursion2_function1 to get un-deployed
2021/12/17 05:38:18 Waiting for app: TestInterBucketRecursion2_function1 to get un-deployed
2021/12/17 05:38:18 App: TestInterBucketRecursion2_function1 got un-deployed
2021/12/17 05:38:18 No eventing-con process running
2021/12/17 05:38:18 Function: TestInterBucketRecursion2_function1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestInterBucketRecursion2_function1 deleting in the background"
}
2021/12/17 05:38:19 Context: Bucket flush, response code: 200 dump: 
2021/12/17 05:38:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 05:38:26 Context: Bucket flush, response code: 200 dump: 
2021/12/17 05:38:31 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 05:38:31 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 05:38:36 Waiting for app: TestInterBucketRecursion2_function2 to get un-deployed
2021/12/17 05:38:41 Waiting for app: TestInterBucketRecursion2_function2 to get un-deployed
2021/12/17 05:38:46 Waiting for app: TestInterBucketRecursion2_function2 to get un-deployed
2021/12/17 05:38:51 Waiting for app: TestInterBucketRecursion2_function2 to get un-deployed
2021/12/17 05:38:51 App: TestInterBucketRecursion2_function2 got un-deployed
2021/12/17 05:38:51 No eventing-con process running
2021/12/17 05:38:51 Function: TestInterBucketRecursion2_function2 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestInterBucketRecursion2_function2 deleting in the background"
}
2021/12/17 05:38:53 Context: Bucket flush, response code: 200 dump: 
2021/12/17 05:38:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 05:39:00 Context: Bucket flush, response code: 200 dump: 
2021/12/17 05:39:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 05:39:05 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 05:39:10 Waiting for app: TestInterBucketRecursion2_function3 to get un-deployed
2021/12/17 05:39:15 Waiting for app: TestInterBucketRecursion2_function3 to get un-deployed
2021/12/17 05:39:20 Waiting for app: TestInterBucketRecursion2_function3 to get un-deployed
2021/12/17 05:39:25 Waiting for app: TestInterBucketRecursion2_function3 to get un-deployed
2021/12/17 05:39:25 App: TestInterBucketRecursion2_function3 got un-deployed
2021/12/17 05:39:25 No eventing-con process running
2021/12/17 05:39:25 Function: TestInterBucketRecursion2_function3 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestInterBucketRecursion2_function3 deleting in the background"
}
2021/12/17 05:39:27 Context: Bucket flush, response code: 200 dump: 
2021/12/17 05:39:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 05:39:33 Context: Bucket flush, response code: 200 dump: 
2021/12/17 05:39:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 05:39:43 Waiting for app: TestInterBucketRecursion2_function4 to get un-deployed
2021/12/17 05:39:43 App: TestInterBucketRecursion2_function4 got un-deployed
2021/12/17 05:39:43 No eventing-con process running
2021/12/17 05:39:43 Function: TestInterBucketRecursion2_function4 delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestInterBucketRecursion2_function4 not found"
 }
}
2021/12/17 05:39:45 Context: Bucket flush, response code: 200 dump: 
2021/12/17 05:39:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 05:39:52 Context: Bucket flush, response code: 200 dump: 
2021/12/17 05:39:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestInterBucketRecursion2 (214.17s)
=== RUN   TestBucketDeleteWithRebOut
2021/12/17 05:40:03 Removing node(s): [127.0.0.1:19001 127.0.0.1:19002 127.0.0.1:19003] from the cluster
2021/12/17 05:40:08 Rebalance progress: 100

VB distribution across eventing nodes and their workers::

2021/12/17 05:40:11 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:43} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/17 05:40:26 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:43} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/17 05:40:41 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:43} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/17 05:40:57 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:43} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/17 05:41:12 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:43} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/17 05:41:27 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:43} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/17 05:41:42 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:43} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/17 05:41:57 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:43} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/17 05:42:12 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:43} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/17 05:42:27 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:43} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/17 05:42:42 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:43} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/17 05:42:57 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:43} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/17 05:43:12 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:43} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/17 05:43:27 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:43} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/17 05:43:42 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:43} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/17 05:43:57 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:43} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/17 05:44:12 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:43} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/17 05:44:27 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:43} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/17 05:44:42 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:43} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/17 05:44:57 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:43} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/17 05:45:12 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:43} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/17 05:45:27 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:43} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/17 05:45:42 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:43} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/17 05:45:57 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:43} {Name:#primary Bucket:hello-world Completion:100}]

DCP Stream statuses:

vbucket curr owner:


2021/12/17 05:46:12 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 05:46:12 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/17 05:46:12 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/17 05:46:17 Rebalance progress: 100

VB distribution across eventing nodes and their workers::


DCP Stream statuses:

vbucket curr owner:


2021/12/17 05:46:17 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 05:46:17 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/17 05:46:18 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/17 05:46:23 Rebalance progress: 100

VB distribution across eventing nodes and their workers::


DCP Stream statuses:

vbucket curr owner:


2021/12/17 05:46:23 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 05:46:28 clusterSetup.go:397: Waiting for app: TestBucketDeleteWithRebOut to get un-deployed
2021/12/17 05:46:28 clusterSetup.go:405: App: TestBucketDeleteWithRebOut got un-deployed
2021/12/17 05:46:28 eventingSetup.go:46: No eventing-con process running
2021/12/17 05:46:28 eventingSetup.go:418: Function: TestBucketDeleteWithRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestBucketDeleteWithRebOut not found"
 }
}
2021/12/17 05:46:29 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/17 05:46:34 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 05:46:36 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/17 05:46:41 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 05:46:46 eventingSetup.go:125: Deploying app: TestBucketDeleteWithRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/17 05:46:47 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestBucketDeleteWithRebOut' in metakv",
  "warnings": null
 }
}
2021/12/17 05:46:52 clusterSetup.go:346: Waiting for app: TestBucketDeleteWithRebOut to get deployed
2021/12/17 05:46:57 clusterSetup.go:346: Waiting for app: TestBucketDeleteWithRebOut to get deployed
2021/12/17 05:47:02 clusterSetup.go:346: Waiting for app: TestBucketDeleteWithRebOut to get deployed
2021/12/17 05:47:07 clusterSetup.go:346: Waiting for app: TestBucketDeleteWithRebOut to get deployed
2021/12/17 05:47:07 clusterSetup.go:354: App: TestBucketDeleteWithRebOut got deployed

VB distribution across eventing nodes and their workers::


DCP Stream statuses:
	stream status: running
	len: 1024
	vb list dump: [0-1023]

vbucket curr owner:
Producer node: 127.0.0.1:9302	Node UUID: ac9391d01b259f92e8c0183354b22941
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 114
	v: [228-341]
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [0-113]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [114-227]
Producer node: 172.31.2.134:9300	Node UUID: dc4c94f59b06fbc7ba2da352eea789a6
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 113
	v: [911-1023]
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [683-796]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [797-910]
Producer node: 127.0.0.1:9303	Node UUID: 4225b86502e5f2f89c3dbd8b33d0b402
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [342-455]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [456-569]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 113
	v: [570-682]


2021/12/17 05:47:07 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 05:47:07 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/17 05:47:12 Rebalance progress: 29.16666666666666
2021/12/17 05:47:17 Rebalance progress: 29.16666666666666
2021/12/17 05:47:22 Rebalance progress: 100

VB distribution across eventing nodes and their workers::


DCP Stream statuses:
	stream status: running
	len: 1024
	vb list dump: [0-1023]

vbucket curr owner:
Producer node: 127.0.0.1:9302	Node UUID: ac9391d01b259f92e8c0183354b22941
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [0-113]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [114-227]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 114
	v: [228-341]
Producer node: 172.31.2.134:9300	Node UUID: dc4c94f59b06fbc7ba2da352eea789a6
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 113
	v: [911-1023]
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [683-796]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [797-910]
Producer node: 127.0.0.1:9303	Node UUID: 4225b86502e5f2f89c3dbd8b33d0b402
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [342-455]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [456-569]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 113
	v: [570-682]


2021/12/17 05:47:22 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 05:47:22 handler_test.go:1674: Deleting source bucket: default
2021/12/17 05:47:24 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/17 05:47:24 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/17 05:47:29 Rebalance progress: 29.16666666666666
2021/12/17 05:47:34 Rebalance progress: 29.16666666666666
2021/12/17 05:47:39 Rebalance progress: 100

VB distribution across eventing nodes and their workers::


DCP Stream statuses:
	stream status: running
	len: 898
	vb list dump: [0-4, 6, 8-17, 19-53, 55-56, 58-62, 64-81, 84-94, 96, 98-105, 107-109, 111-112, 114-124, 126-131, 133-134, 137-139, 141-145, 147-152, 155-169, 171-187, 190-191, 193-215, 217-219, 221, 223-227, 229, 231-232, 234-242, 244-246, 248, 251-265, 267-277, 280-283, 285-292, 295-298, 300-316, 318-327, 329, 331-335, 337-342, 344-349, 351-368, 370-372, 374-382, 384-388, 390-393, 395-399, 401-403, 405-410, 412-419, 421, 423-431, 434-448, 450-457, 459-461, 463-465, 467, 469, 471, 473-475, 477-478, 480, 482, 484-488, 490-494, 496-501, 503-577, 580-595, 597-599, 601-634, 636-640, 642-650, 652-664, 667-671, 673, 676-683, 685-691, 693-699, 701-706, 708-740, 742-749, 751-754, 756-757, 759-781, 785-791, 793-794, 796-810, 812-813, 815-817, 819-840, 842-843, 845, 847-853, 855-870, 872-883, 886-890, 892-893, 896, 898-903, 905-939, 942-965, 967-971, 973-986, 988-991, 993-994, 996-1013, 1016, 1020-1023]
	stream status: stopped
	len: 126
	vb list dump: [5, 7, 18, 54, 57, 63, 82-83, 95, 97, 106, 110, 113, 125, 132, 135-136, 140, 146, 153-154, 170, 188-189, 192, 216, 220, 222, 228, 230, 233, 243, 247, 249-250, 266, 278-279, 284, 293-294, 299, 317, 328, 330, 336, 343, 350, 369, 373, 383, 389, 394, 400, 404, 411, 420, 422, 432-433, 449, 458, 462, 466, 468, 470, 472, 476, 479, 481, 483, 489, 495, 502, 578-579, 596, 600, 635, 641, 651, 665-666, 672, 674-675, 684, 692, 700, 707, 741, 750, 755, 758, 782-784, 792, 795, 811, 814, 818, 841, 844, 846, 854, 871, 884-885, 891, 894-895, 897, 904, 940-941, 966, 972, 987, 992, 995, 1014-1015, 1017-1019]

vbucket curr owner:
Producer node: 127.0.0.1:9303	Node UUID: 4225b86502e5f2f89c3dbd8b33d0b402
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 101
	v: [570-577, 580-595, 597-599, 601-634, 636-640, 642-650, 652-664, 667-671, 673, 676-682]
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 99
	v: [342, 344-349, 351-368, 370-372, 374-382, 384-388, 390-393, 395-399, 401-403, 405-410, 412-419, 421, 423-431, 434-448, 450-455]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 101
	v: [456-457, 459-461, 463-465, 467, 469, 471, 473-475, 477-478, 480, 482, 484-488, 490-494, 496-501, 503-569]
Producer node: 127.0.0.1:9302	Node UUID: ac9391d01b259f92e8c0183354b22941
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 99
	v: [114-124, 126-131, 133-134, 137-139, 141-145, 147-152, 155-169, 171-187, 190-191, 193-215, 217-219, 221, 223-227]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 96
	v: [229, 231-232, 234-242, 244-246, 248, 251-265, 267-277, 280-283, 285-292, 295-298, 300-316, 318-327, 329, 331-335, 337-341]
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 101
	v: [0-4, 6, 8-17, 19-53, 55-56, 58-62, 64-81, 84-94, 96, 98-105, 107-109, 111-112]
Producer node: 172.31.2.134:9300	Node UUID: dc4c94f59b06fbc7ba2da352eea789a6
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 101
	v: [911-939, 942-965, 967-971, 973-986, 988-991, 993-994, 996-1013, 1016, 1020-1023]
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 101
	v: [683, 685-691, 693-699, 701-706, 708-740, 742-749, 751-754, 756-757, 759-781, 785-791, 793-794, 796]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 99
	v: [797-810, 812-813, 815-817, 819-840, 842-843, 845, 847-853, 855-870, 872-883, 886-890, 892-893, 896, 898-903, 905-910]


2021/12/17 05:47:39 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 05:47:44 eventingSetup.go:719: Waiting for function: TestBucketDeleteWithRebOut status to change to undeployed
2021/12/17 05:47:44 eventingSetup.go:727: Function: TestBucketDeleteWithRebOut status changed to undeployed
2021/12/17 05:47:49 clusterSetup.go:397: Waiting for app: TestBucketDeleteWithRebOut to get un-deployed
2021/12/17 05:47:49 clusterSetup.go:405: App: TestBucketDeleteWithRebOut got un-deployed
2021/12/17 05:47:49 eventingSetup.go:46: No eventing-con process running
2021/12/17 05:47:49 eventingSetup.go:418: Function: TestBucketDeleteWithRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestBucketDeleteWithRebOut deleting in the background"
}
2021/12/17 05:47:51 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/17 05:47:56 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 05:47:58 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/17 05:48:03 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 05:48:03 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv,index,n1ql to the cluster
2021/12/17 05:48:03 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/17 05:48:08 Rebalance progress: 100
--- PASS: TestBucketDeleteWithRebOut (492.56s)
=== RUN   TestN1QLRecursion
2021/12/17 05:48:11 Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/17 05:48:11 Error in adding nodes : , response: ["Prepare join failed. Node is already part of cluster."]
2021/12/17 05:48:11 Removing node(s): [] from the cluster
2021/12/17 05:48:16 Rebalance progress: 100
2021/12/17 05:48:22 Context: Bucket flush, response code: 200 dump: 
2021/12/17 05:48:28 Context: Bucket flush, response code: 200 dump: 
2021/12/17 05:48:34 Context: Bucket flush, response code: 200 dump: 
2021/12/17 05:48:39 Deploying app: TestN1QLRecursion_function1. Item count src bucket: 0 dst bucket: 0
2021/12/17 05:48:40 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLRecursion_function1' in metakv",
  "warnings": null
 }
}
2021/12/17 05:48:40 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLRecursion_function1' in metakv",
  "warnings": null
 }
} err 
2021/12/17 05:48:45 Waiting for app: TestN1QLRecursion_function1 to get deployed
2021/12/17 05:48:50 Waiting for app: TestN1QLRecursion_function1 to get deployed
2021/12/17 05:48:56 Waiting for app: TestN1QLRecursion_function1 to get deployed
2021/12/17 05:49:01 Waiting for app: TestN1QLRecursion_function1 to get deployed
2021/12/17 05:49:01 App: TestN1QLRecursion_function1 got deployed
2021/12/17 05:49:01 TestN1QLRecursion_function1 is deployed
2021/12/17 05:49:01 Deploying app: TestN1QLRecursion_function3. Item count src bucket: 0 dst bucket: 0
2021/12/17 05:49:02 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLRecursion_function3' in metakv",
  "warnings": null
 }
}
2021/12/17 05:49:02 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLRecursion_function3' in metakv",
  "warnings": null
 }
} err 
2021/12/17 05:49:07 Waiting for app: TestN1QLRecursion_function3 to get deployed
2021/12/17 05:49:12 Waiting for app: TestN1QLRecursion_function3 to get deployed
2021/12/17 05:49:17 Waiting for app: TestN1QLRecursion_function3 to get deployed
2021/12/17 05:49:22 Waiting for app: TestN1QLRecursion_function3 to get deployed
2021/12/17 05:49:22 App: TestN1QLRecursion_function3 got deployed
2021/12/17 05:49:22 TestN1QLRecursion_function3 is deployed
2021/12/17 05:49:22 Deploying app: TestN1QLRecursion_function2. Item count src bucket: 0 dst bucket: 0
2021/12/17 05:49:22 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_INTER_BUCKET_RECURSION",
 "code": 51,
 "description": "Inter bucket recursion error, deployment of current handler will cause inter bucket recursion",
 "attributes": null,
 "runtime_info": {
  "code": 51,
  "info": "Inter bucket recursion error; function: TestN1QLRecursion_function2 causes a cycle involving functions: [TestN1QLRecursion_function3 TestN1QLRecursion_function1], hence deployment is disallowed"
 }
}
2021/12/17 05:49:22 response body {
 "name": "ERR_INTER_BUCKET_RECURSION",
 "code": 51,
 "description": "Inter bucket recursion error, deployment of current handler will cause inter bucket recursion",
 "attributes": null,
 "runtime_info": {
  "code": 51,
  "info": "Inter bucket recursion error; function: TestN1QLRecursion_function2 causes a cycle involving functions: [TestN1QLRecursion_function3 TestN1QLRecursion_function1], hence deployment is disallowed"
 }
} err 
2021/12/17 05:49:22 Success: TestN1QLRecursion_function2 is not deployed
2021/12/17 05:49:22 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 05:49:27 Waiting for app: TestN1QLRecursion_function1 to get un-deployed
2021/12/17 05:49:32 Waiting for app: TestN1QLRecursion_function1 to get un-deployed
2021/12/17 05:49:37 Waiting for app: TestN1QLRecursion_function1 to get un-deployed
2021/12/17 05:49:42 Waiting for app: TestN1QLRecursion_function1 to get un-deployed
2021/12/17 05:49:42 App: TestN1QLRecursion_function1 got un-deployed
2021/12/17 05:49:42 No eventing-con process running
2021/12/17 05:49:42 Function: TestN1QLRecursion_function1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLRecursion_function1 deleting in the background"
}
2021/12/17 05:49:44 Context: Bucket flush, response code: 200 dump: 
2021/12/17 05:49:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 05:49:51 Context: Bucket flush, response code: 200 dump: 
2021/12/17 05:49:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 05:49:56 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 05:50:01 Waiting for app: TestN1QLRecursion_function3 to get un-deployed
2021/12/17 05:50:06 Waiting for app: TestN1QLRecursion_function3 to get un-deployed
2021/12/17 05:50:11 Waiting for app: TestN1QLRecursion_function3 to get un-deployed
2021/12/17 05:50:16 Waiting for app: TestN1QLRecursion_function3 to get un-deployed
2021/12/17 05:50:16 App: TestN1QLRecursion_function3 got un-deployed
2021/12/17 05:50:16 No eventing-con process running
2021/12/17 05:50:16 Function: TestN1QLRecursion_function3 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLRecursion_function3 deleting in the background"
}
2021/12/17 05:50:18 Context: Bucket flush, response code: 200 dump: 
2021/12/17 05:50:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 05:50:25 Context: Bucket flush, response code: 200 dump: 
2021/12/17 05:50:30 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 05:50:31 Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/17 05:50:36 Rebalance progress: 100
--- PASS: TestN1QLRecursion (145.54s)
=== RUN   TestN1QLAllowRecursion
2021/12/17 05:50:36 Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/17 05:50:37 Error in adding nodes : , response: ["Prepare join failed. Node is already part of cluster."]
2021/12/17 05:50:37 Removing node(s): [] from the cluster
2021/12/17 05:50:42 Rebalance progress: 100
2021/12/17 05:50:48 Context: Bucket flush, response code: 200 dump: 
2021/12/17 05:50:53 Context: Bucket flush, response code: 200 dump: 
2021/12/17 05:51:00 Context: Bucket flush, response code: 200 dump: 
2021/12/17 05:51:05 Deploying app: TestN1QLAllowRecursion_function1. Item count src bucket: 0 dst bucket: 0
2021/12/17 05:51:06 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function1' in metakv",
  "warnings": null
 }
}
2021/12/17 05:51:06 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function1' in metakv",
  "warnings": null
 }
} err 
2021/12/17 05:51:11 Waiting for app: TestN1QLAllowRecursion_function1 to get deployed
2021/12/17 05:51:16 Waiting for app: TestN1QLAllowRecursion_function1 to get deployed
2021/12/17 05:51:21 Waiting for app: TestN1QLAllowRecursion_function1 to get deployed
2021/12/17 05:51:26 Waiting for app: TestN1QLAllowRecursion_function1 to get deployed
2021/12/17 05:51:26 App: TestN1QLAllowRecursion_function1 got deployed
2021/12/17 05:51:26 TestN1QLAllowRecursion_function1 is deployed
2021/12/17 05:51:26 Deploying app: TestN1QLAllowRecursion_function3. Item count src bucket: 0 dst bucket: 0
2021/12/17 05:51:27 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function3' in metakv",
  "warnings": null
 }
}
2021/12/17 05:51:27 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function3' in metakv",
  "warnings": null
 }
} err 
2021/12/17 05:51:32 Waiting for app: TestN1QLAllowRecursion_function3 to get deployed
2021/12/17 05:51:37 Waiting for app: TestN1QLAllowRecursion_function3 to get deployed
2021/12/17 05:51:42 Waiting for app: TestN1QLAllowRecursion_function3 to get deployed
2021/12/17 05:51:47 Waiting for app: TestN1QLAllowRecursion_function3 to get deployed
2021/12/17 05:51:47 App: TestN1QLAllowRecursion_function3 got deployed
2021/12/17 05:51:47 TestN1QLAllowRecursion_function3 is deployed
2021/12/17 05:51:47 Deploying app: TestN1QLAllowRecursion_function2. Item count src bucket: 0 dst bucket: 0
2021/12/17 05:51:48 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function2' in metakv",
  "warnings": null
 }
}
2021/12/17 05:51:48 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function2' in metakv",
  "warnings": null
 }
} err 
2021/12/17 05:51:53 Waiting for app: TestN1QLAllowRecursion_function2 to get deployed
2021/12/17 05:51:58 Waiting for app: TestN1QLAllowRecursion_function2 to get deployed
2021/12/17 05:52:03 Waiting for app: TestN1QLAllowRecursion_function2 to get deployed
2021/12/17 05:52:08 Waiting for app: TestN1QLAllowRecursion_function2 to get deployed
2021/12/17 05:52:08 App: TestN1QLAllowRecursion_function2 got deployed
2021/12/17 05:52:08 Success: TestN1QLAllowRecursion_function2 is deployed after allow_interbucket_recursion
2021/12/17 05:52:08 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 05:52:13 Waiting for app: TestN1QLAllowRecursion_function1 to get un-deployed
2021/12/17 05:52:18 Waiting for app: TestN1QLAllowRecursion_function1 to get un-deployed
2021/12/17 05:52:23 Waiting for app: TestN1QLAllowRecursion_function1 to get un-deployed
2021/12/17 05:52:28 Waiting for app: TestN1QLAllowRecursion_function1 to get un-deployed
2021/12/17 05:52:28 App: TestN1QLAllowRecursion_function1 got un-deployed
2021/12/17 05:52:29 No eventing-con process running
2021/12/17 05:52:29 Function: TestN1QLAllowRecursion_function1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLAllowRecursion_function1 deleting in the background"
}
2021/12/17 05:52:30 Context: Bucket flush, response code: 200 dump: 
2021/12/17 05:52:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 05:52:37 Context: Bucket flush, response code: 200 dump: 
2021/12/17 05:52:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 05:52:42 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 05:52:47 Waiting for app: TestN1QLAllowRecursion_function2 to get un-deployed
2021/12/17 05:52:52 Waiting for app: TestN1QLAllowRecursion_function2 to get un-deployed
2021/12/17 05:52:57 Waiting for app: TestN1QLAllowRecursion_function2 to get un-deployed
2021/12/17 05:53:02 Waiting for app: TestN1QLAllowRecursion_function2 to get un-deployed
2021/12/17 05:53:02 App: TestN1QLAllowRecursion_function2 got un-deployed
2021/12/17 05:53:02 No eventing-con process running
2021/12/17 05:53:02 Function: TestN1QLAllowRecursion_function2 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLAllowRecursion_function2 deleting in the background"
}
2021/12/17 05:53:04 Context: Bucket flush, response code: 200 dump: 
2021/12/17 05:53:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 05:53:11 Context: Bucket flush, response code: 200 dump: 
2021/12/17 05:53:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 05:53:16 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 05:53:21 Waiting for app: TestN1QLAllowRecursion_function3 to get un-deployed
2021/12/17 05:53:26 Waiting for app: TestN1QLAllowRecursion_function3 to get un-deployed
2021/12/17 05:53:31 Waiting for app: TestN1QLAllowRecursion_function3 to get un-deployed
2021/12/17 05:53:36 Waiting for app: TestN1QLAllowRecursion_function3 to get un-deployed
2021/12/17 05:53:36 App: TestN1QLAllowRecursion_function3 got un-deployed
2021/12/17 05:53:36 No eventing-con process running
2021/12/17 05:53:36 Function: TestN1QLAllowRecursion_function3 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLAllowRecursion_function3 deleting in the background"
}
2021/12/17 05:53:38 Context: Bucket flush, response code: 200 dump: 
2021/12/17 05:53:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 05:53:45 Context: Bucket flush, response code: 200 dump: 
2021/12/17 05:53:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 05:53:52 Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/17 05:53:57 Rebalance progress: 100
--- PASS: TestN1QLAllowRecursion (200.23s)
=== RUN   TestOnDeleteExpiryBucketOp
2021/12/17 05:53:57 Sleeping for some time
2021/12/17 05:54:07 Waiting for app: TestOnDeleteExpiryBucketOp to get un-deployed
2021/12/17 05:54:07 App: TestOnDeleteExpiryBucketOp got un-deployed
2021/12/17 05:54:07 No eventing-con process running
2021/12/17 05:54:07 Function: TestOnDeleteExpiryBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnDeleteExpiryBucketOp not found"
 }
}
2021/12/17 05:54:08 Context: Bucket flush, response code: 200 dump: 
2021/12/17 05:54:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 05:54:15 Context: Bucket flush, response code: 200 dump: 
2021/12/17 05:54:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 05:54:20 Deploying app: TestOnDeleteExpiryBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/17 05:54:21 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnDeleteExpiryBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/17 05:54:21 Starting bucket ops to source bucket
2021/12/17 05:55:11 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 05:55:16 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 05:55:16 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/17 05:55:16 Starting bucket ops to source bucket
2021/12/17 05:55:17 Starting bucket ops to source bucket
2021/12/17 05:56:23 Waiting for dst bucket item count to get to: 7000 curr count: 5000
2021/12/17 05:56:28 Waiting for dst bucket item count to get to: 7000 curr count: 6443
2021/12/17 05:56:33 Waiting for dst bucket item count to get to: 7000 curr count: 6443
2021/12/17 05:56:33 src & dst bucket item count matched up. src bucket count: 7000 dst bucket count: 7000
2021/12/17 05:56:33 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 05:56:33 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 05:56:38 Waiting for app: TestOnDeleteExpiryBucketOp to get un-deployed
2021/12/17 05:56:43 Waiting for app: TestOnDeleteExpiryBucketOp to get un-deployed
2021/12/17 05:56:48 Waiting for app: TestOnDeleteExpiryBucketOp to get un-deployed
2021/12/17 05:56:53 Waiting for app: TestOnDeleteExpiryBucketOp to get un-deployed
2021/12/17 05:56:53 App: TestOnDeleteExpiryBucketOp got un-deployed
2021/12/17 05:56:53 No eventing-con process running
2021/12/17 05:56:53 Function: TestOnDeleteExpiryBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnDeleteExpiryBucketOp deleting in the background"
}
2021/12/17 05:56:55 Context: Bucket flush, response code: 200 dump: 
2021/12/17 05:57:05 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/17 05:57:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 05:57:06 Context: Bucket flush, response code: 200 dump: 
2021/12/17 05:57:16 Waiting for dst bucket item count to get to: 0 curr count: 7000
2021/12/17 05:57:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnDeleteExpiryBucketOp (199.85s)
=== RUN   TestTimerOverWriteSameReference
2021/12/17 05:57:21 Waiting for app: TestTimerOverWriteSameReference to get un-deployed
2021/12/17 05:57:22 App: TestTimerOverWriteSameReference got un-deployed
2021/12/17 05:57:22 No eventing-con process running
2021/12/17 05:57:22 Function: TestTimerOverWriteSameReference delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestTimerOverWriteSameReference not found"
 }
}
2021/12/17 05:57:23 Context: Bucket flush, response code: 200 dump: 
2021/12/17 05:57:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 05:57:30 Context: Bucket flush, response code: 200 dump: 
2021/12/17 05:57:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 05:57:35 Deploying app: TestTimerOverWriteSameReference. Item count src bucket: 0 dst bucket: 0
2021/12/17 05:57:36 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestTimerOverWriteSameReference' in metakv",
  "warnings": null
 }
}
2021/12/17 05:57:41 Waiting for app: TestTimerOverWriteSameReference to get deployed
2021/12/17 05:57:46 Waiting for app: TestTimerOverWriteSameReference to get deployed
2021/12/17 05:57:51 Waiting for app: TestTimerOverWriteSameReference to get deployed
2021/12/17 05:57:56 Waiting for app: TestTimerOverWriteSameReference to get deployed
2021/12/17 05:57:56 App: TestTimerOverWriteSameReference got deployed
2021/12/17 05:58:06 Starting bucket ops to source bucket
2021/12/17 05:59:08 src & dst bucket item count matched up. src bucket count: 1 dst bucket count: 1
2021/12/17 05:59:18 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 05:59:18 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 05:59:23 Waiting for app: TestTimerOverWriteSameReference to get un-deployed
2021/12/17 05:59:28 Waiting for app: TestTimerOverWriteSameReference to get un-deployed
2021/12/17 05:59:33 Waiting for app: TestTimerOverWriteSameReference to get un-deployed
2021/12/17 05:59:38 Waiting for app: TestTimerOverWriteSameReference to get un-deployed
2021/12/17 05:59:38 App: TestTimerOverWriteSameReference got un-deployed
2021/12/17 05:59:38 No eventing-con process running
2021/12/17 05:59:38 Function: TestTimerOverWriteSameReference delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestTimerOverWriteSameReference deleting in the background"
}
2021/12/17 05:59:39 Context: Bucket flush, response code: 200 dump: 
2021/12/17 05:59:49 Waiting for dst bucket item count to get to: 0 curr count: 500
2021/12/17 05:59:54 Waiting for dst bucket item count to get to: 0 curr count: 500
2021/12/17 05:59:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 05:59:56 Context: Bucket flush, response code: 200 dump: 
2021/12/17 06:00:06 Waiting for dst bucket item count to get to: 0 curr count: 1
2021/12/17 06:00:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestTimerOverWriteSameReference (169.86s)
=== RUN   TestCancelTimerBucketop
2021/12/17 06:00:11 Waiting for app: TestCancelTimerBucketop to get un-deployed
2021/12/17 06:00:11 App: TestCancelTimerBucketop got un-deployed
2021/12/17 06:00:11 No eventing-con process running
2021/12/17 06:00:11 Function: TestCancelTimerBucketop delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestCancelTimerBucketop not found"
 }
}
2021/12/17 06:00:13 Context: Bucket flush, response code: 200 dump: 
2021/12/17 06:00:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 06:00:20 Context: Bucket flush, response code: 200 dump: 
2021/12/17 06:00:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 06:00:25 Deploying app: TestCancelTimerBucketop. Item count src bucket: 0 dst bucket: 0
2021/12/17 06:00:26 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCancelTimerBucketop' in metakv",
  "warnings": null
 }
}
2021/12/17 06:00:31 Waiting for app: TestCancelTimerBucketop to get deployed
2021/12/17 06:00:36 Waiting for app: TestCancelTimerBucketop to get deployed
2021/12/17 06:00:41 Waiting for app: TestCancelTimerBucketop to get deployed
2021/12/17 06:00:46 Waiting for app: TestCancelTimerBucketop to get deployed
2021/12/17 06:00:46 App: TestCancelTimerBucketop got deployed
2021/12/17 06:00:46 Starting bucket ops to source bucket
2021/12/17 06:00:53 Starting bucket ops to source bucket
2021/12/17 06:01:56 src & dst bucket item count matched up. src bucket count: 1000 dst bucket count: 1000
2021/12/17 06:01:56 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 06:01:56 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 06:02:01 Waiting for app: TestCancelTimerBucketop to get un-deployed
2021/12/17 06:02:06 Waiting for app: TestCancelTimerBucketop to get un-deployed
2021/12/17 06:02:11 Waiting for app: TestCancelTimerBucketop to get un-deployed
2021/12/17 06:02:16 Waiting for app: TestCancelTimerBucketop to get un-deployed
2021/12/17 06:02:16 App: TestCancelTimerBucketop got un-deployed
2021/12/17 06:02:16 No eventing-con process running
2021/12/17 06:02:16 Function: TestCancelTimerBucketop delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCancelTimerBucketop deleting in the background"
}
2021/12/17 06:02:18 Context: Bucket flush, response code: 200 dump: 
2021/12/17 06:02:28 Waiting for dst bucket item count to get to: 0 curr count: 591
2021/12/17 06:02:33 Waiting for dst bucket item count to get to: 0 curr count: 91
2021/12/17 06:02:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 06:02:35 Context: Bucket flush, response code: 200 dump: 
2021/12/17 06:02:45 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/17 06:02:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCancelTimerBucketop (158.25s)
=== RUN   TestJSExpiryDate
2021/12/17 06:02:50 Starting bucket ops to source bucket
2021/12/17 06:03:03 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/17 06:03:03 Deploying app: TestJSExpiryDate. Item count src bucket: 5000 dst bucket: 0
2021/12/17 06:03:04 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestJSExpiryDate' in metakv",
  "warnings": null
 }
}
2021/12/17 06:03:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 06:03:34 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 06:03:34 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 06:03:39 Waiting for app: TestJSExpiryDate to get un-deployed
2021/12/17 06:03:44 Waiting for app: TestJSExpiryDate to get un-deployed
2021/12/17 06:03:49 Waiting for app: TestJSExpiryDate to get un-deployed
2021/12/17 06:03:54 Waiting for app: TestJSExpiryDate to get un-deployed
2021/12/17 06:03:54 App: TestJSExpiryDate got un-deployed
2021/12/17 06:03:54 No eventing-con process running
2021/12/17 06:03:54 Function: TestJSExpiryDate delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestJSExpiryDate deleting in the background"
}
2021/12/17 06:03:56 Context: Bucket flush, response code: 200 dump: 
2021/12/17 06:04:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 06:04:03 Context: Bucket flush, response code: 200 dump: 
2021/12/17 06:04:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestJSExpiryDate (83.80s)
=== RUN   TestBinaryDoc
2021/12/17 06:04:13 Starting bucket ops to source bucket
2021/12/17 06:04:14 Deploying app: TestBinaryDoc. Item count src bucket: 0 dst bucket: 0
2021/12/17 06:04:15 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestBinaryDoc' in metakv",
  "warnings": null
 }
}
2021/12/17 06:04:15 Starting bucket ops to source bucket
2021/12/17 06:04:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 06:04:15 Deleting source bucket docs now
2021/12/17 06:04:15 Starting bucket ops to source bucket
2021/12/17 06:04:52 Waiting for dst bucket item count to get to: 32 curr count: 23
2021/12/17 06:04:57 Waiting for dst bucket item count to get to: 32 curr count: 28
2021/12/17 06:04:57 src & dst bucket item count matched up. src bucket count: 32 dst bucket count: 32
2021/12/17 06:04:57 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 06:04:57 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 06:05:02 Waiting for app: TestBinaryDoc to get un-deployed
2021/12/17 06:05:07 Waiting for app: TestBinaryDoc to get un-deployed
2021/12/17 06:05:12 Waiting for app: TestBinaryDoc to get un-deployed
2021/12/17 06:05:17 Waiting for app: TestBinaryDoc to get un-deployed
2021/12/17 06:05:17 App: TestBinaryDoc got un-deployed
2021/12/17 06:05:17 No eventing-con process running
2021/12/17 06:05:17 Function: TestBinaryDoc delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestBinaryDoc deleting in the background"
}
2021/12/17 06:05:19 Context: Bucket flush, response code: 200 dump: 
2021/12/17 06:05:29 Waiting for dst bucket item count to get to: 0 curr count: 32
2021/12/17 06:05:34 Waiting for dst bucket item count to get to: 0 curr count: 14
2021/12/17 06:05:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 06:05:36 Context: Bucket flush, response code: 200 dump: 
2021/12/17 06:05:46 Waiting for dst bucket item count to get to: 0 curr count: 32
2021/12/17 06:05:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestBinaryDoc (97.32s)
=== RUN   TestConstantBindings
2021/12/17 06:05:46 Deploying app: TestConstantBindings. Item count src bucket: 0 dst bucket: 0
2021/12/17 06:05:47 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestConstantBindings' in metakv",
  "warnings": null
 }
}
2021/12/17 06:05:47 Starting bucket ops to source bucket
2021/12/17 06:05:53 Waiting for dst bucket item count to get to: 6000 curr count: 0
2021/12/17 06:05:58 Waiting for dst bucket item count to get to: 6000 curr count: 0
2021/12/17 06:06:03 Waiting for dst bucket item count to get to: 6000 curr count: 0
2021/12/17 06:06:08 Waiting for dst bucket item count to get to: 6000 curr count: 823
2021/12/17 06:06:13 Waiting for dst bucket item count to get to: 6000 curr count: 823
2021/12/17 06:06:13 src & dst bucket item count matched up. src bucket count: 6000 dst bucket count: 6000
2021/12/17 06:06:13 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 06:06:13 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 06:06:18 Waiting for app: TestConstantBindings to get un-deployed
2021/12/17 06:06:23 Waiting for app: TestConstantBindings to get un-deployed
2021/12/17 06:06:28 Waiting for app: TestConstantBindings to get un-deployed
2021/12/17 06:06:33 Waiting for app: TestConstantBindings to get un-deployed
2021/12/17 06:06:33 App: TestConstantBindings got un-deployed
2021/12/17 06:06:33 No eventing-con process running
2021/12/17 06:06:33 Function: TestConstantBindings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestConstantBindings deleting in the background"
}
2021/12/17 06:06:35 Context: Bucket flush, response code: 200 dump: 
2021/12/17 06:06:45 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/17 06:06:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 06:06:46 Context: Bucket flush, response code: 200 dump: 
2021/12/17 06:06:56 Waiting for dst bucket item count to get to: 0 curr count: 6000
2021/12/17 06:06:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestConstantBindings (70.72s)
=== RUN   TestCodeInsight
2021/12/17 06:07:01 Waiting for app: TestCodeInsight to get un-deployed
2021/12/17 06:07:01 App: TestCodeInsight got un-deployed
2021/12/17 06:07:01 No eventing-con process running
2021/12/17 06:07:01 Function: TestCodeInsight delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestCodeInsight not found"
 }
}
2021/12/17 06:07:03 Context: Bucket flush, response code: 200 dump: 
2021/12/17 06:07:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 06:07:10 Context: Bucket flush, response code: 200 dump: 
2021/12/17 06:07:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 06:07:15 Starting bucket ops to source bucket
2021/12/17 06:07:16 Deploying app: TestCodeInsight. Item count src bucket: 0 dst bucket: 0
2021/12/17 06:07:17 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCodeInsight' in metakv",
  "warnings": null
 }
}
2021/12/17 06:07:22 Waiting for app: TestCodeInsight to get deployed
2021/12/17 06:07:27 Waiting for app: TestCodeInsight to get deployed
2021/12/17 06:07:32 Waiting for app: TestCodeInsight to get deployed
2021/12/17 06:07:37 Waiting for app: TestCodeInsight to get deployed
2021/12/17 06:07:37 App: TestCodeInsight got deployed
2021/12/17 06:07:42 Waiting for dst bucket item count to get to: 1000 curr count: 285
2021/12/17 06:07:42 src & dst bucket item count matched up. src bucket count: 1000 dst bucket count: 1000
2021/12/17 06:07:42 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 06:07:47 Waiting for app: TestCodeInsight to get un-deployed
2021/12/17 06:07:52 Waiting for app: TestCodeInsight to get un-deployed
2021/12/17 06:07:57 Waiting for app: TestCodeInsight to get un-deployed
2021/12/17 06:08:02 Waiting for app: TestCodeInsight to get un-deployed
2021/12/17 06:08:02 App: TestCodeInsight got un-deployed
2021/12/17 06:08:02 No eventing-con process running
2021/12/17 06:08:02 Function: TestCodeInsight delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCodeInsight deleting in the background"
}
2021/12/17 06:08:04 Context: Bucket flush, response code: 200 dump: 
2021/12/17 06:08:14 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/17 06:08:19 Waiting for dst bucket item count to get to: 0 curr count: 500
2021/12/17 06:08:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 06:08:21 Context: Bucket flush, response code: 200 dump: 
2021/12/17 06:08:31 Waiting for dst bucket item count to get to: 0 curr count: 934
2021/12/17 06:08:36 Waiting for dst bucket item count to get to: 0 curr count: 934
2021/12/17 06:08:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCodeInsight (99.46s)
=== RUN   TestParserTransform
--- PASS: TestParserTransform (0.05s)
=== RUN   TestParserDetect
--- PASS: TestParserDetect (0.00s)
PASS
ok  	github.com/couchbase/eventing/tests/functional_tests	13200.445s

Functional tests took 3 hours, 40 minutes and 4 seconds.

N1ql tests

2021/12/17 06:08:46 Started n1ql
2021/12/17 06:08:56 Adding node: https://127.0.0.1:19001 with role: kv,index,n1ql to the cluster
2021/12/17 06:08:56 Error in adding nodes : , response: ["Prepare join failed. Node is already part of cluster."]
2021/12/17 06:08:56 Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/17 06:08:56 Removing node(s): [] from the cluster
2021/12/17 06:09:01 Rebalance progress: 100
=== RUN   TestRecursiveMutationN1QL
2021/12/17 06:09:06 Waiting for app: TestRecursiveMutationN1QL to get un-deployed
2021/12/17 06:09:06 App: TestRecursiveMutationN1QL got un-deployed
2021/12/17 06:09:06 No eventing-con process running
2021/12/17 06:09:06 Function: TestRecursiveMutationN1QL delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestRecursiveMutationN1QL not found"
 }
}
2021/12/17 06:09:08 Context: Bucket flush, response code: 200 dump: 
2021/12/17 06:09:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 06:09:15 Context: Bucket flush, response code: 200 dump: 
2021/12/17 06:09:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 06:09:20 Deploying app: TestRecursiveMutationN1QL. Item count src bucket: 0 dst bucket: 0
2021/12/17 06:09:20 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_INTER_BUCKET_RECURSION",
 "code": 51,
 "description": "Inter bucket recursion error, deployment of current handler will cause inter bucket recursion",
 "attributes": null,
 "runtime_info": {
  "code": 51,
  "info": "Function: TestRecursiveMutationN1QL N1QL DML to source keyspace default:default"
 }
}
--- PASS: TestRecursiveMutationN1QL (18.66s)
=== RUN   TestFlexReset1
2021/12/17 06:09:25 Waiting for app: TestFlexReset1 to get un-deployed
2021/12/17 06:09:25 App: TestFlexReset1 got un-deployed
2021/12/17 06:09:25 No eventing-con process running
2021/12/17 06:09:25 Function: TestFlexReset1 delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestFlexReset1 not found"
 }
}
2021/12/17 06:09:27 Context: Bucket flush, response code: 200 dump: 
2021/12/17 06:09:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 06:09:33 Context: Bucket flush, response code: 200 dump: 
2021/12/17 06:09:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 06:09:38 Starting bucket ops to source bucket
2021/12/17 06:09:39 Deploying app: TestFlexReset1. Item count src bucket: 0 dst bucket: 0
2021/12/17 06:09:40 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestFlexReset1' in metakv",
  "warnings": null
 }
}
2021/12/17 06:09:45 Waiting for app: TestFlexReset1 to get deployed
2021/12/17 06:09:50 Waiting for app: TestFlexReset1 to get deployed
2021/12/17 06:09:55 Waiting for app: TestFlexReset1 to get deployed
2021/12/17 06:10:00 Waiting for app: TestFlexReset1 to get deployed
2021/12/17 06:10:00 App: TestFlexReset1 got deployed
2021/12/17 06:10:05 Waiting for dst bucket item count to get to: 1000 curr count: 500
2021/12/17 06:10:05 src & dst bucket item count matched up. src bucket count: 1000 dst bucket count: 1000
2021/12/17 06:10:05 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 06:10:05 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 06:10:10 Waiting for app: TestFlexReset1 to get un-deployed
2021/12/17 06:10:15 Waiting for app: TestFlexReset1 to get un-deployed
2021/12/17 06:10:20 Waiting for app: TestFlexReset1 to get un-deployed
2021/12/17 06:10:25 Waiting for app: TestFlexReset1 to get un-deployed
2021/12/17 06:10:25 App: TestFlexReset1 got un-deployed
2021/12/17 06:10:25 No eventing-con process running
2021/12/17 06:10:26 Function: TestFlexReset1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestFlexReset1 deleting in the background"
}
2021/12/17 06:10:27 Context: Bucket flush, response code: 200 dump: 
2021/12/17 06:10:37 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/17 06:10:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 06:10:39 Context: Bucket flush, response code: 200 dump: 
2021/12/17 06:10:49 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/17 06:10:54 Waiting for dst bucket item count to get to: 0 curr count: 500
2021/12/17 06:10:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestFlexReset1 (94.24s)
=== RUN   TestFlexReset2
2021/12/17 06:10:59 Waiting for app: TestFlexReset2 to get un-deployed
2021/12/17 06:10:59 App: TestFlexReset2 got un-deployed
2021/12/17 06:10:59 No eventing-con process running
2021/12/17 06:10:59 Function: TestFlexReset2 delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestFlexReset2 not found"
 }
}
2021/12/17 06:11:01 Context: Bucket flush, response code: 200 dump: 
2021/12/17 06:11:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 06:11:08 Context: Bucket flush, response code: 200 dump: 
2021/12/17 06:11:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 06:11:13 Starting bucket ops to source bucket
2021/12/17 06:11:13 Deploying app: TestFlexReset2. Item count src bucket: 0 dst bucket: 0
2021/12/17 06:11:14 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestFlexReset2' in metakv",
  "warnings": null
 }
}
2021/12/17 06:11:19 Waiting for app: TestFlexReset2 to get deployed
2021/12/17 06:11:24 Waiting for app: TestFlexReset2 to get deployed
2021/12/17 06:11:29 Waiting for app: TestFlexReset2 to get deployed
2021/12/17 06:11:34 Waiting for app: TestFlexReset2 to get deployed
2021/12/17 06:11:34 App: TestFlexReset2 got deployed
2021/12/17 06:11:39 Waiting for dst bucket item count to get to: 1000 curr count: 607
2021/12/17 06:11:39 src & dst bucket item count matched up. src bucket count: 1000 dst bucket count: 1000
2021/12/17 06:11:39 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 06:11:40 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 06:11:45 Waiting for app: TestFlexReset2 to get un-deployed
2021/12/17 06:11:50 Waiting for app: TestFlexReset2 to get un-deployed
2021/12/17 06:11:55 Waiting for app: TestFlexReset2 to get un-deployed
2021/12/17 06:12:00 Waiting for app: TestFlexReset2 to get un-deployed
2021/12/17 06:12:00 App: TestFlexReset2 got un-deployed
2021/12/17 06:12:00 No eventing-con process running
2021/12/17 06:12:00 Function: TestFlexReset2 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestFlexReset2 deleting in the background"
}
2021/12/17 06:12:01 Context: Bucket flush, response code: 200 dump: 
2021/12/17 06:12:11 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/17 06:12:16 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/17 06:12:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 06:12:18 Context: Bucket flush, response code: 200 dump: 
2021/12/17 06:12:28 Waiting for dst bucket item count to get to: 0 curr count: 798
2021/12/17 06:12:33 Waiting for dst bucket item count to get to: 0 curr count: 298
2021/12/17 06:12:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestFlexReset2 (99.23s)
=== RUN   TestN1QLLabelledBreak
2021/12/17 06:12:38 Waiting for app: TestN1QLLabelledBreak to get un-deployed
2021/12/17 06:12:38 App: TestN1QLLabelledBreak got un-deployed
2021/12/17 06:12:38 No eventing-con process running
2021/12/17 06:12:38 Function: TestN1QLLabelledBreak delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLLabelledBreak not found"
 }
}
2021/12/17 06:12:40 Context: Bucket flush, response code: 200 dump: 
2021/12/17 06:12:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 06:12:47 Context: Bucket flush, response code: 200 dump: 
2021/12/17 06:12:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 06:12:52 Deploying app: TestN1QLLabelledBreak. Item count src bucket: 0 dst bucket: 0
2021/12/17 06:12:53 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLLabelledBreak' in metakv",
  "warnings": null
 }
}
2021/12/17 06:12:58 Waiting for app: TestN1QLLabelledBreak to get deployed
2021/12/17 06:13:03 Waiting for app: TestN1QLLabelledBreak to get deployed
2021/12/17 06:13:08 Waiting for app: TestN1QLLabelledBreak to get deployed
2021/12/17 06:13:13 Waiting for app: TestN1QLLabelledBreak to get deployed
2021/12/17 06:13:13 App: TestN1QLLabelledBreak got deployed
2021/12/17 06:13:13 Starting bucket ops to source bucket
2021/12/17 06:13:23 Waiting for dst bucket item count to get to: 10000 curr count: 0
2021/12/17 06:13:28 Waiting for dst bucket item count to get to: 10000 curr count: 616
2021/12/17 06:13:33 Waiting for dst bucket item count to get to: 10000 curr count: 898
2021/12/17 06:13:38 Waiting for dst bucket item count to get to: 10000 curr count: 2868
2021/12/17 06:13:43 Waiting for dst bucket item count to get to: 10000 curr count: 4960
2021/12/17 06:13:48 Waiting for dst bucket item count to get to: 10000 curr count: 7011
2021/12/17 06:13:53 Waiting for dst bucket item count to get to: 10000 curr count: 9026
2021/12/17 06:13:58 Waiting for dst bucket item count to get to: 10000 curr count: 9723
2021/12/17 06:13:58 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/17 06:13:58 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 06:13:58 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 06:14:03 Waiting for app: TestN1QLLabelledBreak to get un-deployed
2021/12/17 06:14:08 Waiting for app: TestN1QLLabelledBreak to get un-deployed
2021/12/17 06:14:13 Waiting for app: TestN1QLLabelledBreak to get un-deployed
2021/12/17 06:14:18 Waiting for app: TestN1QLLabelledBreak to get un-deployed
2021/12/17 06:14:18 App: TestN1QLLabelledBreak got un-deployed
2021/12/17 06:14:18 No eventing-con process running
2021/12/17 06:14:18 Function: TestN1QLLabelledBreak delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLLabelledBreak deleting in the background"
}
2021/12/17 06:14:20 Context: Bucket flush, response code: 200 dump: 
2021/12/17 06:14:30 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/17 06:14:35 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/17 06:14:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 06:14:37 Context: Bucket flush, response code: 200 dump: 
2021/12/17 06:14:47 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/17 06:14:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLLabelledBreak (133.28s)
=== RUN   TestN1QLUnlabelledBreak
2021/12/17 06:14:52 Waiting for app: TestN1QLUnlabelledBreak to get un-deployed
2021/12/17 06:14:52 App: TestN1QLUnlabelledBreak got un-deployed
2021/12/17 06:14:52 No eventing-con process running
2021/12/17 06:14:52 Function: TestN1QLUnlabelledBreak delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLUnlabelledBreak not found"
 }
}
2021/12/17 06:14:53 Context: Bucket flush, response code: 200 dump: 
2021/12/17 06:14:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 06:15:00 Context: Bucket flush, response code: 200 dump: 
2021/12/17 06:15:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 06:15:05 Deploying app: TestN1QLUnlabelledBreak. Item count src bucket: 0 dst bucket: 0
2021/12/17 06:15:06 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLUnlabelledBreak' in metakv",
  "warnings": null
 }
}
2021/12/17 06:15:11 Waiting for app: TestN1QLUnlabelledBreak to get deployed
2021/12/17 06:15:16 Waiting for app: TestN1QLUnlabelledBreak to get deployed
2021/12/17 06:15:21 Waiting for app: TestN1QLUnlabelledBreak to get deployed
2021/12/17 06:15:26 Waiting for app: TestN1QLUnlabelledBreak to get deployed
2021/12/17 06:15:26 App: TestN1QLUnlabelledBreak got deployed
2021/12/17 06:15:26 Starting bucket ops to source bucket
2021/12/17 06:15:36 Waiting for dst bucket item count to get to: 10000 curr count: 385
2021/12/17 06:15:41 Waiting for dst bucket item count to get to: 10000 curr count: 385
2021/12/17 06:15:46 Waiting for dst bucket item count to get to: 10000 curr count: 385
2021/12/17 06:15:51 Waiting for dst bucket item count to get to: 10000 curr count: 5165
2021/12/17 06:15:56 Waiting for dst bucket item count to get to: 10000 curr count: 5165
2021/12/17 06:15:56 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/17 06:15:56 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 06:15:56 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 06:16:01 Waiting for app: TestN1QLUnlabelledBreak to get un-deployed
2021/12/17 06:16:06 Waiting for app: TestN1QLUnlabelledBreak to get un-deployed
2021/12/17 06:16:11 Waiting for app: TestN1QLUnlabelledBreak to get un-deployed
2021/12/17 06:16:16 Waiting for app: TestN1QLUnlabelledBreak to get un-deployed
2021/12/17 06:16:16 App: TestN1QLUnlabelledBreak got un-deployed
2021/12/17 06:16:16 No eventing-con process running
2021/12/17 06:16:16 Function: TestN1QLUnlabelledBreak delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLUnlabelledBreak deleting in the background"
}
2021/12/17 06:16:18 Context: Bucket flush, response code: 200 dump: 
2021/12/17 06:16:28 Waiting for dst bucket item count to get to: 0 curr count: 3483
2021/12/17 06:16:33 Waiting for dst bucket item count to get to: 0 curr count: 983
2021/12/17 06:16:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 06:16:35 Context: Bucket flush, response code: 200 dump: 
2021/12/17 06:16:45 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/17 06:16:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLUnlabelledBreak (118.36s)
=== RUN   TestN1QLThrowStatement
2021/12/17 06:16:50 Waiting for app: TestN1QLThrowStatement to get un-deployed
2021/12/17 06:16:50 App: TestN1QLThrowStatement got un-deployed
2021/12/17 06:16:50 No eventing-con process running
2021/12/17 06:16:50 Function: TestN1QLThrowStatement delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLThrowStatement not found"
 }
}
2021/12/17 06:16:52 Context: Bucket flush, response code: 200 dump: 
2021/12/17 06:16:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 06:16:58 Context: Bucket flush, response code: 200 dump: 
2021/12/17 06:17:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 06:17:04 Deploying app: TestN1QLThrowStatement. Item count src bucket: 0 dst bucket: 0
2021/12/17 06:17:05 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLThrowStatement' in metakv",
  "warnings": null
 }
}
2021/12/17 06:17:10 Waiting for app: TestN1QLThrowStatement to get deployed
2021/12/17 06:17:15 Waiting for app: TestN1QLThrowStatement to get deployed
2021/12/17 06:17:20 Waiting for app: TestN1QLThrowStatement to get deployed
2021/12/17 06:17:25 Waiting for app: TestN1QLThrowStatement to get deployed
2021/12/17 06:17:25 App: TestN1QLThrowStatement got deployed
2021/12/17 06:17:25 Starting bucket ops to source bucket
2021/12/17 06:17:34 Waiting for dst bucket item count to get to: 10000 curr count: 0
2021/12/17 06:17:39 Waiting for dst bucket item count to get to: 10000 curr count: 174
2021/12/17 06:17:44 Waiting for dst bucket item count to get to: 10000 curr count: 530
2021/12/17 06:17:49 Waiting for dst bucket item count to get to: 10000 curr count: 2305
2021/12/17 06:17:54 Waiting for dst bucket item count to get to: 10000 curr count: 4328
2021/12/17 06:17:59 Waiting for dst bucket item count to get to: 10000 curr count: 8411
2021/12/17 06:18:04 Waiting for dst bucket item count to get to: 10000 curr count: 8411
2021/12/17 06:18:04 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/17 06:18:04 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 06:18:04 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 06:18:09 Waiting for app: TestN1QLThrowStatement to get un-deployed
2021/12/17 06:18:14 Waiting for app: TestN1QLThrowStatement to get un-deployed
2021/12/17 06:18:19 Waiting for app: TestN1QLThrowStatement to get un-deployed
2021/12/17 06:18:24 Waiting for app: TestN1QLThrowStatement to get un-deployed
2021/12/17 06:18:24 App: TestN1QLThrowStatement got un-deployed
2021/12/17 06:18:24 No eventing-con process running
2021/12/17 06:18:24 Function: TestN1QLThrowStatement delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLThrowStatement deleting in the background"
}
2021/12/17 06:18:26 Context: Bucket flush, response code: 200 dump: 
2021/12/17 06:18:36 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/17 06:18:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 06:18:38 Context: Bucket flush, response code: 200 dump: 
2021/12/17 06:18:48 Waiting for dst bucket item count to get to: 0 curr count: 6949
2021/12/17 06:18:53 Waiting for dst bucket item count to get to: 0 curr count: 1949
2021/12/17 06:18:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLThrowStatement (128.17s)
=== RUN   TestN1QLNestedForLoop
2021/12/17 06:18:58 Waiting for app: TestN1QLNestedForLoop to get un-deployed
2021/12/17 06:18:58 App: TestN1QLNestedForLoop got un-deployed
2021/12/17 06:18:58 No eventing-con process running
2021/12/17 06:18:58 Function: TestN1QLNestedForLoop delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLNestedForLoop not found"
 }
}
2021/12/17 06:19:00 Context: Bucket flush, response code: 200 dump: 
2021/12/17 06:19:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 06:19:07 Context: Bucket flush, response code: 200 dump: 
2021/12/17 06:19:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 06:19:12 Deploying app: TestN1QLNestedForLoop. Item count src bucket: 0 dst bucket: 0
2021/12/17 06:19:13 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLNestedForLoop' in metakv",
  "warnings": null
 }
}
2021/12/17 06:19:18 Waiting for app: TestN1QLNestedForLoop to get deployed
2021/12/17 06:19:23 Waiting for app: TestN1QLNestedForLoop to get deployed
2021/12/17 06:19:28 Waiting for app: TestN1QLNestedForLoop to get deployed
2021/12/17 06:19:33 Waiting for app: TestN1QLNestedForLoop to get deployed
2021/12/17 06:19:33 App: TestN1QLNestedForLoop got deployed
2021/12/17 06:19:33 Starting bucket ops to source bucket
2021/12/17 06:19:42 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 06:19:47 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 06:19:52 Waiting for dst bucket item count to get to: 5000 curr count: 563
2021/12/17 06:19:57 Waiting for dst bucket item count to get to: 5000 curr count: 1741
2021/12/17 06:20:02 Waiting for dst bucket item count to get to: 5000 curr count: 2949
2021/12/17 06:20:07 Waiting for dst bucket item count to get to: 5000 curr count: 4104
2021/12/17 06:20:07 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/17 06:20:07 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 06:20:07 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 06:20:12 Waiting for app: TestN1QLNestedForLoop to get un-deployed
2021/12/17 06:20:17 Waiting for app: TestN1QLNestedForLoop to get un-deployed
2021/12/17 06:20:22 Waiting for app: TestN1QLNestedForLoop to get un-deployed
2021/12/17 06:20:27 Waiting for app: TestN1QLNestedForLoop to get un-deployed
2021/12/17 06:20:27 App: TestN1QLNestedForLoop got un-deployed
2021/12/17 06:20:27 No eventing-con process running
2021/12/17 06:20:27 Function: TestN1QLNestedForLoop delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLNestedForLoop deleting in the background"
}
2021/12/17 06:20:29 Context: Bucket flush, response code: 200 dump: 
2021/12/17 06:20:39 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/17 06:20:44 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/17 06:20:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 06:20:46 Context: Bucket flush, response code: 200 dump: 
2021/12/17 06:20:56 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/17 06:20:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLNestedForLoop (122.95s)
=== RUN   TestN1QLAutoClose
2021/12/17 06:21:01 Waiting for app: TestN1QLAutoClose to get un-deployed
2021/12/17 06:21:01 App: TestN1QLAutoClose got un-deployed
2021/12/17 06:21:01 No eventing-con process running
2021/12/17 06:21:01 Function: TestN1QLAutoClose delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLAutoClose not found"
 }
}
2021/12/17 06:21:03 Context: Bucket flush, response code: 200 dump: 
2021/12/17 06:21:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 06:21:10 Context: Bucket flush, response code: 200 dump: 
2021/12/17 06:21:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 06:21:15 Deploying app: TestN1QLAutoClose. Item count src bucket: 0 dst bucket: 0
2021/12/17 06:21:16 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAutoClose' in metakv",
  "warnings": null
 }
}
2021/12/17 06:21:21 Waiting for app: TestN1QLAutoClose to get deployed
2021/12/17 06:21:26 Waiting for app: TestN1QLAutoClose to get deployed
2021/12/17 06:21:31 Waiting for app: TestN1QLAutoClose to get deployed
2021/12/17 06:21:36 Waiting for app: TestN1QLAutoClose to get deployed
2021/12/17 06:21:36 App: TestN1QLAutoClose got deployed
2021/12/17 06:21:36 Starting bucket ops to source bucket
2021/12/17 06:21:41 Waiting for dst bucket item count to get to: 500 curr count: 0
2021/12/17 06:21:46 Waiting for dst bucket item count to get to: 500 curr count: 2
2021/12/17 06:21:51 Waiting for dst bucket item count to get to: 500 curr count: 2
2021/12/17 06:21:56 Waiting for dst bucket item count to get to: 500 curr count: 26
2021/12/17 06:22:01 Waiting for dst bucket item count to get to: 500 curr count: 26
2021/12/17 06:22:06 Waiting for dst bucket item count to get to: 500 curr count: 48
2021/12/17 06:22:11 Waiting for dst bucket item count to get to: 500 curr count: 48
2021/12/17 06:22:17 Waiting for dst bucket item count to get to: 500 curr count: 70
2021/12/17 06:22:22 Waiting for dst bucket item count to get to: 500 curr count: 70
2021/12/17 06:22:27 Waiting for dst bucket item count to get to: 500 curr count: 91
2021/12/17 06:22:32 Waiting for dst bucket item count to get to: 500 curr count: 91
2021/12/17 06:22:37 Waiting for dst bucket item count to get to: 500 curr count: 112
2021/12/17 06:22:42 Waiting for dst bucket item count to get to: 500 curr count: 112
2021/12/17 06:22:47 Waiting for dst bucket item count to get to: 500 curr count: 112
2021/12/17 06:22:52 Waiting for dst bucket item count to get to: 500 curr count: 134
2021/12/17 06:22:57 Waiting for dst bucket item count to get to: 500 curr count: 134
2021/12/17 06:23:02 Waiting for dst bucket item count to get to: 500 curr count: 154
2021/12/17 06:23:07 Waiting for dst bucket item count to get to: 500 curr count: 154
2021/12/17 06:23:12 Waiting for dst bucket item count to get to: 500 curr count: 176
2021/12/17 06:23:17 Waiting for dst bucket item count to get to: 500 curr count: 176
2021/12/17 06:23:22 Waiting for dst bucket item count to get to: 500 curr count: 198
2021/12/17 06:23:27 Waiting for dst bucket item count to get to: 500 curr count: 198
2021/12/17 06:23:32 Waiting for dst bucket item count to get to: 500 curr count: 221
2021/12/17 06:23:37 Waiting for dst bucket item count to get to: 500 curr count: 230
2021/12/17 06:23:42 Waiting for dst bucket item count to get to: 500 curr count: 241
2021/12/17 06:23:47 Waiting for dst bucket item count to get to: 500 curr count: 250
2021/12/17 06:23:52 Waiting for dst bucket item count to get to: 500 curr count: 260
2021/12/17 06:23:57 Waiting for dst bucket item count to get to: 500 curr count: 269
2021/12/17 06:24:02 Waiting for dst bucket item count to get to: 500 curr count: 282
2021/12/17 06:24:07 Waiting for dst bucket item count to get to: 500 curr count: 296
2021/12/17 06:24:12 Waiting for dst bucket item count to get to: 500 curr count: 305
2021/12/17 06:24:17 Waiting for dst bucket item count to get to: 500 curr count: 317
2021/12/17 06:24:22 Waiting for dst bucket item count to get to: 500 curr count: 325
2021/12/17 06:24:27 Waiting for dst bucket item count to get to: 500 curr count: 336
2021/12/17 06:24:32 Waiting for dst bucket item count to get to: 500 curr count: 346
2021/12/17 06:24:37 Waiting for dst bucket item count to get to: 500 curr count: 357
2021/12/17 06:24:42 Waiting for dst bucket item count to get to: 500 curr count: 365
2021/12/17 06:24:47 Waiting for dst bucket item count to get to: 500 curr count: 375
2021/12/17 06:24:52 Waiting for dst bucket item count to get to: 500 curr count: 387
2021/12/17 06:24:57 Waiting for dst bucket item count to get to: 500 curr count: 398
2021/12/17 06:25:02 Waiting for dst bucket item count to get to: 500 curr count: 407
2021/12/17 06:25:07 Waiting for dst bucket item count to get to: 500 curr count: 428
2021/12/17 06:25:12 Waiting for dst bucket item count to get to: 500 curr count: 428
2021/12/17 06:25:17 Waiting for dst bucket item count to get to: 500 curr count: 451
2021/12/17 06:25:22 Waiting for dst bucket item count to get to: 500 curr count: 451
2021/12/17 06:25:27 Waiting for dst bucket item count to get to: 500 curr count: 473
2021/12/17 06:25:32 Waiting for dst bucket item count to get to: 500 curr count: 473
2021/12/17 06:25:37 Waiting for dst bucket item count to get to: 500 curr count: 492
2021/12/17 06:25:42 Waiting for dst bucket item count to get to: 500 curr count: 492
2021/12/17 06:25:42 src & dst bucket item count matched up. src bucket count: 500 dst bucket count: 500
2021/12/17 06:25:42 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 06:25:42 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 06:25:47 Waiting for app: TestN1QLAutoClose to get un-deployed
2021/12/17 06:25:52 Waiting for app: TestN1QLAutoClose to get un-deployed
2021/12/17 06:25:57 Waiting for app: TestN1QLAutoClose to get un-deployed
2021/12/17 06:26:02 Waiting for app: TestN1QLAutoClose to get un-deployed
2021/12/17 06:26:02 App: TestN1QLAutoClose got un-deployed
2021/12/17 06:26:02 No eventing-con process running
2021/12/17 06:26:02 Function: TestN1QLAutoClose delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLAutoClose deleting in the background"
}
2021/12/17 06:26:04 Context: Bucket flush, response code: 200 dump: 
2021/12/17 06:26:14 Waiting for dst bucket item count to get to: 0 curr count: 500
2021/12/17 06:26:19 Waiting for dst bucket item count to get to: 0 curr count: 250
2021/12/17 06:26:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 06:26:21 Context: Bucket flush, response code: 200 dump: 
2021/12/17 06:26:31 Waiting for dst bucket item count to get to: 0 curr count: 442
2021/12/17 06:26:36 Waiting for dst bucket item count to get to: 0 curr count: 442
2021/12/17 06:26:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLAutoClose (339.68s)
=== RUN   TestN1QLPosParams
2021/12/17 06:26:41 Waiting for app: TestN1QLPosParams to get un-deployed
2021/12/17 06:26:41 App: TestN1QLPosParams got un-deployed
2021/12/17 06:26:41 No eventing-con process running
2021/12/17 06:26:41 Function: TestN1QLPosParams delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLPosParams not found"
 }
}
2021/12/17 06:26:43 Context: Bucket flush, response code: 200 dump: 
2021/12/17 06:26:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 06:26:49 Context: Bucket flush, response code: 200 dump: 
2021/12/17 06:26:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 06:26:54 Deploying app: TestN1QLPosParams. Item count src bucket: 0 dst bucket: 0
2021/12/17 06:26:55 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLPosParams' in metakv",
  "warnings": null
 }
}
2021/12/17 06:27:00 Waiting for app: TestN1QLPosParams to get deployed
2021/12/17 06:27:05 Waiting for app: TestN1QLPosParams to get deployed
2021/12/17 06:27:10 Waiting for app: TestN1QLPosParams to get deployed
2021/12/17 06:27:15 Waiting for app: TestN1QLPosParams to get deployed
2021/12/17 06:27:15 App: TestN1QLPosParams got deployed
2021/12/17 06:27:15 Starting bucket ops to source bucket
2021/12/17 06:27:25 Waiting for dst bucket item count to get to: 5000 curr count: 909
2021/12/17 06:27:30 Waiting for dst bucket item count to get to: 5000 curr count: 909
2021/12/17 06:27:35 Waiting for dst bucket item count to get to: 5000 curr count: 909
2021/12/17 06:27:35 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/17 06:27:35 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 06:27:35 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 06:27:40 Waiting for app: TestN1QLPosParams to get un-deployed
2021/12/17 06:27:45 Waiting for app: TestN1QLPosParams to get un-deployed
2021/12/17 06:27:50 Waiting for app: TestN1QLPosParams to get un-deployed
2021/12/17 06:27:55 Waiting for app: TestN1QLPosParams to get un-deployed
2021/12/17 06:27:55 App: TestN1QLPosParams got un-deployed
2021/12/17 06:27:55 No eventing-con process running
2021/12/17 06:27:55 Function: TestN1QLPosParams delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLPosParams deleting in the background"
}
2021/12/17 06:27:57 Context: Bucket flush, response code: 200 dump: 
2021/12/17 06:28:07 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/17 06:28:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 06:28:09 Context: Bucket flush, response code: 200 dump: 
2021/12/17 06:28:19 Waiting for dst bucket item count to get to: 0 curr count: 4890
2021/12/17 06:28:24 Waiting for dst bucket item count to get to: 0 curr count: 2390
2021/12/17 06:28:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLPosParams (108.21s)
=== RUN   TestN1QLExhaustConnPool
2021/12/17 06:28:29 Waiting for app: TestN1QLExhaustConnPool to get un-deployed
2021/12/17 06:28:29 App: TestN1QLExhaustConnPool got un-deployed
2021/12/17 06:28:29 No eventing-con process running
2021/12/17 06:28:29 Function: TestN1QLExhaustConnPool delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLExhaustConnPool not found"
 }
}
2021/12/17 06:28:31 Context: Bucket flush, response code: 200 dump: 
2021/12/17 06:28:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 06:28:38 Context: Bucket flush, response code: 200 dump: 
2021/12/17 06:28:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 06:28:43 Deploying app: TestN1QLExhaustConnPool. Item count src bucket: 0 dst bucket: 0
2021/12/17 06:28:44 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLExhaustConnPool' in metakv",
  "warnings": null
 }
}
2021/12/17 06:28:49 Waiting for app: TestN1QLExhaustConnPool to get deployed
2021/12/17 06:28:54 Waiting for app: TestN1QLExhaustConnPool to get deployed
2021/12/17 06:28:59 Waiting for app: TestN1QLExhaustConnPool to get deployed
2021/12/17 06:29:04 Waiting for app: TestN1QLExhaustConnPool to get deployed
2021/12/17 06:29:04 App: TestN1QLExhaustConnPool got deployed
2021/12/17 06:29:04 Starting bucket ops to source bucket
2021/12/17 06:29:09 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/17 06:29:14 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/17 06:29:14 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/17 06:29:14 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 06:29:14 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 06:29:19 Waiting for app: TestN1QLExhaustConnPool to get un-deployed
2021/12/17 06:29:24 Waiting for app: TestN1QLExhaustConnPool to get un-deployed
2021/12/17 06:29:29 Waiting for app: TestN1QLExhaustConnPool to get un-deployed
2021/12/17 06:29:34 Waiting for app: TestN1QLExhaustConnPool to get un-deployed
2021/12/17 06:29:34 App: TestN1QLExhaustConnPool got un-deployed
2021/12/17 06:29:34 No eventing-con process running
2021/12/17 06:29:34 Function: TestN1QLExhaustConnPool delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLExhaustConnPool deleting in the background"
}
2021/12/17 06:29:36 Context: Bucket flush, response code: 200 dump: 
2021/12/17 06:29:46 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/17 06:29:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 06:29:48 Context: Bucket flush, response code: 200 dump: 
2021/12/17 06:29:58 Waiting for dst bucket item count to get to: 0 curr count: 56
2021/12/17 06:30:03 Waiting for dst bucket item count to get to: 0 curr count: 6
2021/12/17 06:30:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLExhaustConnPool (98.83s)
=== RUN   TestN1QLGraceSufficientTimeOut
2021/12/17 06:30:08 Waiting for app: TestN1QLGraceSufficientTimeOut to get un-deployed
2021/12/17 06:30:08 App: TestN1QLGraceSufficientTimeOut got un-deployed
2021/12/17 06:30:08 No eventing-con process running
2021/12/17 06:30:08 Function: TestN1QLGraceSufficientTimeOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLGraceSufficientTimeOut not found"
 }
}
2021/12/17 06:30:10 Context: Bucket flush, response code: 200 dump: 
2021/12/17 06:30:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 06:30:16 Context: Bucket flush, response code: 200 dump: 
2021/12/17 06:30:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 06:30:21 Starting bucket ops to source bucket
2021/12/17 06:30:39 Starting bucket ops to source bucket
2021/12/17 06:30:57 Deploying app: TestN1QLGraceSufficientTimeOut. Item count src bucket: 30000 dst bucket: 15580
2021/12/17 06:30:58 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLGraceSufficientTimeOut' in metakv",
  "warnings": null
 }
}
2021/12/17 06:31:03 Waiting for app: TestN1QLGraceSufficientTimeOut to get deployed
2021/12/17 06:31:08 Waiting for app: TestN1QLGraceSufficientTimeOut to get deployed
2021/12/17 06:31:13 Waiting for app: TestN1QLGraceSufficientTimeOut to get deployed
2021/12/17 06:31:18 Waiting for app: TestN1QLGraceSufficientTimeOut to get deployed
2021/12/17 06:31:18 App: TestN1QLGraceSufficientTimeOut got deployed
2021/12/17 06:31:18 Starting bucket ops to source bucket
2021/12/17 06:31:23 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/17 06:31:28 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/17 06:31:33 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/17 06:31:38 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/17 06:31:43 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/17 06:31:48 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/17 06:31:53 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/17 06:31:58 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/17 06:32:03 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/17 06:32:03 src & dst bucket item count matched up. src bucket count: 30001 dst bucket count: 30001
2021/12/17 06:32:03 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 06:32:03 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 06:32:08 Waiting for app: TestN1QLGraceSufficientTimeOut to get un-deployed
2021/12/17 06:32:13 Waiting for app: TestN1QLGraceSufficientTimeOut to get un-deployed
2021/12/17 06:32:18 Waiting for app: TestN1QLGraceSufficientTimeOut to get un-deployed
2021/12/17 06:32:23 Waiting for app: TestN1QLGraceSufficientTimeOut to get un-deployed
2021/12/17 06:32:23 App: TestN1QLGraceSufficientTimeOut got un-deployed
2021/12/17 06:32:23 No eventing-con process running
2021/12/17 06:32:23 Function: TestN1QLGraceSufficientTimeOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLGraceSufficientTimeOut deleting in the background"
}
2021/12/17 06:32:25 Context: Bucket flush, response code: 200 dump: 
2021/12/17 06:32:35 Waiting for dst bucket item count to get to: 0 curr count: 30000
2021/12/17 06:32:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 06:32:37 Context: Bucket flush, response code: 200 dump: 
2021/12/17 06:32:47 Waiting for dst bucket item count to get to: 0 curr count: 30001
2021/12/17 06:32:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLGraceSufficientTimeOut (164.19s)
=== RUN   TestN1QLQueryConsistency
2021/12/17 06:32:52 Waiting for app: n1qlquery_consistency to get un-deployed
2021/12/17 06:32:52 App: n1qlquery_consistency got un-deployed
2021/12/17 06:32:52 No eventing-con process running
2021/12/17 06:32:52 Function: n1qlquery_consistency delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: n1qlquery_consistency not found"
 }
}
2021/12/17 06:32:54 Context: Bucket flush, response code: 200 dump: 
2021/12/17 06:32:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 06:33:01 Context: Bucket flush, response code: 200 dump: 
2021/12/17 06:33:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 06:33:06 Starting bucket ops to source bucket
2021/12/17 06:33:09 Deploying app: n1qlquery_consistency. Item count src bucket: 0 dst bucket: 0
2021/12/17 06:33:10 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'n1qlquery_consistency' in metakv",
  "warnings": null
 }
}
2021/12/17 06:33:15 Waiting for app: n1qlquery_consistency to get deployed
2021/12/17 06:33:20 Waiting for app: n1qlquery_consistency to get deployed
2021/12/17 06:33:25 Waiting for app: n1qlquery_consistency to get deployed
2021/12/17 06:33:30 Waiting for app: n1qlquery_consistency to get deployed
2021/12/17 06:33:30 App: n1qlquery_consistency got deployed
2021/12/17 06:33:35 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 06:33:40 Waiting for dst bucket item count to get to: 5000 curr count: 3978
2021/12/17 06:33:45 Waiting for dst bucket item count to get to: 5000 curr count: 3978
2021/12/17 06:33:45 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/17 06:33:45 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 06:33:45 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 06:33:50 Waiting for app: n1qlquery_consistency to get un-deployed
2021/12/17 06:33:55 Waiting for app: n1qlquery_consistency to get un-deployed
2021/12/17 06:34:00 Waiting for app: n1qlquery_consistency to get un-deployed
2021/12/17 06:34:05 Waiting for app: n1qlquery_consistency to get un-deployed
2021/12/17 06:34:05 App: n1qlquery_consistency got un-deployed
2021/12/17 06:34:05 No eventing-con process running
2021/12/17 06:34:05 Function: n1qlquery_consistency delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: n1qlquery_consistency deleting in the background"
}
2021/12/17 06:34:07 Context: Bucket flush, response code: 200 dump: 
2021/12/17 06:34:17 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/17 06:34:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 06:34:19 Context: Bucket flush, response code: 200 dump: 
2021/12/17 06:34:29 Waiting for dst bucket item count to get to: 0 curr count: 4662
2021/12/17 06:34:34 Waiting for dst bucket item count to get to: 0 curr count: 2162
2021/12/17 06:34:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLQueryConsistency (106.83s)
PASS
ok  	github.com/couchbase/eventing/tests/functional_tests	1543.380s

N1ql tests took 0 hours, 25 minutes and 48 seconds.

Starting server
Starting server: attempt 1

Rebalance tests

2021/12/17 06:36:09 Started rebalance
Post "http://127.0.0.1:9000/pools/default": read tcp 127.0.0.1:38982->127.0.0.1:9000: read: connection reset by peer
Quota setup Post "http://127.0.0.1:9000/pools/default": read tcp 127.0.0.1:38982->127.0.0.1:9000: read: connection reset by peer
=== RUN   TestDuplicateWithEventingRebIn
2021/12/17 06:36:28 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
running apps fetch error json: cannot unmarshal number into Go value of type string
2021/12/17 06:36:33 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
2021/12/17 06:36:33 App: TestDuplicateWithEventingRebIn got un-deployed
2021/12/17 06:36:33 No eventing-con process running
2021/12/17 06:36:33 Function: TestDuplicateWithEventingRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithEventingRebIn not found"
 }
}
2021/12/17 06:36:35 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 06:36:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 06:36:40 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 06:36:41 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 06:36:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 06:36:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 06:36:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 06:36:51 Deploying app: TestDuplicateWithEventingRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/17 06:36:52 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithEventingRebIn' in metakv",
  "warnings": null
 }
}
2021/12/17 06:36:57 Waiting for app: TestDuplicateWithEventingRebIn to get deployed
2021/12/17 06:37:02 Waiting for app: TestDuplicateWithEventingRebIn to get deployed
2021/12/17 06:37:07 Waiting for app: TestDuplicateWithEventingRebIn to get deployed
2021/12/17 06:37:12 Waiting for app: TestDuplicateWithEventingRebIn to get deployed
2021/12/17 06:37:12 App: TestDuplicateWithEventingRebIn got deployed
2021/12/17 06:37:12 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/17 06:37:12 Starting bucket ops to source bucket
2021/12/17 06:37:27 Removing node(s): [] from the cluster
2021/12/17 06:37:32 Rebalance progress: 0
2021/12/17 06:37:37 Rebalance progress: 0
2021/12/17 06:37:42 Rebalance progress: 8.333333333333332
2021/12/17 06:37:47 Rebalance progress: 25
2021/12/17 06:37:52 Rebalance progress: 25
2021/12/17 06:37:57 Rebalance progress: 25
2021/12/17 06:38:02 Rebalance progress: 25
2021/12/17 06:38:07 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 06:38:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 06:38:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 06:38:07 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9302/api/v1/stats": dial tcp 127.0.0.1:9302: connect: connection refused
2021/12/17 06:38:07 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/17 06:38:07 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/17 06:38:12 Rebalance progress: 25
2021/12/17 06:38:17 Rebalance progress: 25
2021/12/17 06:38:22 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 06:38:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 06:38:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 06:38:22 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9302/api/v1/stats": dial tcp 127.0.0.1:9302: connect: connection refused
2021/12/17 06:38:22 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/17 06:38:22 Waiting for src bucket item count to get to: 200000 curr count: 92239
2021/12/17 06:38:27 Waiting for src bucket item count to get to: 200000 curr count: 109071
2021/12/17 06:38:32 Waiting for src bucket item count to get to: 200000 curr count: 109071
2021/12/17 06:38:37 Waiting for src bucket item count to get to: 200000 curr count: 125418
2021/12/17 06:38:42 Waiting for src bucket item count to get to: 200000 curr count: 125418
2021/12/17 06:38:47 Waiting for src bucket item count to get to: 200000 curr count: 141978
2021/12/17 06:38:52 Waiting for src bucket item count to get to: 200000 curr count: 141978
2021/12/17 06:38:57 Waiting for src bucket item count to get to: 200000 curr count: 158569
2021/12/17 06:39:02 Waiting for src bucket item count to get to: 200000 curr count: 158569
2021/12/17 06:39:07 Waiting for src bucket item count to get to: 200000 curr count: 174915
2021/12/17 06:39:12 Waiting for src bucket item count to get to: 200000 curr count: 174915
2021/12/17 06:39:17 Waiting for src bucket item count to get to: 200000 curr count: 191543
2021/12/17 06:39:22 Waiting for src bucket item count to get to: 200000 curr count: 191543
2021/12/17 06:39:27 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/17 06:39:27 Post SET ops. Expected item count: 200000 got 200000
2021/12/17 06:39:27 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 06:39:32 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
2021/12/17 06:39:37 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
2021/12/17 06:39:42 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
2021/12/17 06:39:47 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
2021/12/17 06:39:47 App: TestDuplicateWithEventingRebIn got un-deployed
2021/12/17 06:39:47 No eventing-con process running
2021/12/17 06:39:47 Function: TestDuplicateWithEventingRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithEventingRebIn deleting in the background"
}
2021/12/17 06:39:49 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 06:39:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 06:39:59 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/17 06:39:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 06:40:00 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 06:40:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 06:40:10 Waiting for dst bucket item count to get to: 0 curr count: 91853
2021/12/17 06:40:15 Waiting for dst bucket item count to get to: 0 curr count: 91853
2021/12/17 06:40:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithEventingRebIn (237.54s)
=== RUN   TestDuplicateWithEventingRebOut
2021/12/17 06:40:20 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/17 06:40:21 Removing node(s): [] from the cluster
2021/12/17 06:40:26 Rebalance progress: 100
2021/12/17 06:40:31 Waiting for app: TestDuplicateWithEventingRebOut to get un-deployed
2021/12/17 06:40:31 App: TestDuplicateWithEventingRebOut got un-deployed
2021/12/17 06:40:31 No eventing-con process running
2021/12/17 06:40:31 Function: TestDuplicateWithEventingRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithEventingRebOut not found"
 }
}
2021/12/17 06:40:33 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 06:40:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 06:40:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 06:40:39 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 06:40:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 06:40:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 06:40:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 06:40:49 Deploying app: TestDuplicateWithEventingRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/17 06:40:50 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithEventingRebOut' in metakv",
  "warnings": null
 }
}
2021/12/17 06:40:55 Waiting for app: TestDuplicateWithEventingRebOut to get deployed
2021/12/17 06:41:00 Waiting for app: TestDuplicateWithEventingRebOut to get deployed
2021/12/17 06:41:05 Waiting for app: TestDuplicateWithEventingRebOut to get deployed
2021/12/17 06:41:10 Waiting for app: TestDuplicateWithEventingRebOut to get deployed
2021/12/17 06:41:10 App: TestDuplicateWithEventingRebOut got deployed
2021/12/17 06:41:10 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/17 06:41:10 Starting bucket ops to source bucket
2021/12/17 06:41:16 Rebalance progress: 25
2021/12/17 06:41:21 Rebalance progress: 25
2021/12/17 06:41:26 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 06:41:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 06:41:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 06:41:26 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9302/api/v1/stats": dial tcp 127.0.0.1:9302: connect: connection refused
2021/12/17 06:41:26 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/17 06:41:26 Waiting for src bucket item count to get to: 200000 curr count: 13827
2021/12/17 06:41:31 Waiting for src bucket item count to get to: 200000 curr count: 30094
2021/12/17 06:41:36 Waiting for src bucket item count to get to: 200000 curr count: 30094
2021/12/17 06:41:41 Waiting for src bucket item count to get to: 200000 curr count: 46326
2021/12/17 06:41:46 Waiting for src bucket item count to get to: 200000 curr count: 46326
2021/12/17 06:41:51 Waiting for src bucket item count to get to: 200000 curr count: 62758
2021/12/17 06:41:56 Waiting for src bucket item count to get to: 200000 curr count: 62758
2021/12/17 06:42:01 Waiting for src bucket item count to get to: 200000 curr count: 62758
2021/12/17 06:42:06 Waiting for src bucket item count to get to: 200000 curr count: 78814
2021/12/17 06:42:11 Waiting for src bucket item count to get to: 200000 curr count: 78814
2021/12/17 06:42:16 Waiting for src bucket item count to get to: 200000 curr count: 95165
2021/12/17 06:42:21 Waiting for src bucket item count to get to: 200000 curr count: 95165
2021/12/17 06:42:26 Waiting for src bucket item count to get to: 200000 curr count: 111804
2021/12/17 06:42:31 Waiting for src bucket item count to get to: 200000 curr count: 111804
2021/12/17 06:42:36 Waiting for src bucket item count to get to: 200000 curr count: 127957
2021/12/17 06:42:41 Waiting for src bucket item count to get to: 200000 curr count: 127957
2021/12/17 06:42:46 Waiting for src bucket item count to get to: 200000 curr count: 144235
2021/12/17 06:42:51 Waiting for src bucket item count to get to: 200000 curr count: 144235
2021/12/17 06:42:56 Waiting for src bucket item count to get to: 200000 curr count: 160668
2021/12/17 06:43:01 Waiting for src bucket item count to get to: 200000 curr count: 160668
2021/12/17 06:43:06 Waiting for src bucket item count to get to: 200000 curr count: 176811
2021/12/17 06:43:11 Waiting for src bucket item count to get to: 200000 curr count: 176811
2021/12/17 06:43:16 Waiting for src bucket item count to get to: 200000 curr count: 193180
2021/12/17 06:43:21 Waiting for src bucket item count to get to: 200000 curr count: 193180
2021/12/17 06:43:26 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/17 06:43:26 Post SET ops. Expected item count: 200000 got 200000
2021/12/17 06:43:26 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 06:43:31 Waiting for app: TestDuplicateWithEventingRebOut to get un-deployed
2021/12/17 06:43:36 Waiting for app: TestDuplicateWithEventingRebOut to get un-deployed
2021/12/17 06:43:41 Waiting for app: TestDuplicateWithEventingRebOut to get un-deployed
2021/12/17 06:43:46 Waiting for app: TestDuplicateWithEventingRebOut to get un-deployed
2021/12/17 06:43:46 App: TestDuplicateWithEventingRebOut got un-deployed
2021/12/17 06:43:46 No eventing-con process running
2021/12/17 06:43:46 Function: TestDuplicateWithEventingRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithEventingRebOut deleting in the background"
}
2021/12/17 06:43:48 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 06:43:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 06:43:58 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/17 06:43:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 06:43:59 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 06:44:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 06:44:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithEventingRebOut (228.97s)
=== RUN   TestDuplicateWithEventingSwapReb
2021/12/17 06:44:14 Waiting for app: TestDuplicateWithEventingSwapReb to get un-deployed
2021/12/17 06:44:14 App: TestDuplicateWithEventingSwapReb got un-deployed
2021/12/17 06:44:14 No eventing-con process running
2021/12/17 06:44:14 Function: TestDuplicateWithEventingSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithEventingSwapReb not found"
 }
}
2021/12/17 06:44:16 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 06:44:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 06:44:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 06:44:23 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 06:44:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 06:44:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 06:44:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 06:44:33 Deploying app: TestDuplicateWithEventingSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/17 06:44:34 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithEventingSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/17 06:44:39 Waiting for app: TestDuplicateWithEventingSwapReb to get deployed
2021/12/17 06:44:44 Waiting for app: TestDuplicateWithEventingSwapReb to get deployed
2021/12/17 06:44:49 Waiting for app: TestDuplicateWithEventingSwapReb to get deployed
2021/12/17 06:44:54 Waiting for app: TestDuplicateWithEventingSwapReb to get deployed
2021/12/17 06:44:54 App: TestDuplicateWithEventingSwapReb got deployed
2021/12/17 06:44:54 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/17 06:44:54 Starting bucket ops to source bucket
2021/12/17 06:44:54 Removing node(s): [] from the cluster
2021/12/17 06:44:59 Rebalance progress: 25
2021/12/17 06:45:04 Rebalance progress: 25
2021/12/17 06:45:09 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 06:45:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 06:45:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 06:45:09 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9302/api/v1/stats": dial tcp 127.0.0.1:9302: connect: connection refused
2021/12/17 06:45:09 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/17 06:45:09 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/17 06:45:16 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/17 06:45:21 Rebalance progress: 16.66666666666667
2021/12/17 06:45:26 Rebalance progress: 16.66666666666667
2021/12/17 06:45:31 Rebalance progress: 16.66666666666667
2021/12/17 06:45:36 Rebalance progress: 51.16279069767442
2021/12/17 06:45:41 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 06:45:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 06:45:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 06:45:41 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/17 06:45:41 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/17 06:45:46 Rebalance progress: 16.66666666666667
2021/12/17 06:45:51 Rebalance progress: 16.66666666666667
2021/12/17 06:45:56 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 06:45:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 06:45:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 06:45:56 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/17 06:45:56 Waiting for src bucket item count to get to: 200000 curr count: 82508
2021/12/17 06:46:01 Waiting for src bucket item count to get to: 200000 curr count: 82508
2021/12/17 06:46:06 Waiting for src bucket item count to get to: 200000 curr count: 98528
2021/12/17 06:46:11 Waiting for src bucket item count to get to: 200000 curr count: 114758
2021/12/17 06:46:17 Waiting for src bucket item count to get to: 200000 curr count: 114758
2021/12/17 06:46:22 Waiting for src bucket item count to get to: 200000 curr count: 131063
2021/12/17 06:46:27 Waiting for src bucket item count to get to: 200000 curr count: 131063
2021/12/17 06:46:32 Waiting for src bucket item count to get to: 200000 curr count: 147229
2021/12/17 06:46:37 Waiting for src bucket item count to get to: 200000 curr count: 147229
2021/12/17 06:46:42 Waiting for src bucket item count to get to: 200000 curr count: 163868
2021/12/17 06:46:47 Waiting for src bucket item count to get to: 200000 curr count: 163868
2021/12/17 06:46:52 Waiting for src bucket item count to get to: 200000 curr count: 179758
2021/12/17 06:46:57 Waiting for src bucket item count to get to: 200000 curr count: 179758
2021/12/17 06:47:02 Waiting for src bucket item count to get to: 200000 curr count: 196267
2021/12/17 06:47:07 Waiting for src bucket item count to get to: 200000 curr count: 196267
2021/12/17 06:47:12 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/17 06:47:12 Post SET ops. Expected item count: 200000 got 200000
2021/12/17 06:47:12 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 06:47:17 Waiting for app: TestDuplicateWithEventingSwapReb to get un-deployed
2021/12/17 06:47:22 Waiting for app: TestDuplicateWithEventingSwapReb to get un-deployed
2021/12/17 06:47:27 Waiting for app: TestDuplicateWithEventingSwapReb to get un-deployed
2021/12/17 06:47:32 Waiting for app: TestDuplicateWithEventingSwapReb to get un-deployed
2021/12/17 06:47:32 App: TestDuplicateWithEventingSwapReb got un-deployed
2021/12/17 06:47:32 No eventing-con process running
2021/12/17 06:47:32 Function: TestDuplicateWithEventingSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithEventingSwapReb deleting in the background"
}
2021/12/17 06:47:33 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 06:47:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 06:47:43 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/17 06:47:48 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/17 06:47:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 06:47:50 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 06:47:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 06:48:00 Waiting for dst bucket item count to get to: 0 curr count: 59774
2021/12/17 06:48:05 Waiting for dst bucket item count to get to: 0 curr count: 59774
2021/12/17 06:48:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithEventingSwapReb (240.86s)
=== RUN   TestDuplicateWithKVRebIn
2021/12/17 06:48:15 Waiting for app: TestDuplicateWithKVRebIn to get un-deployed
2021/12/17 06:48:15 App: TestDuplicateWithKVRebIn got un-deployed
2021/12/17 06:48:15 No eventing-con process running
2021/12/17 06:48:15 Function: TestDuplicateWithKVRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithKVRebIn not found"
 }
}
2021/12/17 06:48:17 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 06:48:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 06:48:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 06:48:24 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 06:48:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 06:48:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 06:48:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 06:48:34 Deploying app: TestDuplicateWithKVRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/17 06:48:35 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithKVRebIn' in metakv",
  "warnings": null
 }
}
2021/12/17 06:48:40 Waiting for app: TestDuplicateWithKVRebIn to get deployed
2021/12/17 06:48:45 Waiting for app: TestDuplicateWithKVRebIn to get deployed
2021/12/17 06:48:50 Waiting for app: TestDuplicateWithKVRebIn to get deployed
2021/12/17 06:48:55 Waiting for app: TestDuplicateWithKVRebIn to get deployed
2021/12/17 06:48:55 App: TestDuplicateWithKVRebIn got deployed
2021/12/17 06:48:55 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/17 06:48:55 Starting bucket ops to source bucket
2021/12/17 06:48:55 Removing node(s): [] from the cluster
2021/12/17 06:49:01 Rebalance progress: 16.66666666666667
2021/12/17 06:49:06 Rebalance progress: 16.66666666666667
2021/12/17 06:49:11 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 06:49:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 06:49:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 06:49:11 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/17 06:49:11 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/17 06:49:16 Rebalance progress: 16.66666666666667
2021/12/17 06:49:21 Rebalance progress: 16.66666666666667
2021/12/17 06:49:26 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 06:49:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 06:49:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 06:49:26 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/17 06:49:26 Waiting for src bucket item count to get to: 200000 curr count: 37285
2021/12/17 06:49:31 Waiting for src bucket item count to get to: 200000 curr count: 53075
2021/12/17 06:49:36 Waiting for src bucket item count to get to: 200000 curr count: 53075
2021/12/17 06:49:41 Waiting for src bucket item count to get to: 200000 curr count: 69424
2021/12/17 06:49:46 Waiting for src bucket item count to get to: 200000 curr count: 69424
2021/12/17 06:49:51 Waiting for src bucket item count to get to: 200000 curr count: 84839
2021/12/17 06:49:56 Waiting for src bucket item count to get to: 200000 curr count: 84839
2021/12/17 06:50:01 Waiting for src bucket item count to get to: 200000 curr count: 100849
2021/12/17 06:50:06 Waiting for src bucket item count to get to: 200000 curr count: 100849
2021/12/17 06:50:11 Waiting for src bucket item count to get to: 200000 curr count: 117125
2021/12/17 06:50:16 Waiting for src bucket item count to get to: 200000 curr count: 117125
2021/12/17 06:50:21 Waiting for src bucket item count to get to: 200000 curr count: 133537
2021/12/17 06:50:26 Waiting for src bucket item count to get to: 200000 curr count: 133537
2021/12/17 06:50:31 Waiting for src bucket item count to get to: 200000 curr count: 133537
2021/12/17 06:50:36 Waiting for src bucket item count to get to: 200000 curr count: 149579
2021/12/17 06:50:41 Waiting for src bucket item count to get to: 200000 curr count: 149579
2021/12/17 06:50:46 Waiting for src bucket item count to get to: 200000 curr count: 165667
2021/12/17 06:50:51 Waiting for src bucket item count to get to: 200000 curr count: 165667
2021/12/17 06:50:56 Waiting for src bucket item count to get to: 200000 curr count: 180662
2021/12/17 06:51:01 Waiting for src bucket item count to get to: 200000 curr count: 180662
2021/12/17 06:51:06 Waiting for src bucket item count to get to: 200000 curr count: 196891
2021/12/17 06:51:11 Waiting for src bucket item count to get to: 200000 curr count: 196891
2021/12/17 06:51:16 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/17 06:51:16 Post SET ops. Expected item count: 200000 got 200000
2021/12/17 06:51:16 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 06:51:21 Waiting for app: TestDuplicateWithKVRebIn to get un-deployed
2021/12/17 06:51:26 Waiting for app: TestDuplicateWithKVRebIn to get un-deployed
2021/12/17 06:51:31 Waiting for app: TestDuplicateWithKVRebIn to get un-deployed
2021/12/17 06:51:36 Waiting for app: TestDuplicateWithKVRebIn to get un-deployed
2021/12/17 06:51:36 App: TestDuplicateWithKVRebIn got un-deployed
2021/12/17 06:51:36 No eventing-con process running
2021/12/17 06:51:36 Function: TestDuplicateWithKVRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithKVRebIn deleting in the background"
}
2021/12/17 06:51:38 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 06:51:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 06:51:48 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/17 06:51:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 06:51:50 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 06:51:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 06:51:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithKVRebIn (229.34s)
=== RUN   TestDuplicateWithKVRebOut
2021/12/17 06:52:00 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/17 06:52:00 Removing node(s): [] from the cluster
2021/12/17 06:52:05 Rebalance progress: 100
2021/12/17 06:52:10 Waiting for app: TestDuplicateWithKVRebOut to get un-deployed
2021/12/17 06:52:10 App: TestDuplicateWithKVRebOut got un-deployed
2021/12/17 06:52:10 No eventing-con process running
2021/12/17 06:52:10 Function: TestDuplicateWithKVRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithKVRebOut not found"
 }
}
2021/12/17 06:52:12 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 06:52:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 06:52:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 06:52:18 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 06:52:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 06:52:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 06:52:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 06:52:28 Deploying app: TestDuplicateWithKVRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/17 06:52:30 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithKVRebOut' in metakv",
  "warnings": null
 }
}
2021/12/17 06:52:35 Waiting for app: TestDuplicateWithKVRebOut to get deployed
2021/12/17 06:52:40 Waiting for app: TestDuplicateWithKVRebOut to get deployed
2021/12/17 06:52:45 Waiting for app: TestDuplicateWithKVRebOut to get deployed
2021/12/17 06:52:50 Waiting for app: TestDuplicateWithKVRebOut to get deployed
2021/12/17 06:52:50 App: TestDuplicateWithKVRebOut got deployed
2021/12/17 06:52:50 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/17 06:52:50 Starting bucket ops to source bucket
2021/12/17 06:52:55 Rebalance progress: 16.66666666666667
2021/12/17 06:53:00 Rebalance progress: 16.66666666666667
2021/12/17 06:53:05 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 06:53:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 06:53:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 06:53:05 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/17 06:53:05 Waiting for src bucket item count to get to: 200000 curr count: 15151
2021/12/17 06:53:10 Waiting for src bucket item count to get to: 200000 curr count: 15151
2021/12/17 06:53:15 Waiting for src bucket item count to get to: 200000 curr count: 31330
2021/12/17 06:53:20 Waiting for src bucket item count to get to: 200000 curr count: 31330
2021/12/17 06:53:25 Waiting for src bucket item count to get to: 200000 curr count: 47587
2021/12/17 06:53:30 Waiting for src bucket item count to get to: 200000 curr count: 63721
2021/12/17 06:53:35 Waiting for src bucket item count to get to: 200000 curr count: 63721
2021/12/17 06:53:40 Waiting for src bucket item count to get to: 200000 curr count: 63721
2021/12/17 06:53:45 Waiting for src bucket item count to get to: 200000 curr count: 79484
2021/12/17 06:53:50 Waiting for src bucket item count to get to: 200000 curr count: 79484
2021/12/17 06:53:55 Waiting for src bucket item count to get to: 200000 curr count: 95283
2021/12/17 06:54:00 Waiting for src bucket item count to get to: 200000 curr count: 95283
2021/12/17 06:54:05 Waiting for src bucket item count to get to: 200000 curr count: 111620
2021/12/17 06:54:10 Waiting for src bucket item count to get to: 200000 curr count: 111620
2021/12/17 06:54:15 Waiting for src bucket item count to get to: 200000 curr count: 127967
2021/12/17 06:54:20 Waiting for src bucket item count to get to: 200000 curr count: 127967
2021/12/17 06:54:25 Waiting for src bucket item count to get to: 200000 curr count: 144490
2021/12/17 06:54:30 Waiting for src bucket item count to get to: 200000 curr count: 144490
2021/12/17 06:54:35 Waiting for src bucket item count to get to: 200000 curr count: 160923
2021/12/17 06:54:40 Waiting for src bucket item count to get to: 200000 curr count: 160923
2021/12/17 06:54:45 Waiting for src bucket item count to get to: 200000 curr count: 176613
2021/12/17 06:54:50 Waiting for src bucket item count to get to: 200000 curr count: 176613
2021/12/17 06:54:55 Waiting for src bucket item count to get to: 200000 curr count: 192769
2021/12/17 06:55:00 Waiting for src bucket item count to get to: 200000 curr count: 192769
2021/12/17 06:55:05 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/17 06:55:05 Post SET ops. Expected item count: 200000 got 200000
2021/12/17 06:55:05 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 06:55:10 Waiting for app: TestDuplicateWithKVRebOut to get un-deployed
2021/12/17 06:55:15 Waiting for app: TestDuplicateWithKVRebOut to get un-deployed
2021/12/17 06:55:20 Waiting for app: TestDuplicateWithKVRebOut to get un-deployed
2021/12/17 06:55:25 Waiting for app: TestDuplicateWithKVRebOut to get un-deployed
2021/12/17 06:55:25 App: TestDuplicateWithKVRebOut got un-deployed
2021/12/17 06:55:25 No eventing-con process running
2021/12/17 06:55:25 Function: TestDuplicateWithKVRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithKVRebOut deleting in the background"
}
2021/12/17 06:55:27 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 06:55:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 06:55:37 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/17 06:55:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 06:55:39 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 06:55:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 06:55:49 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/17 06:55:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithKVRebOut (233.99s)
=== RUN   TestDuplicateWithKVSwapReb
2021/12/17 06:55:59 Waiting for app: TestDuplicateWithKVSwapReb to get un-deployed
2021/12/17 06:55:59 App: TestDuplicateWithKVSwapReb got un-deployed
2021/12/17 06:55:59 No eventing-con process running
2021/12/17 06:55:59 Function: TestDuplicateWithKVSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithKVSwapReb not found"
 }
}
2021/12/17 06:56:00 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 06:56:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 06:56:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 06:56:07 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 06:56:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 06:56:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 06:56:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 06:56:17 Deploying app: TestDuplicateWithKVSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/17 06:56:18 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithKVSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/17 06:56:23 Waiting for app: TestDuplicateWithKVSwapReb to get deployed
2021/12/17 06:56:28 Waiting for app: TestDuplicateWithKVSwapReb to get deployed
2021/12/17 06:56:33 Waiting for app: TestDuplicateWithKVSwapReb to get deployed
2021/12/17 06:56:38 Waiting for app: TestDuplicateWithKVSwapReb to get deployed
2021/12/17 06:56:38 App: TestDuplicateWithKVSwapReb got deployed
2021/12/17 06:56:38 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/17 06:56:38 Starting bucket ops to source bucket
2021/12/17 06:56:38 Removing node(s): [] from the cluster
2021/12/17 06:56:43 Rebalance progress: 16.66666666666667
2021/12/17 06:56:48 Rebalance progress: 16.66666666666667
2021/12/17 06:56:53 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 06:56:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 06:56:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 06:56:54 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/17 06:56:54 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/17 06:56:54 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/17 06:56:59 Rebalance progress: 16.66666666666667
2021/12/17 06:57:04 Rebalance progress: 16.66666666666667
2021/12/17 06:57:09 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 06:57:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 06:57:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 06:57:09 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/17 06:57:09 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/17 06:57:14 Rebalance progress: 16.66666666666667
2021/12/17 06:57:19 Rebalance progress: 16.66666666666667
2021/12/17 06:57:24 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 06:57:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 06:57:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 06:57:24 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/17 06:57:24 Waiting for src bucket item count to get to: 200000 curr count: 64481
2021/12/17 06:57:30 Waiting for src bucket item count to get to: 200000 curr count: 64481
2021/12/17 06:57:35 Waiting for src bucket item count to get to: 200000 curr count: 79440
2021/12/17 06:57:40 Waiting for src bucket item count to get to: 200000 curr count: 79440
2021/12/17 06:57:45 Waiting for src bucket item count to get to: 200000 curr count: 95213
2021/12/17 06:57:50 Waiting for src bucket item count to get to: 200000 curr count: 95213
2021/12/17 06:57:55 Waiting for src bucket item count to get to: 200000 curr count: 111522
2021/12/17 06:58:00 Waiting for src bucket item count to get to: 200000 curr count: 111522
2021/12/17 06:58:05 Waiting for src bucket item count to get to: 200000 curr count: 127901
2021/12/17 06:58:10 Waiting for src bucket item count to get to: 200000 curr count: 127901
2021/12/17 06:58:15 Waiting for src bucket item count to get to: 200000 curr count: 144253
2021/12/17 06:58:20 Waiting for src bucket item count to get to: 200000 curr count: 144253
2021/12/17 06:58:25 Waiting for src bucket item count to get to: 200000 curr count: 160173
2021/12/17 06:58:30 Waiting for src bucket item count to get to: 200000 curr count: 160173
2021/12/17 06:58:35 Waiting for src bucket item count to get to: 200000 curr count: 175491
2021/12/17 06:58:40 Waiting for src bucket item count to get to: 200000 curr count: 175491
2021/12/17 06:58:45 Waiting for src bucket item count to get to: 200000 curr count: 191257
2021/12/17 06:58:50 Waiting for src bucket item count to get to: 200000 curr count: 191257
2021/12/17 06:58:55 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/17 06:58:55 Post SET ops. Expected item count: 200000 got 200000
2021/12/17 06:58:55 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 06:59:00 Waiting for app: TestDuplicateWithKVSwapReb to get un-deployed
2021/12/17 06:59:05 Waiting for app: TestDuplicateWithKVSwapReb to get un-deployed
2021/12/17 06:59:10 Waiting for app: TestDuplicateWithKVSwapReb to get un-deployed
2021/12/17 06:59:15 Waiting for app: TestDuplicateWithKVSwapReb to get un-deployed
2021/12/17 06:59:15 App: TestDuplicateWithKVSwapReb got un-deployed
2021/12/17 06:59:15 No eventing-con process running
2021/12/17 06:59:15 Function: TestDuplicateWithKVSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithKVSwapReb deleting in the background"
}
2021/12/17 06:59:17 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 06:59:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 06:59:27 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/17 06:59:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 06:59:28 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 06:59:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 06:59:38 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/17 06:59:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithKVSwapReb (229.83s)
=== RUN   TestEventingRebKVOpsOnUpdateBucketOpOneByOne
2021/12/17 06:59:48 Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/17 06:59:48 App: TestEventingRebKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/17 06:59:48 No eventing-con process running
2021/12/17 06:59:48 Function: TestEventingRebKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpOneByOne not found"
 }
}
2021/12/17 06:59:50 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 06:59:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 06:59:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 06:59:57 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:00:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 07:00:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:00:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 07:00:07 Deploying app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne. Item count src bucket: 0 dst bucket: 0
2021/12/17 07:00:08 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebKVOpsOnUpdateBucketOpOneByOne' in metakv",
  "warnings": null
 }
}
2021/12/17 07:00:13 Starting bucket ops to source bucket
2021/12/17 07:00:18 Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/17 07:00:23 Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/17 07:00:28 Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/17 07:00:28 App: TestEventingRebKVOpsOnUpdateBucketOpOneByOne got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:00:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:00:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 07:00:28 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/17 07:00:28 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/17 07:00:28 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/17 07:00:34 Rebalance progress: 16.66666666666667
2021/12/17 07:00:39 Rebalance progress: 16.66666666666667
2021/12/17 07:00:44 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:00:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:00:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 07:00:44 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/17 07:00:44 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/17 07:00:44 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/17 07:00:49 Rebalance progress: 16.66666666666667
2021/12/17 07:00:54 Rebalance progress: 16.66666666666667
2021/12/17 07:00:59 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:00:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:00:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 07:00:59 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/17 07:00:59 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/17 07:01:04 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/17 07:01:10 Rebalance progress: 12.5
2021/12/17 07:01:15 Rebalance progress: 12.5
2021/12/17 07:01:20 Rebalance progress: 12.5
2021/12/17 07:01:25 Rebalance progress: 62.70491803278689
2021/12/17 07:01:30 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:01:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:01:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 07:01:30 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/17 07:01:35 Rebalance progress: 12.5
2021/12/17 07:01:40 Rebalance progress: 12.5
2021/12/17 07:01:45 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:01:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:01:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 07:01:45 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/17 07:01:50 Rebalance progress: 12.5
2021/12/17 07:01:55 Rebalance progress: 12.5
2021/12/17 07:02:00 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:02:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:02:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 07:02:00 clusterSetup.go:253: Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/17 07:02:05 Rebalance progress: 12.5
2021/12/17 07:02:10 Rebalance progress: 12.5
2021/12/17 07:02:15 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:02:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:02:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 07:02:15 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 07:02:20 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/17 07:02:25 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/17 07:02:30 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/17 07:02:35 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/17 07:02:35 clusterSetup.go:405: App: TestEventingRebKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/17 07:02:35 eventingSetup.go:46: No eventing-con process running
2021/12/17 07:02:35 eventingSetup.go:418: Function: TestEventingRebKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpOneByOne deleting in the background"
}
2021/12/17 07:02:37 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:02:42 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 07:02:47 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12175
2021/12/17 07:02:47 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 07:02:49 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:02:54 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 07:02:59 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12175
2021/12/17 07:02:59 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebKVOpsOnUpdateBucketOpOneByOne (200.24s)
=== RUN   TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce
2021/12/17 07:03:09 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get un-deployed
2021/12/17 07:03:09 clusterSetup.go:405: App: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce got un-deployed
2021/12/17 07:03:09 eventingSetup.go:46: No eventing-con process running
2021/12/17 07:03:09 eventingSetup.go:418: Function: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce not found"
 }
}
2021/12/17 07:03:10 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:03:15 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 07:03:15 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 07:03:17 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:03:22 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 07:03:22 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:03:27 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 07:03:27 eventingSetup.go:125: Deploying app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce. Item count src bucket: 0 dst bucket: 0
2021/12/17 07:03:28 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce' in metakv",
  "warnings": null
 }
}
2021/12/17 07:03:33 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/17 07:03:38 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get deployed
2021/12/17 07:03:43 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get deployed
2021/12/17 07:03:48 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get deployed
2021/12/17 07:03:48 clusterSetup.go:354: App: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:03:48 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:03:48 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 07:03:48 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/17 07:03:49 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/17 07:03:49 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/17 07:03:49 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/17 07:03:54 Rebalance progress: 12.5
2021/12/17 07:03:59 Rebalance progress: 12.5
2021/12/17 07:04:04 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:04:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:04:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 07:04:04 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002 127.0.0.1:19003] from the cluster
2021/12/17 07:04:09 Rebalance progress: 12.5
2021/12/17 07:04:14 Rebalance progress: 12.5
2021/12/17 07:04:19 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:04:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:04:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 07:04:19 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 07:04:24 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get un-deployed
2021/12/17 07:04:29 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get un-deployed
2021/12/17 07:04:34 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get un-deployed
2021/12/17 07:04:39 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get un-deployed
2021/12/17 07:04:39 clusterSetup.go:405: App: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce got un-deployed
2021/12/17 07:04:39 eventingSetup.go:46: No eventing-con process running
2021/12/17 07:04:39 eventingSetup.go:418: Function: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce deleting in the background"
}
2021/12/17 07:04:41 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:04:46 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 07:04:51 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4572
2021/12/17 07:04:56 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4572
2021/12/17 07:04:56 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 07:04:58 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:05:03 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 07:05:08 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4572
2021/12/17 07:05:08 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce (129.35s)
=== RUN   TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings
2021/12/17 07:05:18 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/17 07:05:18 clusterSetup.go:405: App: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings got un-deployed
2021/12/17 07:05:18 eventingSetup.go:46: No eventing-con process running
2021/12/17 07:05:18 eventingSetup.go:418: Function: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings not found"
 }
}
2021/12/17 07:05:20 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:05:25 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 07:05:25 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 07:05:27 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:05:32 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 07:05:32 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:05:37 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 07:05:37 eventingSetup.go:125: Deploying app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/17 07:05:38 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/17 07:05:43 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/17 07:05:48 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get deployed
2021/12/17 07:05:53 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get deployed
2021/12/17 07:05:58 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get deployed
2021/12/17 07:05:58 clusterSetup.go:354: App: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:05:58 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:05:58 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 07:05:58 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/17 07:05:58 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/17 07:06:03 Rebalance progress: 12.5
2021/12/17 07:06:08 Rebalance progress: 12.5
2021/12/17 07:06:13 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:06:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:06:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 07:06:13 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/17 07:06:13 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/17 07:06:18 Rebalance progress: 12.5
2021/12/17 07:06:23 Rebalance progress: 12.5
2021/12/17 07:06:28 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:06:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:06:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 07:06:28 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/17 07:06:29 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/17 07:06:34 Rebalance progress: 12.5
2021/12/17 07:06:39 Rebalance progress: 12.5
2021/12/17 07:06:44 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:06:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:06:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 07:06:44 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/17 07:06:49 Rebalance progress: 12.5
2021/12/17 07:06:54 Rebalance progress: 12.5
2021/12/17 07:06:59 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:06:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:06:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 07:06:59 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/17 07:07:04 Rebalance progress: 12.5
2021/12/17 07:07:09 Rebalance progress: 12.5
2021/12/17 07:07:14 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:07:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:07:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 07:07:14 clusterSetup.go:253: Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/17 07:07:19 Rebalance progress: 12.5
2021/12/17 07:07:24 Rebalance progress: 12.5
2021/12/17 07:07:29 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:07:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:07:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 07:07:29 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 07:07:34 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/17 07:07:39 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/17 07:07:44 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/17 07:07:49 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/17 07:07:49 clusterSetup.go:405: App: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings got un-deployed
2021/12/17 07:07:50 eventingSetup.go:46: No eventing-con process running
2021/12/17 07:07:50 eventingSetup.go:418: Function: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings deleting in the background"
}
2021/12/17 07:07:51 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:07:56 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 07:08:01 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 10651
2021/12/17 07:08:06 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 10651
2021/12/17 07:08:06 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 07:08:08 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:08:13 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 07:08:18 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 10650
2021/12/17 07:08:18 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings (190.08s)
=== RUN   TestEventingSwapRebOnUpdateBucketOp
2021/12/17 07:08:28 clusterSetup.go:397: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get un-deployed
2021/12/17 07:08:28 clusterSetup.go:405: App: TestEventingSwapRebOnUpdateBucketOp got un-deployed
2021/12/17 07:08:28 eventingSetup.go:46: No eventing-con process running
2021/12/17 07:08:28 eventingSetup.go:418: Function: TestEventingSwapRebOnUpdateBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingSwapRebOnUpdateBucketOp not found"
 }
}
2021/12/17 07:08:30 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:08:35 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 07:08:35 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 07:08:37 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:08:42 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 07:08:42 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:08:42 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 07:08:42 eventingSetup.go:125: Deploying app: TestEventingSwapRebOnUpdateBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/17 07:08:43 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingSwapRebOnUpdateBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/17 07:08:48 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/17 07:08:53 clusterSetup.go:346: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get deployed
2021/12/17 07:08:58 clusterSetup.go:346: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get deployed
2021/12/17 07:09:03 clusterSetup.go:346: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get deployed
2021/12/17 07:09:03 clusterSetup.go:354: App: TestEventingSwapRebOnUpdateBucketOp got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:09:03 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:09:03 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 07:09:03 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/17 07:09:03 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/17 07:09:08 Rebalance progress: 12.5
2021/12/17 07:09:13 Rebalance progress: 12.5
2021/12/17 07:09:18 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:09:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:09:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 07:09:18 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/17 07:09:19 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/17 07:09:24 Rebalance progress: 12.5
2021/12/17 07:09:29 Rebalance progress: 12.5
2021/12/17 07:09:34 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:09:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:09:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 07:09:34 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/17 07:09:39 Rebalance progress: 12.5
2021/12/17 07:09:44 Rebalance progress: 12.5
2021/12/17 07:09:49 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:09:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:09:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 07:09:49 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 07:09:54 clusterSetup.go:397: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get un-deployed
2021/12/17 07:09:59 clusterSetup.go:397: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get un-deployed
2021/12/17 07:10:04 clusterSetup.go:397: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get un-deployed
2021/12/17 07:10:09 clusterSetup.go:397: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get un-deployed
2021/12/17 07:10:09 clusterSetup.go:405: App: TestEventingSwapRebOnUpdateBucketOp got un-deployed
2021/12/17 07:10:09 eventingSetup.go:46: No eventing-con process running
2021/12/17 07:10:09 eventingSetup.go:418: Function: TestEventingSwapRebOnUpdateBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingSwapRebOnUpdateBucketOp deleting in the background"
}
2021/12/17 07:10:11 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:10:16 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 07:10:21 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5337
2021/12/17 07:10:26 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5337
2021/12/17 07:10:26 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 07:10:28 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:10:33 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 07:10:38 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6105
2021/12/17 07:10:38 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingSwapRebOnUpdateBucketOp (139.53s)
=== RUN   TestMetaRollbackWithEventingReb
2021/12/17 07:10:43 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/17 07:10:43 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/17 07:10:48 Rebalance progress: 100
2021/12/17 07:10:53 Waiting for app: TestMetaRollbackWithEventingReb to get un-deployed
2021/12/17 07:10:53 App: TestMetaRollbackWithEventingReb got un-deployed
2021/12/17 07:10:53 No eventing-con process running
2021/12/17 07:10:53 Function: TestMetaRollbackWithEventingReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMetaRollbackWithEventingReb not found"
 }
}
2021/12/17 07:10:55 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:11:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 07:11:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 07:11:01 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:11:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 07:11:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:11:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 07:11:06 Deploying app: TestMetaRollbackWithEventingReb. Item count src bucket: 0 dst bucket: 0
2021/12/17 07:11:08 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMetaRollbackWithEventingReb' in metakv",
  "warnings": null
 }
}
2021/12/17 07:11:13 Starting bucket ops to source bucket
2021/12/17 07:11:18 Waiting for app: TestMetaRollbackWithEventingReb to get deployed
2021/12/17 07:11:23 Waiting for app: TestMetaRollbackWithEventingReb to get deployed
2021/12/17 07:11:28 Waiting for app: TestMetaRollbackWithEventingReb to get deployed
2021/12/17 07:11:28 App: TestMetaRollbackWithEventingReb got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:11:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:11:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 07:11:28 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/17 07:11:28 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/17 07:11:33 Rebalance progress: 12.5
2021/12/17 07:11:38 Rebalance progress: 12.5
2021/12/17 07:11:43 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:11:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:11:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 07:11:43 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002] from the cluster
2021/12/17 07:11:43 bucketOps.go:134: Metakv lookup failed, err: Get "/_metakv/eventing/tempApps/TestMetaRollbackWithEventingReb/0": Unable to initialize cbauth's revrpc: cbauth environment variable CBAUTH_REVRPC_URL is not set
2021/12/17 07:11:48 Rebalance progress: 12.5
2021/12/17 07:11:53 Rebalance progress: 12.5
2021/12/17 07:11:58 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:11:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:11:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 07:11:58 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 07:12:03 clusterSetup.go:397: Waiting for app: TestMetaRollbackWithEventingReb to get un-deployed
2021/12/17 07:12:08 clusterSetup.go:397: Waiting for app: TestMetaRollbackWithEventingReb to get un-deployed
2021/12/17 07:12:13 clusterSetup.go:397: Waiting for app: TestMetaRollbackWithEventingReb to get un-deployed
2021/12/17 07:12:18 clusterSetup.go:397: Waiting for app: TestMetaRollbackWithEventingReb to get un-deployed
2021/12/17 07:12:18 clusterSetup.go:405: App: TestMetaRollbackWithEventingReb got un-deployed
2021/12/17 07:12:18 eventingSetup.go:46: No eventing-con process running
2021/12/17 07:12:18 eventingSetup.go:418: Function: TestMetaRollbackWithEventingReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMetaRollbackWithEventingReb deleting in the background"
}
2021/12/17 07:12:20 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:12:25 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 07:12:30 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 614
2021/12/17 07:12:35 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 614
2021/12/17 07:12:35 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 07:12:37 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:12:42 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 07:12:47 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4537
2021/12/17 07:12:47 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestMetaRollbackWithEventingReb (129.25s)
=== RUN   TestMetaPartialRollbackWithEventingReb
2021/12/17 07:12:52 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/17 07:12:52 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/17 07:12:57 Rebalance progress: 100
2021/12/17 07:13:02 Waiting for app: TestMetaPartialRollbackWithEventingReb to get un-deployed
2021/12/17 07:13:02 App: TestMetaPartialRollbackWithEventingReb got un-deployed
2021/12/17 07:13:02 No eventing-con process running
2021/12/17 07:13:02 Function: TestMetaPartialRollbackWithEventingReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMetaPartialRollbackWithEventingReb not found"
 }
}
2021/12/17 07:13:04 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:13:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 07:13:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 07:13:11 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:13:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 07:13:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:13:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 07:13:16 Deploying app: TestMetaPartialRollbackWithEventingReb. Item count src bucket: 0 dst bucket: 0
2021/12/17 07:13:17 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMetaPartialRollbackWithEventingReb' in metakv",
  "warnings": null
 }
}
2021/12/17 07:13:22 Starting bucket ops to source bucket
2021/12/17 07:13:27 Waiting for app: TestMetaPartialRollbackWithEventingReb to get deployed
2021/12/17 07:13:32 Waiting for app: TestMetaPartialRollbackWithEventingReb to get deployed
2021/12/17 07:13:37 Waiting for app: TestMetaPartialRollbackWithEventingReb to get deployed
2021/12/17 07:13:37 App: TestMetaPartialRollbackWithEventingReb got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:13:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:13:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 07:13:37 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/17 07:13:37 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/17 07:13:43 Rebalance progress: 12.5
2021/12/17 07:13:48 Rebalance progress: 12.5
2021/12/17 07:13:53 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:13:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:13:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 07:13:53 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002] from the cluster
2021/12/17 07:13:53 bucketOps.go:65: Metakv lookup failed, err: Get "/_metakv/eventing/tempApps/TestMetaPartialRollbackWithEventingReb/0": Unable to initialize cbauth's revrpc: cbauth environment variable CBAUTH_REVRPC_URL is not set
2021/12/17 07:13:58 Rebalance progress: 12.5
2021/12/17 07:14:03 Rebalance progress: 12.5
2021/12/17 07:14:08 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:14:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:14:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 07:14:08 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 07:14:08 bucketOps.go:65: Metakv lookup failed, err: Get "/_metakv/eventing/tempApps/TestMetaPartialRollbackWithEventingReb/0": Unable to initialize cbauth's revrpc: cbauth environment variable CBAUTH_REVRPC_URL is not set
2021/12/17 07:14:13 clusterSetup.go:397: Waiting for app: TestMetaPartialRollbackWithEventingReb to get un-deployed
2021/12/17 07:14:18 clusterSetup.go:397: Waiting for app: TestMetaPartialRollbackWithEventingReb to get un-deployed
2021/12/17 07:14:23 clusterSetup.go:397: Waiting for app: TestMetaPartialRollbackWithEventingReb to get un-deployed
2021/12/17 07:14:28 clusterSetup.go:397: Waiting for app: TestMetaPartialRollbackWithEventingReb to get un-deployed
2021/12/17 07:14:28 clusterSetup.go:405: App: TestMetaPartialRollbackWithEventingReb got un-deployed
2021/12/17 07:14:28 eventingSetup.go:46: No eventing-con process running
2021/12/17 07:14:28 eventingSetup.go:418: Function: TestMetaPartialRollbackWithEventingReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMetaPartialRollbackWithEventingReb deleting in the background"
}
2021/12/17 07:14:30 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:14:35 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 07:14:35 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 07:14:36 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:14:41 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 07:14:46 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 37806
2021/12/17 07:14:46 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestMetaPartialRollbackWithEventingReb (119.54s)
=== RUN   TestEventingRebMultipleHandlersOneByOne
2021/12/17 07:14:56 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get un-deployed
2021/12/17 07:14:56 clusterSetup.go:405: App: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op got un-deployed
2021/12/17 07:14:56 eventingSetup.go:46: No eventing-con process running
2021/12/17 07:14:56 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op not found"
 }
}
2021/12/17 07:14:58 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:15:03 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 07:15:03 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 07:15:05 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:15:10 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 07:15:10 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:15:15 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 07:15:15 eventingSetup.go:125: Deploying app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op. Item count src bucket: 0 dst bucket: 0
2021/12/17 07:15:16 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op' in metakv",
  "warnings": null
 }
}
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:15:16 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 07:15:16 eventingSetup.go:125: Deploying app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/17 07:15:17 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebMultipleHandlersOneByOnebucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/17 07:15:22 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/17 07:15:27 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get deployed
2021/12/17 07:15:32 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get deployed
2021/12/17 07:15:37 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get deployed
2021/12/17 07:15:37 clusterSetup.go:354: App: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op got deployed
2021/12/17 07:15:42 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get deployed
2021/12/17 07:15:47 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get deployed
2021/12/17 07:15:52 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get deployed
2021/12/17 07:15:52 clusterSetup.go:354: App: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:15:52 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:15:52 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 07:15:52 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/17 07:15:53 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/17 07:15:58 Rebalance progress: 12.5
2021/12/17 07:16:03 Rebalance progress: 12.5
2021/12/17 07:16:08 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:16:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:16:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 07:16:08 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/17 07:16:08 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/17 07:16:13 Rebalance progress: 12.5
2021/12/17 07:16:18 Rebalance progress: 12.5
2021/12/17 07:16:23 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:16:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:16:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 07:16:23 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/17 07:16:23 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/17 07:16:28 Rebalance progress: 12.5
2021/12/17 07:16:33 Rebalance progress: 12.5
2021/12/17 07:16:38 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:16:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:16:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 07:16:38 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/17 07:16:44 Rebalance progress: 12.5
2021/12/17 07:16:49 Rebalance progress: 12.5
2021/12/17 07:16:54 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:16:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:16:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 07:16:54 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/17 07:16:59 Rebalance progress: 12.5
2021/12/17 07:17:04 Rebalance progress: 12.5
2021/12/17 07:17:09 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:17:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:17:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 07:17:09 clusterSetup.go:253: Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/17 07:17:14 Rebalance progress: 12.5
2021/12/17 07:17:19 Rebalance progress: 12.5
2021/12/17 07:17:24 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:17:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:17:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 07:17:24 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 07:17:29 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get un-deployed
2021/12/17 07:17:34 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get un-deployed
2021/12/17 07:17:39 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get un-deployed
2021/12/17 07:17:44 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get un-deployed
2021/12/17 07:17:44 clusterSetup.go:405: App: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op got un-deployed
2021/12/17 07:17:44 eventingSetup.go:46: No eventing-con process running
2021/12/17 07:17:44 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op deleting in the background"
}
2021/12/17 07:17:48 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:17:53 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 07:17:58 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12141
2021/12/17 07:17:58 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 07:17:59 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:18:04 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 07:18:04 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 07:18:05 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 07:18:10 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get un-deployed
2021/12/17 07:18:15 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get un-deployed
2021/12/17 07:18:20 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get un-deployed
2021/12/17 07:18:25 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get un-deployed
2021/12/17 07:18:25 clusterSetup.go:405: App: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update got un-deployed
2021/12/17 07:18:25 eventingSetup.go:46: No eventing-con process running
2021/12/17 07:18:25 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update deleting in the background"
}
2021/12/17 07:18:26 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:18:31 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 07:18:31 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 07:18:33 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:18:38 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 07:18:38 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebMultipleHandlersOneByOne (231.76s)
=== RUN   TestEventingRebMultipleHandlersAllAtOnce
2021/12/17 07:18:48 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get un-deployed
2021/12/17 07:18:48 clusterSetup.go:405: App: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op got un-deployed
2021/12/17 07:18:48 eventingSetup.go:46: No eventing-con process running
2021/12/17 07:18:48 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op not found"
 }
}
2021/12/17 07:18:50 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:18:55 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 07:18:55 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 07:18:57 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:19:02 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 07:19:02 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:19:07 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 07:19:07 eventingSetup.go:125: Deploying app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op. Item count src bucket: 0 dst bucket: 0
2021/12/17 07:19:08 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op' in metakv",
  "warnings": null
 }
}
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:19:08 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 07:19:08 eventingSetup.go:125: Deploying app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/17 07:19:09 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/17 07:19:14 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/17 07:19:19 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get deployed
2021/12/17 07:19:24 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get deployed
2021/12/17 07:19:29 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get deployed
2021/12/17 07:19:29 clusterSetup.go:354: App: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op got deployed
2021/12/17 07:19:34 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get deployed
2021/12/17 07:19:39 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get deployed
2021/12/17 07:19:44 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get deployed
2021/12/17 07:19:44 clusterSetup.go:354: App: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:19:44 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:19:44 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 07:19:44 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/17 07:19:44 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/17 07:19:45 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/17 07:19:45 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/17 07:19:50 Rebalance progress: 12.5
2021/12/17 07:19:55 Rebalance progress: 12.5
2021/12/17 07:20:00 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:20:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:20:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 07:20:00 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002 127.0.0.1:19003] from the cluster
2021/12/17 07:20:05 Rebalance progress: 12.5
2021/12/17 07:20:10 Rebalance progress: 12.5
2021/12/17 07:20:15 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:20:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:20:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 07:20:16 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 07:20:21 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get un-deployed
2021/12/17 07:20:26 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get un-deployed
2021/12/17 07:20:31 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get un-deployed
2021/12/17 07:20:36 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get un-deployed
2021/12/17 07:20:36 clusterSetup.go:405: App: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op got un-deployed
2021/12/17 07:20:36 eventingSetup.go:46: No eventing-con process running
2021/12/17 07:20:36 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op deleting in the background"
}
2021/12/17 07:20:39 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:20:44 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 07:20:49 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6129
2021/12/17 07:20:49 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 07:20:51 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:20:56 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 07:21:01 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4069
2021/12/17 07:21:06 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4069
2021/12/17 07:21:06 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 07:21:06 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 07:21:11 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get un-deployed
2021/12/17 07:21:16 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get un-deployed
2021/12/17 07:21:21 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get un-deployed
2021/12/17 07:21:26 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get un-deployed
2021/12/17 07:21:26 clusterSetup.go:405: App: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update got un-deployed
2021/12/17 07:21:26 eventingSetup.go:46: No eventing-con process running
2021/12/17 07:21:26 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update deleting in the background"
}
2021/12/17 07:21:28 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:21:33 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 07:21:33 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 07:21:34 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:21:39 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 07:21:39 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebMultipleHandlersAllAtOnce (181.27s)
=== RUN   TestEventingSwapRebMultipleHandlers
2021/12/17 07:21:49 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get un-deployed
2021/12/17 07:21:49 clusterSetup.go:405: App: TestEventingSwapRebMultipleHandlersbucket_op_on_update got un-deployed
2021/12/17 07:21:49 eventingSetup.go:46: No eventing-con process running
2021/12/17 07:21:49 eventingSetup.go:418: Function: TestEventingSwapRebMultipleHandlersbucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingSwapRebMultipleHandlersbucket_op_on_update not found"
 }
}
2021/12/17 07:21:51 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:21:56 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 07:21:56 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 07:21:58 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:22:03 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 07:22:03 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 07:22:08 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get un-deployed
2021/12/17 07:22:08 clusterSetup.go:405: App: TestEventingSwapRebMultipleHandlerssys_test_bucket_op got un-deployed
2021/12/17 07:22:08 eventingSetup.go:46: No eventing-con process running
2021/12/17 07:22:08 eventingSetup.go:418: Function: TestEventingSwapRebMultipleHandlerssys_test_bucket_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingSwapRebMultipleHandlerssys_test_bucket_op not found"
 }
}
2021/12/17 07:22:10 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:22:15 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 07:22:15 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 07:22:17 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:22:22 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 07:22:22 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:22:22 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 07:22:22 eventingSetup.go:125: Deploying app: TestEventingSwapRebMultipleHandlersbucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/17 07:22:23 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingSwapRebMultipleHandlersbucket_op_on_update' in metakv",
  "warnings": null
 }
}
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:22:23 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 07:22:23 eventingSetup.go:125: Deploying app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op. Item count src bucket: 0 dst bucket: 0
2021/12/17 07:22:24 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingSwapRebMultipleHandlerssys_test_bucket_op' in metakv",
  "warnings": null
 }
}
2021/12/17 07:22:29 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/17 07:22:34 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get deployed
2021/12/17 07:22:39 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get deployed
2021/12/17 07:22:44 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get deployed
2021/12/17 07:22:44 clusterSetup.go:354: App: TestEventingSwapRebMultipleHandlersbucket_op_on_update got deployed
2021/12/17 07:22:49 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get deployed
2021/12/17 07:22:54 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get deployed
2021/12/17 07:22:59 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get deployed
2021/12/17 07:22:59 clusterSetup.go:354: App: TestEventingSwapRebMultipleHandlerssys_test_bucket_op got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:22:59 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:22:59 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 07:22:59 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/17 07:22:59 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/17 07:23:04 Rebalance progress: 12.5
2021/12/17 07:23:09 Rebalance progress: 12.5
2021/12/17 07:23:14 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:23:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:23:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 07:23:14 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/17 07:23:15 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/17 07:23:20 Rebalance progress: 12.5
2021/12/17 07:23:25 Rebalance progress: 12.5
2021/12/17 07:23:30 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:23:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:23:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 07:23:30 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/17 07:23:35 Rebalance progress: 12.5
2021/12/17 07:23:40 Rebalance progress: 12.5
2021/12/17 07:23:45 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:23:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:23:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 07:23:45 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 07:23:50 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get un-deployed
2021/12/17 07:23:55 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get un-deployed
2021/12/17 07:24:00 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get un-deployed
2021/12/17 07:24:05 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get un-deployed
2021/12/17 07:24:05 clusterSetup.go:405: App: TestEventingSwapRebMultipleHandlersbucket_op_on_update got un-deployed
2021/12/17 07:24:05 eventingSetup.go:46: No eventing-con process running
2021/12/17 07:24:05 eventingSetup.go:418: Function: TestEventingSwapRebMultipleHandlersbucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingSwapRebMultipleHandlersbucket_op_on_update deleting in the background"
}
2021/12/17 07:24:09 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:24:14 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 07:24:19 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7613
2021/12/17 07:24:19 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 07:24:21 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:24:26 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 07:24:31 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5428
2021/12/17 07:24:36 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5428
2021/12/17 07:24:36 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 07:24:36 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 07:24:41 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get un-deployed
2021/12/17 07:24:46 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get un-deployed
2021/12/17 07:24:51 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get un-deployed
2021/12/17 07:24:56 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get un-deployed
2021/12/17 07:24:56 clusterSetup.go:405: App: TestEventingSwapRebMultipleHandlerssys_test_bucket_op got un-deployed
2021/12/17 07:24:56 eventingSetup.go:46: No eventing-con process running
2021/12/17 07:24:56 eventingSetup.go:418: Function: TestEventingSwapRebMultipleHandlerssys_test_bucket_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingSwapRebMultipleHandlerssys_test_bucket_op deleting in the background"
}
2021/12/17 07:24:58 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:25:03 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 07:25:03 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 07:25:04 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:25:09 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 07:25:09 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingSwapRebMultipleHandlers (210.03s)
=== RUN   TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne
2021/12/17 07:25:19 clusterSetup.go:397: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/17 07:25:19 clusterSetup.go:405: App: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/17 07:25:19 eventingSetup.go:46: No eventing-con process running
2021/12/17 07:25:19 eventingSetup.go:418: Function: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne not found"
 }
}
2021/12/17 07:25:21 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:25:26 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 07:25:26 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 07:25:28 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:25:33 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 07:25:33 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:25:38 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 07:25:38 eventingSetup.go:125: Deploying app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne. Item count src bucket: 0 dst bucket: 0
2021/12/17 07:25:39 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne' in metakv",
  "warnings": null
 }
}
2021/12/17 07:25:44 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/17 07:25:49 clusterSetup.go:346: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/17 07:25:54 clusterSetup.go:346: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/17 07:25:59 clusterSetup.go:346: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/17 07:25:59 clusterSetup.go:354: App: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:25:59 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:25:59 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 07:25:59 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/17 07:25:59 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/17 07:26:19 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:26:20 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:26:20 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:26:50 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:26:50 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 07:26:50 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/17 07:26:55 Rebalance progress: 12.5
2021/12/17 07:27:00 Rebalance progress: 12.5
2021/12/17 07:27:05 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:27:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:27:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 07:27:05 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/17 07:27:06 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/17 07:27:11 Rebalance progress: 12.5
2021/12/17 07:27:16 Rebalance progress: 12.5
2021/12/17 07:27:21 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:27:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:27:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 07:27:21 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/17 07:27:26 Rebalance progress: 12.5
2021/12/17 07:27:31 Rebalance progress: 12.5
2021/12/17 07:27:36 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:27:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:27:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 07:27:36 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 07:27:41 clusterSetup.go:397: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/17 07:27:46 clusterSetup.go:397: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/17 07:27:51 clusterSetup.go:397: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/17 07:27:56 clusterSetup.go:397: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/17 07:27:56 clusterSetup.go:405: App: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/17 07:27:56 eventingSetup.go:46: No eventing-con process running
2021/12/17 07:27:56 eventingSetup.go:418: Function: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne deleting in the background"
}
2021/12/17 07:27:58 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:28:03 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 07:28:08 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 11152
2021/12/17 07:28:08 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 07:28:10 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:28:15 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 07:28:15 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne (185.08s)
=== RUN   TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne
2021/12/17 07:28:25 clusterSetup.go:397: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/17 07:28:25 clusterSetup.go:405: App: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/17 07:28:25 eventingSetup.go:46: No eventing-con process running
2021/12/17 07:28:25 eventingSetup.go:418: Function: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne not found"
 }
}
2021/12/17 07:28:26 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:28:31 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 07:28:31 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 07:28:33 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:28:38 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 07:28:38 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:28:43 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 07:28:43 eventingSetup.go:125: Deploying app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne. Item count src bucket: 0 dst bucket: 0
2021/12/17 07:28:44 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne' in metakv",
  "warnings": null
 }
}
2021/12/17 07:28:49 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/17 07:28:54 clusterSetup.go:346: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/17 07:28:59 clusterSetup.go:346: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/17 07:29:04 clusterSetup.go:346: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/17 07:29:04 clusterSetup.go:354: App: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:29:04 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:29:04 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 07:29:04 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/17 07:29:04 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/17 07:29:25 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:29:25 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:29:25 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:29:35 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:29:35 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 07:29:35 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/17 07:29:35 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/17 07:29:55 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:29:56 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:29:56 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:30:06 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:30:06 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 07:30:06 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/17 07:30:06 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/17 07:30:26 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:30:27 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:30:27 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:30:37 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:30:37 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 07:30:37 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/17 07:30:38 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/17 07:30:58 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:30:58 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:30:58 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:31:08 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:31:08 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 07:31:08 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/17 07:31:09 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/17 07:31:29 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:31:29 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:31:29 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:31:39 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:31:39 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 07:31:39 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/17 07:31:44 Rebalance progress: 12.5
2021/12/17 07:31:49 Rebalance progress: 12.5
2021/12/17 07:31:54 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:31:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:31:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 07:31:54 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/17 07:31:54 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/17 07:32:00 Rebalance progress: 12.5
2021/12/17 07:32:05 Rebalance progress: 12.5
2021/12/17 07:32:10 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:32:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:32:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 07:32:10 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/17 07:32:15 Rebalance progress: 12.5
2021/12/17 07:32:20 Rebalance progress: 12.5
2021/12/17 07:32:25 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:32:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:32:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 07:32:25 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 07:32:30 clusterSetup.go:397: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/17 07:32:35 clusterSetup.go:397: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/17 07:32:40 clusterSetup.go:397: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/17 07:32:45 clusterSetup.go:397: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/17 07:32:45 clusterSetup.go:405: App: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/17 07:32:45 eventingSetup.go:46: No eventing-con process running
2021/12/17 07:32:45 eventingSetup.go:418: Function: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne deleting in the background"
}
2021/12/17 07:32:47 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:32:52 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 07:32:57 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 21524
2021/12/17 07:32:57 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 07:32:58 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:33:03 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 07:33:08 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 21524
2021/12/17 07:33:08 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne (293.94s)
=== RUN   TestEventingFailoverOnUpdateBucketOp
2021/12/17 07:33:18 clusterSetup.go:397: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get un-deployed
2021/12/17 07:33:18 clusterSetup.go:405: App: TestEventingFailoverOnUpdateBucketOp got un-deployed
2021/12/17 07:33:18 eventingSetup.go:46: No eventing-con process running
2021/12/17 07:33:18 eventingSetup.go:418: Function: TestEventingFailoverOnUpdateBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingFailoverOnUpdateBucketOp not found"
 }
}
2021/12/17 07:33:20 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:33:25 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 07:33:25 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 07:33:27 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:33:32 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 07:33:32 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:33:37 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 07:33:37 eventingSetup.go:125: Deploying app: TestEventingFailoverOnUpdateBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/17 07:33:38 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingFailoverOnUpdateBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/17 07:33:43 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/17 07:33:48 clusterSetup.go:346: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get deployed
2021/12/17 07:33:53 clusterSetup.go:346: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get deployed
2021/12/17 07:33:58 clusterSetup.go:346: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get deployed
2021/12/17 07:33:58 clusterSetup.go:354: App: TestEventingFailoverOnUpdateBucketOp got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:33:58 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:33:58 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 07:33:58 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/17 07:33:59 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/17 07:34:04 Rebalance progress: 12.5
2021/12/17 07:34:09 Rebalance progress: 12.5
2021/12/17 07:34:14 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:34:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:34:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 07:34:14 clusterSetup.go:229: Failing over: [127.0.0.1:19001]
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:34:24 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:34:24 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 07:34:24 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/17 07:34:29 Rebalance progress: 12.5
2021/12/17 07:34:34 Rebalance progress: 12.5
2021/12/17 07:34:39 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:34:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:34:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 07:34:39 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/17 07:34:39 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/17 07:34:44 Rebalance progress: 12.5
2021/12/17 07:34:49 Rebalance progress: 12.5
2021/12/17 07:34:54 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:34:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:34:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 07:34:54 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/17 07:34:59 Rebalance progress: 12.5
2021/12/17 07:35:04 Rebalance progress: 12.5
2021/12/17 07:35:09 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:35:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:35:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 07:35:09 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 07:35:14 clusterSetup.go:397: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get un-deployed
2021/12/17 07:35:19 clusterSetup.go:397: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get un-deployed
2021/12/17 07:35:24 clusterSetup.go:397: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get un-deployed
2021/12/17 07:35:30 clusterSetup.go:397: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get un-deployed
2021/12/17 07:35:30 clusterSetup.go:405: App: TestEventingFailoverOnUpdateBucketOp got un-deployed
2021/12/17 07:35:30 eventingSetup.go:46: No eventing-con process running
2021/12/17 07:35:30 eventingSetup.go:418: Function: TestEventingFailoverOnUpdateBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingFailoverOnUpdateBucketOp deleting in the background"
}
2021/12/17 07:35:31 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:35:36 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 07:35:41 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 8607
2021/12/17 07:35:46 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 8607
2021/12/17 07:35:46 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 07:35:48 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:35:53 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 07:35:58 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 8607
2021/12/17 07:35:58 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingFailoverOnUpdateBucketOp (169.62s)
=== RUN   TestEventingKVRebalanceOnUpdateBucketOp
2021/12/17 07:36:08 clusterSetup.go:397: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get un-deployed
2021/12/17 07:36:08 clusterSetup.go:405: App: TestEventingKVRebalanceOnUpdateBucketOp got un-deployed
2021/12/17 07:36:08 eventingSetup.go:46: No eventing-con process running
2021/12/17 07:36:08 eventingSetup.go:418: Function: TestEventingKVRebalanceOnUpdateBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingKVRebalanceOnUpdateBucketOp not found"
 }
}
2021/12/17 07:36:10 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:36:15 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 07:36:15 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 07:36:17 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:36:22 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 07:36:22 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:36:27 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 07:36:27 eventingSetup.go:125: Deploying app: TestEventingKVRebalanceOnUpdateBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/17 07:36:28 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingKVRebalanceOnUpdateBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/17 07:36:33 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/17 07:36:38 clusterSetup.go:346: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get deployed
2021/12/17 07:36:43 clusterSetup.go:346: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get deployed
2021/12/17 07:36:48 clusterSetup.go:346: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get deployed
2021/12/17 07:36:48 clusterSetup.go:354: App: TestEventingKVRebalanceOnUpdateBucketOp got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:36:48 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:36:48 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 07:36:48 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing,kv to the cluster
2021/12/17 07:36:49 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/17 07:36:54 Rebalance progress: 12.5
2021/12/17 07:36:59 Rebalance progress: 12.5
2021/12/17 07:37:04 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:37:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:37:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 07:37:04 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing,kv to the cluster
2021/12/17 07:37:04 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/17 07:37:09 Rebalance progress: 12.5
2021/12/17 07:37:14 Rebalance progress: 12.5
2021/12/17 07:37:19 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:37:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:37:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 07:37:19 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/17 07:37:24 Rebalance progress: 12.5
2021/12/17 07:37:29 Rebalance progress: 12.5
2021/12/17 07:37:34 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:37:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:37:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 07:37:34 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 07:37:39 clusterSetup.go:397: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get un-deployed
2021/12/17 07:37:45 clusterSetup.go:397: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get un-deployed
2021/12/17 07:37:50 clusterSetup.go:397: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get un-deployed
2021/12/17 07:37:55 clusterSetup.go:397: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get un-deployed
2021/12/17 07:37:55 clusterSetup.go:405: App: TestEventingKVRebalanceOnUpdateBucketOp got un-deployed
2021/12/17 07:37:55 eventingSetup.go:46: No eventing-con process running
2021/12/17 07:37:55 eventingSetup.go:418: Function: TestEventingKVRebalanceOnUpdateBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingKVRebalanceOnUpdateBucketOp deleting in the background"
}
2021/12/17 07:37:56 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:38:01 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 07:38:06 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6155
2021/12/17 07:38:06 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 07:38:08 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:38:13 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 07:38:18 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6155
2021/12/17 07:38:18 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingKVRebalanceOnUpdateBucketOp (140.06s)
=== RUN   TestEventingRebInWithOps
2021/12/17 07:38:28 clusterSetup.go:397: Waiting for app: TestEventingRebInWithOps to get un-deployed
2021/12/17 07:38:28 clusterSetup.go:405: App: TestEventingRebInWithOps got un-deployed
2021/12/17 07:38:28 eventingSetup.go:46: No eventing-con process running
2021/12/17 07:38:28 eventingSetup.go:418: Function: TestEventingRebInWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebInWithOps not found"
 }
}
2021/12/17 07:38:30 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:38:35 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 07:38:35 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 07:38:37 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:38:42 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 07:38:42 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:38:47 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 07:38:47 eventingSetup.go:125: Deploying app: TestEventingRebInWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/17 07:38:48 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebInWithOps' in metakv",
  "warnings": null
 }
}
2021/12/17 07:38:53 clusterSetup.go:346: Waiting for app: TestEventingRebInWithOps to get deployed
2021/12/17 07:38:58 clusterSetup.go:346: Waiting for app: TestEventingRebInWithOps to get deployed
2021/12/17 07:39:03 clusterSetup.go:346: Waiting for app: TestEventingRebInWithOps to get deployed
2021/12/17 07:39:08 clusterSetup.go:346: Waiting for app: TestEventingRebInWithOps to get deployed
2021/12/17 07:39:08 clusterSetup.go:354: App: TestEventingRebInWithOps got deployed
2021/12/17 07:39:08 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/17 07:39:08 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/17 07:39:08 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/17 07:39:13 Rebalance progress: 12.5
2021/12/17 07:39:18 Rebalance progress: 12.5
2021/12/17 07:39:23 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:39:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:39:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 07:39:24 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/17 07:39:29 Rebalance progress: 12.5
2021/12/17 07:39:34 Rebalance progress: 12.5
2021/12/17 07:39:39 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:39:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:39:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 07:39:39 Waiting for src bucket item count to get to: 200000 curr count: 32791
2021/12/17 07:39:44 Waiting for src bucket item count to get to: 200000 curr count: 48652
2021/12/17 07:39:49 Waiting for src bucket item count to get to: 200000 curr count: 48652
2021/12/17 07:39:54 Waiting for src bucket item count to get to: 200000 curr count: 64743
2021/12/17 07:39:59 Waiting for src bucket item count to get to: 200000 curr count: 64743
2021/12/17 07:40:04 Waiting for src bucket item count to get to: 200000 curr count: 80043
2021/12/17 07:40:09 Waiting for src bucket item count to get to: 200000 curr count: 80043
2021/12/17 07:40:14 Waiting for src bucket item count to get to: 200000 curr count: 95840
2021/12/17 07:40:19 Waiting for src bucket item count to get to: 200000 curr count: 95840
2021/12/17 07:40:24 Waiting for src bucket item count to get to: 200000 curr count: 111899
2021/12/17 07:40:29 Waiting for src bucket item count to get to: 200000 curr count: 111899
2021/12/17 07:40:34 Waiting for src bucket item count to get to: 200000 curr count: 127943
2021/12/17 07:40:39 Waiting for src bucket item count to get to: 200000 curr count: 127943
2021/12/17 07:40:44 Waiting for src bucket item count to get to: 200000 curr count: 127943
2021/12/17 07:40:49 Waiting for src bucket item count to get to: 200000 curr count: 144135
2021/12/17 07:40:54 Waiting for src bucket item count to get to: 200000 curr count: 144135
2021/12/17 07:40:59 Waiting for src bucket item count to get to: 200000 curr count: 160414
2021/12/17 07:41:04 Waiting for src bucket item count to get to: 200000 curr count: 160414
2021/12/17 07:41:09 Waiting for src bucket item count to get to: 200000 curr count: 175618
2021/12/17 07:41:14 Waiting for src bucket item count to get to: 200000 curr count: 175618
2021/12/17 07:41:19 Waiting for src bucket item count to get to: 200000 curr count: 191499
2021/12/17 07:41:24 Waiting for src bucket item count to get to: 200000 curr count: 191499
2021/12/17 07:41:29 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/17 07:41:29 Post SET ops. Expected item count: 200000 got 200000
2021/12/17 07:41:29 Starting bucket ops to source bucket
2021/12/17 07:45:19 Waiting for dst bucket item count to get to: 0 curr count: 7715
2021/12/17 07:45:24 Waiting for dst bucket item count to get to: 0 curr count: 7715
2021/12/17 07:45:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 07:45:24 Post DELETE ops. Expected item count: 0 got 0
2021/12/17 07:45:24 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 07:45:29 Waiting for app: TestEventingRebInWithOps to get un-deployed
2021/12/17 07:45:34 Waiting for app: TestEventingRebInWithOps to get un-deployed
2021/12/17 07:45:39 Waiting for app: TestEventingRebInWithOps to get un-deployed
2021/12/17 07:45:44 Waiting for app: TestEventingRebInWithOps to get un-deployed
2021/12/17 07:45:44 App: TestEventingRebInWithOps got un-deployed
2021/12/17 07:45:44 No eventing-con process running
2021/12/17 07:45:44 Function: TestEventingRebInWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebInWithOps deleting in the background"
}
2021/12/17 07:45:45 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:45:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 07:45:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 07:45:52 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:45:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 07:45:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebInWithOps (459.12s)
=== RUN   TestEventingRebOutWithOps
2021/12/17 07:46:02 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/17 07:46:02 Removing node(s): [] from the cluster
2021/12/17 07:46:08 Rebalance progress: 100
2021/12/17 07:46:13 Waiting for app: TestEventingRebOutWithOps to get un-deployed
2021/12/17 07:46:13 App: TestEventingRebOutWithOps got un-deployed
2021/12/17 07:46:13 No eventing-con process running
2021/12/17 07:46:13 Function: TestEventingRebOutWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebOutWithOps not found"
 }
}
2021/12/17 07:46:14 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:46:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 07:46:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 07:46:21 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:46:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 07:46:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:46:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 07:46:31 Deploying app: TestEventingRebOutWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/17 07:46:32 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebOutWithOps' in metakv",
  "warnings": null
 }
}
2021/12/17 07:46:37 Waiting for app: TestEventingRebOutWithOps to get deployed
2021/12/17 07:46:42 Waiting for app: TestEventingRebOutWithOps to get deployed
2021/12/17 07:46:47 Waiting for app: TestEventingRebOutWithOps to get deployed
2021/12/17 07:46:52 Waiting for app: TestEventingRebOutWithOps to get deployed
2021/12/17 07:46:52 App: TestEventingRebOutWithOps got deployed
2021/12/17 07:46:52 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/17 07:46:52 Starting bucket ops to source bucket
2021/12/17 07:46:57 Rebalance progress: 12.5
2021/12/17 07:47:02 Rebalance progress: 12.5
2021/12/17 07:47:07 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:47:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:47:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 07:47:07 Waiting for src bucket item count to get to: 200000 curr count: 10335
2021/12/17 07:47:12 Waiting for src bucket item count to get to: 200000 curr count: 26081
2021/12/17 07:47:17 Waiting for src bucket item count to get to: 200000 curr count: 26081
2021/12/17 07:47:22 Waiting for src bucket item count to get to: 200000 curr count: 42144
2021/12/17 07:47:27 Waiting for src bucket item count to get to: 200000 curr count: 42144
2021/12/17 07:47:32 Waiting for src bucket item count to get to: 200000 curr count: 58458
2021/12/17 07:47:37 Waiting for src bucket item count to get to: 200000 curr count: 58458
2021/12/17 07:47:42 Waiting for src bucket item count to get to: 200000 curr count: 58458
2021/12/17 07:47:47 Waiting for src bucket item count to get to: 200000 curr count: 73603
2021/12/17 07:47:52 Waiting for src bucket item count to get to: 200000 curr count: 73603
2021/12/17 07:47:57 Waiting for src bucket item count to get to: 200000 curr count: 89433
2021/12/17 07:48:03 Waiting for src bucket item count to get to: 200000 curr count: 89433
2021/12/17 07:48:08 Waiting for src bucket item count to get to: 200000 curr count: 105538
2021/12/17 07:48:13 Waiting for src bucket item count to get to: 200000 curr count: 105538
2021/12/17 07:48:18 Waiting for src bucket item count to get to: 200000 curr count: 121735
2021/12/17 07:48:23 Waiting for src bucket item count to get to: 200000 curr count: 121735
2021/12/17 07:48:28 Waiting for src bucket item count to get to: 200000 curr count: 138123
2021/12/17 07:48:33 Waiting for src bucket item count to get to: 200000 curr count: 138123
2021/12/17 07:48:38 Waiting for src bucket item count to get to: 200000 curr count: 154353
2021/12/17 07:48:43 Waiting for src bucket item count to get to: 200000 curr count: 154353
2021/12/17 07:48:48 Waiting for src bucket item count to get to: 200000 curr count: 169685
2021/12/17 07:48:53 Waiting for src bucket item count to get to: 200000 curr count: 169685
2021/12/17 07:48:58 Waiting for src bucket item count to get to: 200000 curr count: 185601
2021/12/17 07:49:03 Waiting for src bucket item count to get to: 200000 curr count: 185601
2021/12/17 07:49:08 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/17 07:49:08 Post SET ops. Expected item count: 200000 got 200000
2021/12/17 07:49:08 Starting bucket ops to source bucket
2021/12/17 07:52:57 Waiting for dst bucket item count to get to: 0 curr count: 23171
2021/12/17 07:53:02 Waiting for dst bucket item count to get to: 0 curr count: 5050
2021/12/17 07:53:07 Waiting for dst bucket item count to get to: 0 curr count: 5050
2021/12/17 07:53:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 07:53:07 Post DELETE ops. Expected item count: 0 got 0
2021/12/17 07:53:07 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 07:53:12 Waiting for app: TestEventingRebOutWithOps to get un-deployed
2021/12/17 07:53:17 Waiting for app: TestEventingRebOutWithOps to get un-deployed
2021/12/17 07:53:22 Waiting for app: TestEventingRebOutWithOps to get un-deployed
2021/12/17 07:53:27 Waiting for app: TestEventingRebOutWithOps to get un-deployed
2021/12/17 07:53:27 App: TestEventingRebOutWithOps got un-deployed
2021/12/17 07:53:27 No eventing-con process running
2021/12/17 07:53:27 Function: TestEventingRebOutWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebOutWithOps deleting in the background"
}
2021/12/17 07:53:29 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:53:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 07:53:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 07:53:36 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:53:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 07:53:41 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebOutWithOps (463.62s)
=== RUN   TestEventingSwapRebWithOps
2021/12/17 07:53:51 Waiting for app: TestEventingSwapRebWithOps to get un-deployed
2021/12/17 07:53:51 App: TestEventingSwapRebWithOps got un-deployed
2021/12/17 07:53:51 No eventing-con process running
2021/12/17 07:53:51 Function: TestEventingSwapRebWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingSwapRebWithOps not found"
 }
}
2021/12/17 07:53:53 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:53:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 07:53:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 07:53:59 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:54:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 07:54:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:54:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 07:54:09 Deploying app: TestEventingSwapRebWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/17 07:54:11 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingSwapRebWithOps' in metakv",
  "warnings": null
 }
}
2021/12/17 07:54:16 Waiting for app: TestEventingSwapRebWithOps to get deployed
2021/12/17 07:54:21 Waiting for app: TestEventingSwapRebWithOps to get deployed
2021/12/17 07:54:26 Waiting for app: TestEventingSwapRebWithOps to get deployed
2021/12/17 07:54:31 Waiting for app: TestEventingSwapRebWithOps to get deployed
2021/12/17 07:54:31 App: TestEventingSwapRebWithOps got deployed
2021/12/17 07:54:31 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/17 07:54:31 Starting bucket ops to source bucket
2021/12/17 07:54:31 Removing node(s): [] from the cluster
2021/12/17 07:54:36 Rebalance progress: 12.5
2021/12/17 07:54:41 Rebalance progress: 12.5
2021/12/17 07:54:46 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:54:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:54:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 07:54:46 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/17 07:54:47 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/17 07:54:52 Rebalance progress: 12.5
2021/12/17 07:54:57 Rebalance progress: 12.5
2021/12/17 07:55:02 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:55:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:55:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 07:55:02 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/17 07:55:07 Rebalance progress: 12.5
2021/12/17 07:55:12 Rebalance progress: 12.5
2021/12/17 07:55:17 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:55:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 07:55:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 07:55:17 Waiting for src bucket item count to get to: 200000 curr count: 58865
2021/12/17 07:55:22 Waiting for src bucket item count to get to: 200000 curr count: 73904
2021/12/17 07:55:27 Waiting for src bucket item count to get to: 200000 curr count: 73904
2021/12/17 07:55:32 Waiting for src bucket item count to get to: 200000 curr count: 89829
2021/12/17 07:55:37 Waiting for src bucket item count to get to: 200000 curr count: 89829
2021/12/17 07:55:42 Waiting for src bucket item count to get to: 200000 curr count: 105934
2021/12/17 07:55:47 Waiting for src bucket item count to get to: 200000 curr count: 105934
2021/12/17 07:55:52 Waiting for src bucket item count to get to: 200000 curr count: 122343
2021/12/17 07:55:57 Waiting for src bucket item count to get to: 200000 curr count: 122343
2021/12/17 07:56:02 Waiting for src bucket item count to get to: 200000 curr count: 138515
2021/12/17 07:56:07 Waiting for src bucket item count to get to: 200000 curr count: 138515
2021/12/17 07:56:12 Waiting for src bucket item count to get to: 200000 curr count: 154791
2021/12/17 07:56:17 Waiting for src bucket item count to get to: 200000 curr count: 154791
2021/12/17 07:56:22 Waiting for src bucket item count to get to: 200000 curr count: 170142
2021/12/17 07:56:27 Waiting for src bucket item count to get to: 200000 curr count: 170142
2021/12/17 07:56:32 Waiting for src bucket item count to get to: 200000 curr count: 170142
2021/12/17 07:56:37 Waiting for src bucket item count to get to: 200000 curr count: 186119
2021/12/17 07:56:42 Waiting for src bucket item count to get to: 200000 curr count: 186119
2021/12/17 07:56:47 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/17 07:56:47 Post SET ops. Expected item count: 200000 got 200000
2021/12/17 07:56:47 Starting bucket ops to source bucket
2021/12/17 08:00:36 Waiting for dst bucket item count to get to: 0 curr count: 3479
2021/12/17 08:00:41 Waiting for dst bucket item count to get to: 0 curr count: 3479
2021/12/17 08:00:41 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 08:00:41 Post DELETE ops. Expected item count: 0 got 0
2021/12/17 08:00:41 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 08:00:46 Waiting for app: TestEventingSwapRebWithOps to get un-deployed
2021/12/17 08:00:51 Waiting for app: TestEventingSwapRebWithOps to get un-deployed
2021/12/17 08:00:56 Waiting for app: TestEventingSwapRebWithOps to get un-deployed
2021/12/17 08:01:01 Waiting for app: TestEventingSwapRebWithOps to get un-deployed
2021/12/17 08:01:01 App: TestEventingSwapRebWithOps got un-deployed
2021/12/17 08:01:01 No eventing-con process running
2021/12/17 08:01:01 Function: TestEventingSwapRebWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingSwapRebWithOps deleting in the background"
}
2021/12/17 08:01:03 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 08:01:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 08:01:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 08:01:10 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 08:01:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 08:01:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingSwapRebWithOps (454.05s)
=== RUN   TestKVRebInWithOps
2021/12/17 08:01:25 Waiting for app: TestKVRebInWithOps to get un-deployed
2021/12/17 08:01:25 App: TestKVRebInWithOps got un-deployed
2021/12/17 08:01:25 No eventing-con process running
2021/12/17 08:01:25 Function: TestKVRebInWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVRebInWithOps not found"
 }
}
2021/12/17 08:01:27 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 08:01:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 08:01:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 08:01:34 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 08:01:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 08:01:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 08:01:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 08:01:44 Deploying app: TestKVRebInWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/17 08:01:45 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVRebInWithOps' in metakv",
  "warnings": null
 }
}
2021/12/17 08:01:50 Waiting for app: TestKVRebInWithOps to get deployed
2021/12/17 08:01:55 Waiting for app: TestKVRebInWithOps to get deployed
2021/12/17 08:02:00 Waiting for app: TestKVRebInWithOps to get deployed
2021/12/17 08:02:05 Waiting for app: TestKVRebInWithOps to get deployed
2021/12/17 08:02:05 App: TestKVRebInWithOps got deployed
2021/12/17 08:02:05 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/17 08:02:05 Starting bucket ops to source bucket
2021/12/17 08:02:05 Removing node(s): [] from the cluster
2021/12/17 08:02:10 Rebalance progress: 12.5
2021/12/17 08:02:15 Rebalance progress: 12.5
2021/12/17 08:02:20 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 08:02:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 08:02:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 08:02:20 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/17 08:02:25 Rebalance progress: 12.5
2021/12/17 08:02:30 Rebalance progress: 12.5
2021/12/17 08:02:35 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 08:02:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 08:02:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 08:02:35 Waiting for src bucket item count to get to: 200000 curr count: 37098
2021/12/17 08:02:40 Waiting for src bucket item count to get to: 200000 curr count: 53019
2021/12/17 08:02:45 Waiting for src bucket item count to get to: 200000 curr count: 53019
2021/12/17 08:02:50 Waiting for src bucket item count to get to: 200000 curr count: 68707
2021/12/17 08:02:55 Waiting for src bucket item count to get to: 200000 curr count: 68707
2021/12/17 08:03:00 Waiting for src bucket item count to get to: 200000 curr count: 83736
2021/12/17 08:03:05 Waiting for src bucket item count to get to: 200000 curr count: 83736
2021/12/17 08:03:10 Waiting for src bucket item count to get to: 200000 curr count: 83736
2021/12/17 08:03:15 Waiting for src bucket item count to get to: 200000 curr count: 99736
2021/12/17 08:03:20 Waiting for src bucket item count to get to: 200000 curr count: 99736
2021/12/17 08:03:25 Waiting for src bucket item count to get to: 200000 curr count: 115974
2021/12/17 08:03:30 Waiting for src bucket item count to get to: 200000 curr count: 115974
2021/12/17 08:03:35 Waiting for src bucket item count to get to: 200000 curr count: 132124
2021/12/17 08:03:40 Waiting for src bucket item count to get to: 200000 curr count: 132124
2021/12/17 08:03:45 Waiting for src bucket item count to get to: 200000 curr count: 148291
2021/12/17 08:03:50 Waiting for src bucket item count to get to: 200000 curr count: 148291
2021/12/17 08:03:55 Waiting for src bucket item count to get to: 200000 curr count: 164138
2021/12/17 08:04:00 Waiting for src bucket item count to get to: 200000 curr count: 164138
2021/12/17 08:04:05 Waiting for src bucket item count to get to: 200000 curr count: 179051
2021/12/17 08:04:10 Waiting for src bucket item count to get to: 200000 curr count: 179051
2021/12/17 08:04:15 Waiting for src bucket item count to get to: 200000 curr count: 195395
2021/12/17 08:04:20 Waiting for src bucket item count to get to: 200000 curr count: 195395
2021/12/17 08:04:25 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/17 08:04:25 Post SET ops. Expected item count: 200000 got 200000
2021/12/17 08:04:25 Starting bucket ops to source bucket
2021/12/17 08:08:15 Waiting for dst bucket item count to get to: 0 curr count: 1498
2021/12/17 08:08:20 Waiting for dst bucket item count to get to: 0 curr count: 1498
2021/12/17 08:08:25 Waiting for dst bucket item count to get to: 0 curr count: 1498
2021/12/17 08:08:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 08:08:25 Post DELETE ops. Expected item count: 0 got 0
2021/12/17 08:08:25 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 08:08:30 Waiting for app: TestKVRebInWithOps to get un-deployed
2021/12/17 08:08:35 Waiting for app: TestKVRebInWithOps to get un-deployed
2021/12/17 08:08:40 Waiting for app: TestKVRebInWithOps to get un-deployed
2021/12/17 08:08:45 Waiting for app: TestKVRebInWithOps to get un-deployed
2021/12/17 08:08:45 App: TestKVRebInWithOps got un-deployed
2021/12/17 08:08:45 No eventing-con process running
2021/12/17 08:08:45 Function: TestKVRebInWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVRebInWithOps deleting in the background"
}
2021/12/17 08:08:47 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 08:08:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 08:08:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 08:08:54 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 08:08:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 08:08:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebInWithOps (463.98s)
=== RUN   TestKVRebOutWithOps
2021/12/17 08:09:04 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/17 08:09:04 Removing node(s): [] from the cluster
2021/12/17 08:09:09 Rebalance progress: 100
2021/12/17 08:09:14 Waiting for app: TestKVRebOutWithOps to get un-deployed
2021/12/17 08:09:14 App: TestKVRebOutWithOps got un-deployed
2021/12/17 08:09:14 No eventing-con process running
2021/12/17 08:09:14 Function: TestKVRebOutWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVRebOutWithOps not found"
 }
}
2021/12/17 08:09:16 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 08:09:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 08:09:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 08:09:23 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 08:09:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 08:09:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 08:09:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 08:09:33 Deploying app: TestKVRebOutWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/17 08:09:34 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVRebOutWithOps' in metakv",
  "warnings": null
 }
}
2021/12/17 08:09:39 Waiting for app: TestKVRebOutWithOps to get deployed
2021/12/17 08:09:44 Waiting for app: TestKVRebOutWithOps to get deployed
2021/12/17 08:09:49 Waiting for app: TestKVRebOutWithOps to get deployed
2021/12/17 08:09:54 Waiting for app: TestKVRebOutWithOps to get deployed
2021/12/17 08:09:54 App: TestKVRebOutWithOps got deployed
2021/12/17 08:09:54 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/17 08:09:54 Starting bucket ops to source bucket
2021/12/17 08:09:59 Rebalance progress: 12.5
2021/12/17 08:10:04 Rebalance progress: 12.5
2021/12/17 08:10:09 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 08:10:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 08:10:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 08:10:09 Waiting for src bucket item count to get to: 200000 curr count: 7236
2021/12/17 08:10:14 Waiting for src bucket item count to get to: 200000 curr count: 23211
2021/12/17 08:10:19 Waiting for src bucket item count to get to: 200000 curr count: 23211
2021/12/17 08:10:24 Waiting for src bucket item count to get to: 200000 curr count: 39257
2021/12/17 08:10:29 Waiting for src bucket item count to get to: 200000 curr count: 39257
2021/12/17 08:10:34 Waiting for src bucket item count to get to: 200000 curr count: 55608
2021/12/17 08:10:39 Waiting for src bucket item count to get to: 200000 curr count: 55608
2021/12/17 08:10:44 Waiting for src bucket item count to get to: 200000 curr count: 55608
2021/12/17 08:10:49 Waiting for src bucket item count to get to: 200000 curr count: 71623
2021/12/17 08:10:54 Waiting for src bucket item count to get to: 200000 curr count: 71623
2021/12/17 08:10:59 Waiting for src bucket item count to get to: 200000 curr count: 86523
2021/12/17 08:11:04 Waiting for src bucket item count to get to: 200000 curr count: 102467
2021/12/17 08:11:09 Waiting for src bucket item count to get to: 200000 curr count: 102467
2021/12/17 08:11:14 Waiting for src bucket item count to get to: 200000 curr count: 118180
2021/12/17 08:11:19 Waiting for src bucket item count to get to: 200000 curr count: 118180
2021/12/17 08:11:24 Waiting for src bucket item count to get to: 200000 curr count: 134152
2021/12/17 08:11:29 Waiting for src bucket item count to get to: 200000 curr count: 134152
2021/12/17 08:11:34 Waiting for src bucket item count to get to: 200000 curr count: 150139
2021/12/17 08:11:39 Waiting for src bucket item count to get to: 200000 curr count: 150139
2021/12/17 08:11:44 Waiting for src bucket item count to get to: 200000 curr count: 166153
2021/12/17 08:11:49 Waiting for src bucket item count to get to: 200000 curr count: 166153
2021/12/17 08:11:54 Waiting for src bucket item count to get to: 200000 curr count: 181019
2021/12/17 08:11:59 Waiting for src bucket item count to get to: 200000 curr count: 181019
2021/12/17 08:12:04 Waiting for src bucket item count to get to: 200000 curr count: 197184
2021/12/17 08:12:09 Waiting for src bucket item count to get to: 200000 curr count: 197184
2021/12/17 08:12:14 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/17 08:12:14 Post SET ops. Expected item count: 200000 got 200000
2021/12/17 08:12:14 Starting bucket ops to source bucket
2021/12/17 08:16:04 Waiting for dst bucket item count to get to: 0 curr count: 17839
2021/12/17 08:16:09 Waiting for dst bucket item count to get to: 0 curr count: 17839
2021/12/17 08:16:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 08:16:09 Post DELETE ops. Expected item count: 0 got 0
2021/12/17 08:16:09 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 08:16:14 Waiting for app: TestKVRebOutWithOps to get un-deployed
2021/12/17 08:16:19 Waiting for app: TestKVRebOutWithOps to get un-deployed
2021/12/17 08:16:24 Waiting for app: TestKVRebOutWithOps to get un-deployed
2021/12/17 08:16:29 Waiting for app: TestKVRebOutWithOps to get un-deployed
2021/12/17 08:16:29 App: TestKVRebOutWithOps got un-deployed
2021/12/17 08:16:29 No eventing-con process running
2021/12/17 08:16:29 Function: TestKVRebOutWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVRebOutWithOps deleting in the background"
}
2021/12/17 08:16:31 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 08:16:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 08:16:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 08:16:38 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 08:16:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 08:16:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebOutWithOps (463.94s)
=== RUN   TestKVSwapRebWithOps
2021/12/17 08:16:53 Waiting for app: TestKVSwapRebWithOps to get un-deployed
2021/12/17 08:16:53 App: TestKVSwapRebWithOps got un-deployed
2021/12/17 08:16:53 No eventing-con process running
2021/12/17 08:16:53 Function: TestKVSwapRebWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVSwapRebWithOps not found"
 }
}
2021/12/17 08:16:55 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 08:17:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 08:17:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 08:17:01 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 08:17:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 08:17:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 08:17:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 08:17:11 Deploying app: TestKVSwapRebWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/17 08:17:13 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVSwapRebWithOps' in metakv",
  "warnings": null
 }
}
2021/12/17 08:17:18 Waiting for app: TestKVSwapRebWithOps to get deployed
2021/12/17 08:17:23 Waiting for app: TestKVSwapRebWithOps to get deployed
2021/12/17 08:17:28 Waiting for app: TestKVSwapRebWithOps to get deployed
2021/12/17 08:17:33 Waiting for app: TestKVSwapRebWithOps to get deployed
2021/12/17 08:17:33 App: TestKVSwapRebWithOps got deployed
2021/12/17 08:17:33 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/17 08:17:33 Starting bucket ops to source bucket
2021/12/17 08:17:34 Removing node(s): [] from the cluster
2021/12/17 08:17:39 Rebalance progress: 12.5
2021/12/17 08:17:44 Rebalance progress: 12.5
2021/12/17 08:17:49 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 08:17:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 08:17:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 08:17:49 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/17 08:17:49 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/17 08:17:54 Rebalance progress: 12.5
2021/12/17 08:17:59 Rebalance progress: 12.5
2021/12/17 08:18:04 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 08:18:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 08:18:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 08:18:04 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/17 08:18:09 Rebalance progress: 12.5
2021/12/17 08:18:14 Rebalance progress: 12.5
2021/12/17 08:18:19 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 08:18:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 08:18:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 08:18:19 Waiting for src bucket item count to get to: 200000 curr count: 55559
2021/12/17 08:18:24 Waiting for src bucket item count to get to: 200000 curr count: 70663
2021/12/17 08:18:29 Waiting for src bucket item count to get to: 200000 curr count: 70663
2021/12/17 08:18:34 Waiting for src bucket item count to get to: 200000 curr count: 86220
2021/12/17 08:18:39 Waiting for src bucket item count to get to: 200000 curr count: 86220
2021/12/17 08:18:44 Waiting for src bucket item count to get to: 200000 curr count: 102483
2021/12/17 08:18:49 Waiting for src bucket item count to get to: 200000 curr count: 102483
2021/12/17 08:18:54 Waiting for src bucket item count to get to: 200000 curr count: 118762
2021/12/17 08:18:59 Waiting for src bucket item count to get to: 200000 curr count: 118762
2021/12/17 08:19:04 Waiting for src bucket item count to get to: 200000 curr count: 134691
2021/12/17 08:19:09 Waiting for src bucket item count to get to: 200000 curr count: 134691
2021/12/17 08:19:14 Waiting for src bucket item count to get to: 200000 curr count: 150533
2021/12/17 08:19:19 Waiting for src bucket item count to get to: 200000 curr count: 150533
2021/12/17 08:19:24 Waiting for src bucket item count to get to: 200000 curr count: 165880
2021/12/17 08:19:29 Waiting for src bucket item count to get to: 200000 curr count: 165880
2021/12/17 08:19:34 Waiting for src bucket item count to get to: 200000 curr count: 180913
2021/12/17 08:19:39 Waiting for src bucket item count to get to: 200000 curr count: 180913
2021/12/17 08:19:44 Waiting for src bucket item count to get to: 200000 curr count: 196764
2021/12/17 08:19:49 Waiting for src bucket item count to get to: 200000 curr count: 196764
2021/12/17 08:19:54 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/17 08:19:54 Post SET ops. Expected item count: 200000 got 200000
2021/12/17 08:19:54 Starting bucket ops to source bucket
2021/12/17 08:23:44 Waiting for dst bucket item count to get to: 0 curr count: 17566
2021/12/17 08:23:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 08:23:44 Post DELETE ops. Expected item count: 0 got 0
2021/12/17 08:23:44 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 08:23:49 Waiting for app: TestKVSwapRebWithOps to get un-deployed
2021/12/17 08:23:54 Waiting for app: TestKVSwapRebWithOps to get un-deployed
2021/12/17 08:23:59 Waiting for app: TestKVSwapRebWithOps to get un-deployed
2021/12/17 08:24:04 Waiting for app: TestKVSwapRebWithOps to get un-deployed
2021/12/17 08:24:04 App: TestKVSwapRebWithOps got un-deployed
2021/12/17 08:24:04 No eventing-con process running
2021/12/17 08:24:04 Function: TestKVSwapRebWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVSwapRebWithOps deleting in the background"
}
2021/12/17 08:24:06 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 08:24:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 08:24:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 08:24:13 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 08:24:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 08:24:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVSwapRebWithOps (454.81s)
=== RUN   TestEventingRebInWithTimers
2021/12/17 08:24:28 Waiting for app: TestEventingRebInWithTimers to get un-deployed
2021/12/17 08:24:28 App: TestEventingRebInWithTimers got un-deployed
2021/12/17 08:24:28 No eventing-con process running
2021/12/17 08:24:28 Function: TestEventingRebInWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebInWithTimers not found"
 }
}
2021/12/17 08:24:29 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 08:24:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 08:24:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 08:24:36 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 08:24:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 08:24:41 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 08:24:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 08:24:46 Deploying app: TestEventingRebInWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/17 08:24:47 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebInWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/17 08:24:52 Waiting for app: TestEventingRebInWithTimers to get deployed
2021/12/17 08:24:57 Waiting for app: TestEventingRebInWithTimers to get deployed
2021/12/17 08:25:02 Waiting for app: TestEventingRebInWithTimers to get deployed
2021/12/17 08:25:07 Waiting for app: TestEventingRebInWithTimers to get deployed
2021/12/17 08:25:07 App: TestEventingRebInWithTimers got deployed
2021/12/17 08:25:07 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/17 08:25:07 Starting bucket ops to source bucket
2021/12/17 08:25:08 Removing node(s): [] from the cluster
2021/12/17 08:25:13 Rebalance progress: 12.5
2021/12/17 08:25:18 Rebalance progress: 12.5
2021/12/17 08:25:23 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 08:25:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 08:25:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 08:25:23 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/17 08:25:28 Rebalance progress: 12.5
2021/12/17 08:25:33 Rebalance progress: 12.5
2021/12/17 08:25:38 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 08:25:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 08:25:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 08:25:38 Waiting for src bucket item count to get to: 50000 curr count: 28230
2021/12/17 08:25:43 Waiting for src bucket item count to get to: 50000 curr count: 37819
2021/12/17 08:25:48 Waiting for src bucket item count to get to: 50000 curr count: 37819
2021/12/17 08:25:58 Waiting for dst bucket item count to get to: 50000 curr count: 29102
2021/12/17 08:26:03 Waiting for dst bucket item count to get to: 50000 curr count: 29102
2021/12/17 08:26:08 Waiting for dst bucket item count to get to: 50000 curr count: 41403
2021/12/17 08:26:13 Waiting for dst bucket item count to get to: 50000 curr count: 41403
2021/12/17 08:26:18 Waiting for dst bucket item count to get to: 50000 curr count: 49758
2021/12/17 08:26:23 Waiting for dst bucket item count to get to: 50000 curr count: 49758
2021/12/17 08:26:23 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/17 08:26:23 Post SET ops. Expected item count: 50000 got 50000
2021/12/17 08:26:23 Starting bucket ops to source bucket
2021/12/17 08:27:33 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/17 08:27:38 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/17 08:27:43 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/17 08:27:48 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/17 08:27:53 Waiting for dst bucket item count to get to: 0 curr count: 46373
2021/12/17 08:27:58 Waiting for dst bucket item count to get to: 0 curr count: 46373
2021/12/17 08:28:03 Waiting for dst bucket item count to get to: 0 curr count: 29507
2021/12/17 08:28:08 Waiting for dst bucket item count to get to: 0 curr count: 29507
2021/12/17 08:28:13 Waiting for dst bucket item count to get to: 0 curr count: 14400
2021/12/17 08:28:18 Waiting for dst bucket item count to get to: 0 curr count: 238
2021/12/17 08:28:23 Waiting for dst bucket item count to get to: 0 curr count: 238
2021/12/17 08:28:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 08:28:23 Post DELETE ops. Expected item count: 0 got 0
2021/12/17 08:28:23 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 08:28:28 Waiting for app: TestEventingRebInWithTimers to get un-deployed
2021/12/17 08:28:33 Waiting for app: TestEventingRebInWithTimers to get un-deployed
2021/12/17 08:28:38 Waiting for app: TestEventingRebInWithTimers to get un-deployed
2021/12/17 08:28:43 Waiting for app: TestEventingRebInWithTimers to get un-deployed
2021/12/17 08:28:43 App: TestEventingRebInWithTimers got un-deployed
2021/12/17 08:28:43 No eventing-con process running
2021/12/17 08:28:43 Function: TestEventingRebInWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebInWithTimers deleting in the background"
}
2021/12/17 08:28:45 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 08:28:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 08:28:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 08:28:52 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 08:28:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 08:28:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebInWithTimers (279.11s)
=== RUN   TestEventingRebOutWithTimers
2021/12/17 08:29:02 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/17 08:29:02 Removing node(s): [] from the cluster
2021/12/17 08:29:07 Rebalance progress: 100
2021/12/17 08:29:12 Waiting for app: TestEventingRebOutWithTimers to get un-deployed
2021/12/17 08:29:12 App: TestEventingRebOutWithTimers got un-deployed
2021/12/17 08:29:12 No eventing-con process running
2021/12/17 08:29:12 Function: TestEventingRebOutWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebOutWithTimers not found"
 }
}
2021/12/17 08:29:14 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 08:29:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 08:29:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 08:29:21 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 08:29:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 08:29:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 08:29:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 08:29:31 Deploying app: TestEventingRebOutWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/17 08:29:32 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebOutWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/17 08:29:37 Waiting for app: TestEventingRebOutWithTimers to get deployed
2021/12/17 08:29:42 Waiting for app: TestEventingRebOutWithTimers to get deployed
2021/12/17 08:29:47 Waiting for app: TestEventingRebOutWithTimers to get deployed
2021/12/17 08:29:52 Waiting for app: TestEventingRebOutWithTimers to get deployed
2021/12/17 08:29:52 App: TestEventingRebOutWithTimers got deployed
2021/12/17 08:29:52 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/17 08:29:52 Starting bucket ops to source bucket
2021/12/17 08:29:57 Rebalance progress: 12.5
2021/12/17 08:30:02 Rebalance progress: 12.5
2021/12/17 08:30:07 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 08:30:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 08:30:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 08:30:07 Waiting for src bucket item count to get to: 50000 curr count: 9882
2021/12/17 08:30:12 Waiting for src bucket item count to get to: 50000 curr count: 23064
2021/12/17 08:30:17 Waiting for src bucket item count to get to: 50000 curr count: 23064
2021/12/17 08:30:22 Waiting for src bucket item count to get to: 50000 curr count: 34484
2021/12/17 08:30:27 Waiting for src bucket item count to get to: 50000 curr count: 34484
2021/12/17 08:30:32 Waiting for src bucket item count to get to: 50000 curr count: 34484
2021/12/17 08:30:37 Waiting for src bucket item count to get to: 50000 curr count: 44499
2021/12/17 08:30:42 Waiting for src bucket item count to get to: 50000 curr count: 44499
2021/12/17 08:30:52 Waiting for dst bucket item count to get to: 50000 curr count: 38175
2021/12/17 08:30:57 Waiting for dst bucket item count to get to: 50000 curr count: 38175
2021/12/17 08:30:57 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/17 08:30:57 Post SET ops. Expected item count: 50000 got 50000
2021/12/17 08:30:57 Starting bucket ops to source bucket
2021/12/17 08:32:07 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/17 08:32:12 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/17 08:32:17 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/17 08:32:22 Waiting for dst bucket item count to get to: 0 curr count: 49194
2021/12/17 08:32:27 Waiting for dst bucket item count to get to: 0 curr count: 49194
2021/12/17 08:32:32 Waiting for dst bucket item count to get to: 0 curr count: 35185
2021/12/17 08:32:37 Waiting for dst bucket item count to get to: 0 curr count: 35185
2021/12/17 08:32:42 Waiting for dst bucket item count to get to: 0 curr count: 16313
2021/12/17 08:32:47 Waiting for dst bucket item count to get to: 0 curr count: 16313
2021/12/17 08:32:52 Waiting for dst bucket item count to get to: 0 curr count: 3467
2021/12/17 08:32:57 Waiting for dst bucket item count to get to: 0 curr count: 3467
2021/12/17 08:32:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 08:32:57 Post DELETE ops. Expected item count: 0 got 0
2021/12/17 08:32:57 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 08:33:02 Waiting for app: TestEventingRebOutWithTimers to get un-deployed
2021/12/17 08:33:07 Waiting for app: TestEventingRebOutWithTimers to get un-deployed
2021/12/17 08:33:12 Waiting for app: TestEventingRebOutWithTimers to get un-deployed
2021/12/17 08:33:17 Waiting for app: TestEventingRebOutWithTimers to get un-deployed
2021/12/17 08:33:17 App: TestEventingRebOutWithTimers got un-deployed
2021/12/17 08:33:17 No eventing-con process running
2021/12/17 08:33:17 Function: TestEventingRebOutWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebOutWithTimers deleting in the background"
}
2021/12/17 08:33:19 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 08:33:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 08:33:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 08:33:26 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 08:33:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 08:33:31 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebOutWithTimers (273.98s)
=== RUN   TestEventingSwapRebWithTimers
2021/12/17 08:33:41 Waiting for app: TestEventingSwapRebWithTimers to get un-deployed
2021/12/17 08:33:41 App: TestEventingSwapRebWithTimers got un-deployed
2021/12/17 08:33:41 No eventing-con process running
2021/12/17 08:33:41 Function: TestEventingSwapRebWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingSwapRebWithTimers not found"
 }
}
2021/12/17 08:33:43 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 08:33:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 08:33:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 08:33:49 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 08:33:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 08:33:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 08:33:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 08:33:59 Deploying app: TestEventingSwapRebWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/17 08:34:00 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingSwapRebWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/17 08:34:05 Waiting for app: TestEventingSwapRebWithTimers to get deployed
2021/12/17 08:34:10 Waiting for app: TestEventingSwapRebWithTimers to get deployed
2021/12/17 08:34:15 Waiting for app: TestEventingSwapRebWithTimers to get deployed
2021/12/17 08:34:20 Waiting for app: TestEventingSwapRebWithTimers to get deployed
2021/12/17 08:34:20 App: TestEventingSwapRebWithTimers got deployed
2021/12/17 08:34:20 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/17 08:34:20 Starting bucket ops to source bucket
2021/12/17 08:34:21 Removing node(s): [] from the cluster
2021/12/17 08:34:26 Rebalance progress: 12.5
2021/12/17 08:34:31 Rebalance progress: 12.5
2021/12/17 08:34:36 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 08:34:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 08:34:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 08:34:36 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/17 08:34:37 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/17 08:34:42 Rebalance progress: 12.5
2021/12/17 08:34:47 Rebalance progress: 12.5
2021/12/17 08:34:52 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 08:34:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 08:34:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 08:34:52 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/17 08:34:57 Rebalance progress: 12.5
2021/12/17 08:35:02 Rebalance progress: 12.5
2021/12/17 08:35:07 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 08:35:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 08:35:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 08:35:07 Waiting for src bucket item count to get to: 50000 curr count: 45807
2021/12/17 08:35:17 Waiting for dst bucket item count to get to: 50000 curr count: 36880
2021/12/17 08:35:22 Waiting for dst bucket item count to get to: 50000 curr count: 36880
2021/12/17 08:35:27 Waiting for dst bucket item count to get to: 50000 curr count: 49259
2021/12/17 08:35:32 Waiting for dst bucket item count to get to: 50000 curr count: 49259
2021/12/17 08:35:32 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/17 08:35:32 Post SET ops. Expected item count: 50000 got 50000
2021/12/17 08:35:32 Starting bucket ops to source bucket
2021/12/17 08:36:43 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/17 08:36:48 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/17 08:36:53 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/17 08:36:58 Waiting for dst bucket item count to get to: 0 curr count: 40620
2021/12/17 08:37:03 Waiting for dst bucket item count to get to: 0 curr count: 40620
2021/12/17 08:37:08 Waiting for dst bucket item count to get to: 0 curr count: 30658
2021/12/17 08:37:13 Waiting for dst bucket item count to get to: 0 curr count: 30658
2021/12/17 08:37:18 Waiting for dst bucket item count to get to: 0 curr count: 30658
2021/12/17 08:37:23 Waiting for dst bucket item count to get to: 0 curr count: 13742
2021/12/17 08:37:28 Waiting for dst bucket item count to get to: 0 curr count: 13742
2021/12/17 08:37:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 08:37:28 Post DELETE ops. Expected item count: 0 got 0
2021/12/17 08:37:28 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 08:37:33 Waiting for app: TestEventingSwapRebWithTimers to get un-deployed
2021/12/17 08:37:38 Waiting for app: TestEventingSwapRebWithTimers to get un-deployed
2021/12/17 08:37:43 Waiting for app: TestEventingSwapRebWithTimers to get un-deployed
2021/12/17 08:37:48 Waiting for app: TestEventingSwapRebWithTimers to get un-deployed
2021/12/17 08:37:48 App: TestEventingSwapRebWithTimers got un-deployed
2021/12/17 08:37:48 No eventing-con process running
2021/12/17 08:37:48 Function: TestEventingSwapRebWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingSwapRebWithTimers deleting in the background"
}
2021/12/17 08:37:50 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 08:37:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 08:37:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 08:37:56 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 08:38:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 08:38:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingSwapRebWithTimers (270.62s)
=== RUN   TestKVRebInWithTimers
2021/12/17 08:38:11 Waiting for app: TestKVRebInWithTimers to get un-deployed
2021/12/17 08:38:11 App: TestKVRebInWithTimers got un-deployed
2021/12/17 08:38:11 No eventing-con process running
2021/12/17 08:38:11 Function: TestKVRebInWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVRebInWithTimers not found"
 }
}
2021/12/17 08:38:13 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 08:38:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 08:38:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 08:38:20 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 08:38:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 08:38:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 08:38:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 08:38:30 Deploying app: TestKVRebInWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/17 08:38:31 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVRebInWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/17 08:38:36 Waiting for app: TestKVRebInWithTimers to get deployed
2021/12/17 08:38:41 Waiting for app: TestKVRebInWithTimers to get deployed
2021/12/17 08:38:46 Waiting for app: TestKVRebInWithTimers to get deployed
2021/12/17 08:38:51 Waiting for app: TestKVRebInWithTimers to get deployed
2021/12/17 08:38:51 App: TestKVRebInWithTimers got deployed
2021/12/17 08:38:51 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/17 08:38:51 Starting bucket ops to source bucket
2021/12/17 08:38:51 Removing node(s): [] from the cluster
2021/12/17 08:38:56 Rebalance progress: 12.5
2021/12/17 08:39:01 Rebalance progress: 12.5
2021/12/17 08:39:06 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 08:39:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 08:39:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 08:39:06 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/17 08:39:11 Rebalance progress: 12.5
2021/12/17 08:39:16 Rebalance progress: 12.5
2021/12/17 08:39:21 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 08:39:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 08:39:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 08:39:22 Waiting for src bucket item count to get to: 50000 curr count: 34265
2021/12/17 08:39:27 Waiting for src bucket item count to get to: 50000 curr count: 34265
2021/12/17 08:39:32 Waiting for src bucket item count to get to: 50000 curr count: 44369
2021/12/17 08:39:37 Waiting for src bucket item count to get to: 50000 curr count: 44369
2021/12/17 08:39:47 Waiting for dst bucket item count to get to: 50000 curr count: 37025
2021/12/17 08:39:52 Waiting for dst bucket item count to get to: 50000 curr count: 37025
2021/12/17 08:39:52 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/17 08:39:52 Post SET ops. Expected item count: 50000 got 50000
2021/12/17 08:39:52 Starting bucket ops to source bucket
2021/12/17 08:41:02 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/17 08:41:07 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/17 08:41:12 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/17 08:41:17 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/17 08:41:22 Waiting for dst bucket item count to get to: 0 curr count: 43576
2021/12/17 08:41:27 Waiting for dst bucket item count to get to: 0 curr count: 43576
2021/12/17 08:41:32 Waiting for dst bucket item count to get to: 0 curr count: 25316
2021/12/17 08:41:37 Waiting for dst bucket item count to get to: 0 curr count: 25316
2021/12/17 08:41:42 Waiting for dst bucket item count to get to: 0 curr count: 9239
2021/12/17 08:41:47 Waiting for dst bucket item count to get to: 0 curr count: 9239
2021/12/17 08:41:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 08:41:47 Post DELETE ops. Expected item count: 0 got 0
2021/12/17 08:41:47 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 08:41:52 Waiting for app: TestKVRebInWithTimers to get un-deployed
2021/12/17 08:41:57 Waiting for app: TestKVRebInWithTimers to get un-deployed
2021/12/17 08:42:02 Waiting for app: TestKVRebInWithTimers to get un-deployed
2021/12/17 08:42:07 Waiting for app: TestKVRebInWithTimers to get un-deployed
2021/12/17 08:42:07 App: TestKVRebInWithTimers got un-deployed
2021/12/17 08:42:07 No eventing-con process running
2021/12/17 08:42:07 Function: TestKVRebInWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVRebInWithTimers deleting in the background"
}
2021/12/17 08:42:09 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 08:42:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 08:42:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 08:42:16 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 08:42:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 08:42:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebInWithTimers (259.30s)
=== RUN   TestKVRebOutWithTimers
2021/12/17 08:42:26 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/17 08:42:26 Removing node(s): [] from the cluster
2021/12/17 08:42:31 Rebalance progress: 100
2021/12/17 08:42:36 Waiting for app: TestKVRebOutWithTimers to get un-deployed
2021/12/17 08:42:36 App: TestKVRebOutWithTimers got un-deployed
2021/12/17 08:42:36 No eventing-con process running
2021/12/17 08:42:36 Function: TestKVRebOutWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVRebOutWithTimers not found"
 }
}
2021/12/17 08:42:38 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 08:42:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 08:42:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 08:42:45 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 08:42:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 08:42:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 08:42:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 08:42:55 Deploying app: TestKVRebOutWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/17 08:42:56 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVRebOutWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/17 08:43:01 Waiting for app: TestKVRebOutWithTimers to get deployed
2021/12/17 08:43:06 Waiting for app: TestKVRebOutWithTimers to get deployed
2021/12/17 08:43:11 Waiting for app: TestKVRebOutWithTimers to get deployed
2021/12/17 08:43:16 Waiting for app: TestKVRebOutWithTimers to get deployed
2021/12/17 08:43:16 App: TestKVRebOutWithTimers got deployed
2021/12/17 08:43:16 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/17 08:43:16 Starting bucket ops to source bucket
2021/12/17 08:43:21 Rebalance progress: 12.5
2021/12/17 08:43:26 Rebalance progress: 12.5
2021/12/17 08:43:31 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 08:43:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 08:43:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 08:43:31 Waiting for src bucket item count to get to: 50000 curr count: 18680
2021/12/17 08:43:36 Waiting for src bucket item count to get to: 50000 curr count: 18680
2021/12/17 08:43:41 Waiting for src bucket item count to get to: 50000 curr count: 28981
2021/12/17 08:43:46 Waiting for src bucket item count to get to: 50000 curr count: 28981
2021/12/17 08:43:51 Waiting for src bucket item count to get to: 50000 curr count: 40693
2021/12/17 08:43:56 Waiting for src bucket item count to get to: 50000 curr count: 40693
2021/12/17 08:44:01 Waiting for src bucket item count to get to: 50000 curr count: 40693
2021/12/17 08:44:11 Waiting for dst bucket item count to get to: 50000 curr count: 39435
2021/12/17 08:44:16 Waiting for dst bucket item count to get to: 50000 curr count: 39435
2021/12/17 08:44:21 Waiting for dst bucket item count to get to: 50000 curr count: 47808
2021/12/17 08:44:26 Waiting for dst bucket item count to get to: 50000 curr count: 47808
2021/12/17 08:44:26 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/17 08:44:26 Post SET ops. Expected item count: 50000 got 50000
2021/12/17 08:44:26 Starting bucket ops to source bucket
2021/12/17 08:45:36 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/17 08:45:41 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/17 08:45:46 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/17 08:45:51 Waiting for dst bucket item count to get to: 0 curr count: 48591
2021/12/17 08:45:56 Waiting for dst bucket item count to get to: 0 curr count: 48591
2021/12/17 08:46:01 Waiting for dst bucket item count to get to: 0 curr count: 37980
2021/12/17 08:46:06 Waiting for dst bucket item count to get to: 0 curr count: 37980
2021/12/17 08:46:11 Waiting for dst bucket item count to get to: 0 curr count: 11423
2021/12/17 08:46:16 Waiting for dst bucket item count to get to: 0 curr count: 11423
2021/12/17 08:46:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 08:46:16 Post DELETE ops. Expected item count: 0 got 0
2021/12/17 08:46:16 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 08:46:21 Waiting for app: TestKVRebOutWithTimers to get un-deployed
2021/12/17 08:46:26 Waiting for app: TestKVRebOutWithTimers to get un-deployed
2021/12/17 08:46:31 Waiting for app: TestKVRebOutWithTimers to get un-deployed
2021/12/17 08:46:36 Waiting for app: TestKVRebOutWithTimers to get un-deployed
2021/12/17 08:46:36 App: TestKVRebOutWithTimers got un-deployed
2021/12/17 08:46:36 No eventing-con process running
2021/12/17 08:46:36 Function: TestKVRebOutWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVRebOutWithTimers deleting in the background"
}
2021/12/17 08:46:38 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 08:46:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 08:46:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 08:46:45 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 08:46:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 08:46:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebOutWithTimers (269.33s)
=== RUN   TestKVSwapRebWithTimers
2021/12/17 08:47:00 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/17 08:47:00 App: TestKVSwapRebWithTimers got un-deployed
2021/12/17 08:47:00 No eventing-con process running
2021/12/17 08:47:00 Function: TestKVSwapRebWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVSwapRebWithTimers not found"
 }
}
2021/12/17 08:47:02 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 08:47:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 08:47:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 08:47:09 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 08:47:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 08:47:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 08:47:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 08:47:19 Deploying app: TestKVSwapRebWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/17 08:47:20 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVSwapRebWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/17 08:47:25 Waiting for app: TestKVSwapRebWithTimers to get deployed
2021/12/17 08:47:30 Waiting for app: TestKVSwapRebWithTimers to get deployed
2021/12/17 08:47:35 Waiting for app: TestKVSwapRebWithTimers to get deployed
2021/12/17 08:47:40 Waiting for app: TestKVSwapRebWithTimers to get deployed
2021/12/17 08:47:40 App: TestKVSwapRebWithTimers got deployed
2021/12/17 08:47:40 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/17 08:47:40 Starting bucket ops to source bucket
2021/12/17 08:47:40 Removing node(s): [] from the cluster
2021/12/17 08:47:45 Rebalance progress: 12.5
2021/12/17 08:47:50 Rebalance progress: 12.5
2021/12/17 08:47:55 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 08:47:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 08:47:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 08:47:55 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/17 08:47:56 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/17 08:48:01 Rebalance progress: 12.5
2021/12/17 08:48:06 Rebalance progress: 12.5
2021/12/17 08:48:11 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 08:48:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 08:48:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 08:48:11 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/17 08:48:16 Rebalance progress: 12.5
2021/12/17 08:48:21 Rebalance progress: 12.5
2021/12/17 08:48:26 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 08:48:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 08:48:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 08:48:26 Waiting for src bucket item count to get to: 50000 curr count: 44985
2021/12/17 08:48:36 Waiting for dst bucket item count to get to: 50000 curr count: 44141
2021/12/17 08:48:41 Waiting for dst bucket item count to get to: 50000 curr count: 44141
2021/12/17 08:48:41 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/17 08:48:41 Post SET ops. Expected item count: 50000 got 50000
2021/12/17 08:48:41 Starting bucket ops to source bucket
2021/12/17 08:49:51 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/17 08:49:56 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/17 08:50:01 Waiting for dst bucket item count to get to: 0 curr count: 48533
2021/12/17 08:50:06 Waiting for dst bucket item count to get to: 0 curr count: 48533
2021/12/17 08:50:11 Waiting for dst bucket item count to get to: 0 curr count: 37176
2021/12/17 08:50:16 Waiting for dst bucket item count to get to: 0 curr count: 37176
2021/12/17 08:50:21 Waiting for dst bucket item count to get to: 0 curr count: 16462
2021/12/17 08:50:26 Waiting for dst bucket item count to get to: 0 curr count: 16462
2021/12/17 08:50:31 Waiting for dst bucket item count to get to: 0 curr count: 4452
2021/12/17 08:50:36 Waiting for dst bucket item count to get to: 0 curr count: 4452
2021/12/17 08:50:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 08:50:36 Post DELETE ops. Expected item count: 0 got 0
2021/12/17 08:50:36 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 08:50:41 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/17 08:50:47 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/17 08:50:52 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/17 08:50:57 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/17 08:51:02 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/17 08:51:02 App: TestKVSwapRebWithTimers got un-deployed
2021/12/17 08:51:02 No eventing-con process running
2021/12/17 08:51:02 Function: TestKVSwapRebWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVSwapRebWithTimers deleting in the background"
}
2021/12/17 08:51:03 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 08:51:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 08:51:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 08:51:10 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 08:51:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 08:51:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVSwapRebWithTimers (265.26s)
=== RUN   TestDuplicateTimerCreationEventingRebIn
2021/12/17 08:51:25 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/17 08:51:25 App: TestDuplicateTimerCreationEventingRebIn got un-deployed
2021/12/17 08:51:25 No eventing-con process running
2021/12/17 08:51:25 Function: TestDuplicateTimerCreationEventingRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationEventingRebIn not found"
 }
}
2021/12/17 08:51:27 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 08:51:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 08:51:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 08:51:34 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 08:51:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 08:51:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 08:51:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 08:51:44 Deploying app: TestDuplicateTimerCreationEventingRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/17 08:51:45 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationEventingRebIn' in metakv",
  "warnings": null
 }
}
2021/12/17 08:51:50 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get deployed
2021/12/17 08:51:55 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get deployed
2021/12/17 08:52:00 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get deployed
2021/12/17 08:52:05 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get deployed
2021/12/17 08:52:05 App: TestDuplicateTimerCreationEventingRebIn got deployed
2021/12/17 08:52:05 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/17 08:52:05 Starting bucket ops to source bucket
2021/12/17 08:52:05 Removing node(s): [] from the cluster
2021/12/17 08:52:11 Rebalance progress: 12.5
2021/12/17 08:52:16 Rebalance progress: 12.5
2021/12/17 08:52:21 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 08:52:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 08:52:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 08:52:21 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/17 08:52:26 Rebalance progress: 12.5
2021/12/17 08:52:31 Rebalance progress: 12.5
2021/12/17 08:52:36 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 08:52:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 08:52:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 08:52:36 Waiting for src bucket item count to get to: 50000 curr count: 31105
2021/12/17 08:52:41 Waiting for src bucket item count to get to: 50000 curr count: 42399
2021/12/17 08:52:46 Waiting for src bucket item count to get to: 50000 curr count: 42399
2021/12/17 08:52:56 Waiting for dst bucket item count to get to: 50000 curr count: 28391
2021/12/17 08:53:01 Waiting for dst bucket item count to get to: 50000 curr count: 28391
2021/12/17 08:53:06 Waiting for dst bucket item count to get to: 50000 curr count: 36953
2021/12/17 08:53:11 Waiting for dst bucket item count to get to: 50000 curr count: 36953
2021/12/17 08:53:16 Waiting for dst bucket item count to get to: 50000 curr count: 49462
2021/12/17 08:53:21 Waiting for dst bucket item count to get to: 50000 curr count: 49462
2021/12/17 08:53:21 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/17 08:53:21 Post SET ops. Expected item count: 50000 got 50000
2021/12/17 08:53:21 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 08:53:26 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/17 08:53:31 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/17 08:53:36 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/17 08:53:41 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/17 08:53:46 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/17 08:53:51 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/17 08:53:51 App: TestDuplicateTimerCreationEventingRebIn got un-deployed
2021/12/17 08:53:51 No eventing-con process running
2021/12/17 08:53:51 Function: TestDuplicateTimerCreationEventingRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationEventingRebIn deleting in the background"
}
2021/12/17 08:53:53 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 08:53:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 08:54:03 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/17 08:54:08 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/17 08:54:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 08:54:10 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 08:54:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 08:54:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationEventingRebIn (179.46s)
=== RUN   TestDuplicateTimerCreationEventingRebOut
2021/12/17 08:54:20 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/17 08:54:20 Removing node(s): [] from the cluster
2021/12/17 08:54:25 Rebalance progress: 100
2021/12/17 08:54:30 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/17 08:54:30 App: TestDuplicateTimerCreationEventingRebOut got un-deployed
2021/12/17 08:54:30 No eventing-con process running
2021/12/17 08:54:30 Function: TestDuplicateTimerCreationEventingRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationEventingRebOut not found"
 }
}
2021/12/17 08:54:32 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 08:54:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 08:54:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 08:54:39 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 08:54:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 08:54:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 08:54:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 08:54:49 Deploying app: TestDuplicateTimerCreationEventingRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/17 08:54:50 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationEventingRebOut' in metakv",
  "warnings": null
 }
}
2021/12/17 08:54:55 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get deployed
2021/12/17 08:55:00 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get deployed
2021/12/17 08:55:05 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get deployed
2021/12/17 08:55:10 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get deployed
2021/12/17 08:55:10 App: TestDuplicateTimerCreationEventingRebOut got deployed
2021/12/17 08:55:10 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/17 08:55:10 Starting bucket ops to source bucket
2021/12/17 08:55:15 Rebalance progress: 12.5
2021/12/17 08:55:20 Rebalance progress: 12.5
2021/12/17 08:55:25 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 08:55:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 08:55:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 08:55:25 Waiting for src bucket item count to get to: 50000 curr count: 12576
2021/12/17 08:55:30 Waiting for src bucket item count to get to: 50000 curr count: 12576
2021/12/17 08:55:35 Waiting for src bucket item count to get to: 50000 curr count: 26656
2021/12/17 08:55:40 Waiting for src bucket item count to get to: 50000 curr count: 37725
2021/12/17 08:55:45 Waiting for src bucket item count to get to: 50000 curr count: 37725
2021/12/17 08:55:50 Waiting for src bucket item count to get to: 50000 curr count: 37725
2021/12/17 08:55:55 Waiting for src bucket item count to get to: 50000 curr count: 48034
2021/12/17 08:56:00 Waiting for src bucket item count to get to: 50000 curr count: 48034
2021/12/17 08:56:10 Waiting for dst bucket item count to get to: 50000 curr count: 37834
2021/12/17 08:56:15 Waiting for dst bucket item count to get to: 50000 curr count: 37834
2021/12/17 08:56:20 Waiting for dst bucket item count to get to: 50000 curr count: 46857
2021/12/17 08:56:25 Waiting for dst bucket item count to get to: 50000 curr count: 46857
2021/12/17 08:56:25 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/17 08:56:25 Post SET ops. Expected item count: 50000 got 50000
2021/12/17 08:56:25 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 08:56:30 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/17 08:56:35 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/17 08:56:40 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/17 08:56:45 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/17 08:56:50 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/17 08:56:55 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/17 08:56:55 App: TestDuplicateTimerCreationEventingRebOut got un-deployed
2021/12/17 08:56:55 No eventing-con process running
2021/12/17 08:56:55 Function: TestDuplicateTimerCreationEventingRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationEventingRebOut deleting in the background"
}
2021/12/17 08:56:57 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 08:57:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 08:57:07 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/17 08:57:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 08:57:09 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 08:57:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 08:57:19 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/17 08:57:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationEventingRebOut (184.13s)
=== RUN   TestDuplicateTimerCreationEventingSwapReb
2021/12/17 08:57:29 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/17 08:57:29 App: TestDuplicateTimerCreationEventingSwapReb got un-deployed
2021/12/17 08:57:29 No eventing-con process running
2021/12/17 08:57:29 Function: TestDuplicateTimerCreationEventingSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationEventingSwapReb not found"
 }
}
2021/12/17 08:57:31 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 08:57:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 08:57:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 08:57:37 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 08:57:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 08:57:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 08:57:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 08:57:47 Deploying app: TestDuplicateTimerCreationEventingSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/17 08:57:49 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationEventingSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/17 08:57:54 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get deployed
2021/12/17 08:57:59 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get deployed
2021/12/17 08:58:04 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get deployed
2021/12/17 08:58:09 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get deployed
2021/12/17 08:58:09 App: TestDuplicateTimerCreationEventingSwapReb got deployed
2021/12/17 08:58:09 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/17 08:58:09 Starting bucket ops to source bucket
2021/12/17 08:58:09 Removing node(s): [] from the cluster
2021/12/17 08:58:14 Rebalance progress: 12.5
2021/12/17 08:58:19 Rebalance progress: 12.5
2021/12/17 08:58:24 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 08:58:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 08:58:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 08:58:24 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/17 08:58:25 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/17 08:58:30 Rebalance progress: 12.5
2021/12/17 08:58:35 Rebalance progress: 12.5
2021/12/17 08:58:40 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 08:58:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 08:58:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 08:58:40 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/17 08:58:45 Rebalance progress: 12.5
2021/12/17 08:58:50 Rebalance progress: 12.5
2021/12/17 08:58:55 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 08:58:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 08:58:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 08:58:55 Waiting for src bucket item count to get to: 50000 curr count: 47689
2021/12/17 08:59:00 Waiting for src bucket item count to get to: 50000 curr count: 47689
2021/12/17 08:59:10 Waiting for dst bucket item count to get to: 50000 curr count: 35015
2021/12/17 08:59:15 Waiting for dst bucket item count to get to: 50000 curr count: 35015
2021/12/17 08:59:20 Waiting for dst bucket item count to get to: 50000 curr count: 41233
2021/12/17 08:59:25 Waiting for dst bucket item count to get to: 50000 curr count: 41233
2021/12/17 08:59:25 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/17 08:59:25 Post SET ops. Expected item count: 50000 got 50000
2021/12/17 08:59:25 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 08:59:30 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/17 08:59:35 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/17 08:59:40 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/17 08:59:45 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/17 08:59:50 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/17 08:59:55 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/17 08:59:55 App: TestDuplicateTimerCreationEventingSwapReb got un-deployed
2021/12/17 08:59:55 No eventing-con process running
2021/12/17 08:59:55 Function: TestDuplicateTimerCreationEventingSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationEventingSwapReb deleting in the background"
}
2021/12/17 08:59:57 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:00:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 09:00:07 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/17 09:00:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 09:00:09 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:00:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 09:00:19 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/17 09:00:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationEventingSwapReb (179.80s)
=== RUN   TestDuplicateTimerCreationKVRebIn
2021/12/17 09:00:29 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/17 09:00:29 App: TestDuplicateTimerCreationKVRebIn got un-deployed
2021/12/17 09:00:29 No eventing-con process running
2021/12/17 09:00:29 Function: TestDuplicateTimerCreationKVRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationKVRebIn not found"
 }
}
2021/12/17 09:00:31 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:00:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 09:00:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 09:00:37 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:00:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 09:00:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:00:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 09:00:47 Deploying app: TestDuplicateTimerCreationKVRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/17 09:00:48 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationKVRebIn' in metakv",
  "warnings": null
 }
}
2021/12/17 09:00:53 Waiting for app: TestDuplicateTimerCreationKVRebIn to get deployed
2021/12/17 09:00:58 Waiting for app: TestDuplicateTimerCreationKVRebIn to get deployed
2021/12/17 09:01:03 Waiting for app: TestDuplicateTimerCreationKVRebIn to get deployed
2021/12/17 09:01:09 Waiting for app: TestDuplicateTimerCreationKVRebIn to get deployed
2021/12/17 09:01:09 App: TestDuplicateTimerCreationKVRebIn got deployed
2021/12/17 09:01:09 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/17 09:01:09 Starting bucket ops to source bucket
2021/12/17 09:01:09 Removing node(s): [] from the cluster
2021/12/17 09:01:14 Rebalance progress: 12.5
2021/12/17 09:01:19 Rebalance progress: 12.5
2021/12/17 09:01:24 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:01:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:01:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 09:01:24 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/17 09:01:29 Rebalance progress: 12.5
2021/12/17 09:01:34 Rebalance progress: 12.5
2021/12/17 09:01:39 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:01:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:01:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 09:01:39 Waiting for src bucket item count to get to: 50000 curr count: 26220
2021/12/17 09:01:44 Waiting for src bucket item count to get to: 50000 curr count: 37011
2021/12/17 09:01:49 Waiting for src bucket item count to get to: 50000 curr count: 37011
2021/12/17 09:01:54 Waiting for src bucket item count to get to: 50000 curr count: 47793
2021/12/17 09:01:59 Waiting for src bucket item count to get to: 50000 curr count: 47793
2021/12/17 09:02:09 Waiting for dst bucket item count to get to: 50000 curr count: 34925
2021/12/17 09:02:14 Waiting for dst bucket item count to get to: 50000 curr count: 34925
2021/12/17 09:02:19 Waiting for dst bucket item count to get to: 50000 curr count: 41483
2021/12/17 09:02:24 Waiting for dst bucket item count to get to: 50000 curr count: 41483
2021/12/17 09:02:29 Waiting for dst bucket item count to get to: 50000 curr count: 41483
2021/12/17 09:02:29 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/17 09:02:29 Post SET ops. Expected item count: 50000 got 50000
2021/12/17 09:02:29 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 09:02:34 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/17 09:02:39 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/17 09:02:44 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/17 09:02:49 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/17 09:02:54 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/17 09:02:59 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/17 09:02:59 App: TestDuplicateTimerCreationKVRebIn got un-deployed
2021/12/17 09:02:59 No eventing-con process running
2021/12/17 09:02:59 Function: TestDuplicateTimerCreationKVRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationKVRebIn deleting in the background"
}
2021/12/17 09:03:01 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:03:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 09:03:11 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/17 09:03:16 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/17 09:03:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 09:03:18 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:03:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 09:03:28 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/17 09:03:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationKVRebIn (189.38s)
=== RUN   TestDuplicateTimerCreationKVRebOut
2021/12/17 09:03:33 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/17 09:03:33 Removing node(s): [] from the cluster
2021/12/17 09:03:38 Rebalance progress: 100
2021/12/17 09:03:43 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/17 09:03:43 App: TestDuplicateTimerCreationKVRebOut got un-deployed
2021/12/17 09:03:43 No eventing-con process running
2021/12/17 09:03:43 Function: TestDuplicateTimerCreationKVRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationKVRebOut not found"
 }
}
2021/12/17 09:03:45 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:03:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 09:03:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 09:03:52 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:03:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 09:03:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:04:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 09:04:02 Deploying app: TestDuplicateTimerCreationKVRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/17 09:04:03 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationKVRebOut' in metakv",
  "warnings": null
 }
}
2021/12/17 09:04:08 Waiting for app: TestDuplicateTimerCreationKVRebOut to get deployed
2021/12/17 09:04:13 Waiting for app: TestDuplicateTimerCreationKVRebOut to get deployed
2021/12/17 09:04:18 Waiting for app: TestDuplicateTimerCreationKVRebOut to get deployed
2021/12/17 09:04:23 Waiting for app: TestDuplicateTimerCreationKVRebOut to get deployed
2021/12/17 09:04:23 App: TestDuplicateTimerCreationKVRebOut got deployed
2021/12/17 09:04:23 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/17 09:04:23 Starting bucket ops to source bucket
2021/12/17 09:04:28 Rebalance progress: 12.5
2021/12/17 09:04:33 Rebalance progress: 12.5
2021/12/17 09:04:38 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:04:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:04:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 09:04:38 Waiting for src bucket item count to get to: 50000 curr count: 7279
2021/12/17 09:04:43 Waiting for src bucket item count to get to: 50000 curr count: 20927
2021/12/17 09:04:48 Waiting for src bucket item count to get to: 50000 curr count: 20927
2021/12/17 09:04:53 Waiting for src bucket item count to get to: 50000 curr count: 33981
2021/12/17 09:04:58 Waiting for src bucket item count to get to: 50000 curr count: 33981
2021/12/17 09:05:03 Waiting for src bucket item count to get to: 50000 curr count: 44101
2021/12/17 09:05:08 Waiting for src bucket item count to get to: 50000 curr count: 44101
2021/12/17 09:05:13 Waiting for src bucket item count to get to: 50000 curr count: 44101
2021/12/17 09:05:23 Waiting for dst bucket item count to get to: 50000 curr count: 31364
2021/12/17 09:05:28 Waiting for dst bucket item count to get to: 50000 curr count: 31364
2021/12/17 09:05:33 Waiting for dst bucket item count to get to: 50000 curr count: 42104
2021/12/17 09:05:38 Waiting for dst bucket item count to get to: 50000 curr count: 42104
2021/12/17 09:05:38 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/17 09:05:38 Post SET ops. Expected item count: 50000 got 50000
2021/12/17 09:05:38 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 09:05:43 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/17 09:05:48 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/17 09:05:54 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/17 09:05:59 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/17 09:06:04 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/17 09:06:09 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/17 09:06:09 App: TestDuplicateTimerCreationKVRebOut got un-deployed
2021/12/17 09:06:09 No eventing-con process running
2021/12/17 09:06:09 Function: TestDuplicateTimerCreationKVRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationKVRebOut deleting in the background"
}
2021/12/17 09:06:10 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:06:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 09:06:21 Waiting for dst bucket item count to get to: 0 curr count: 27183
2021/12/17 09:06:26 Waiting for dst bucket item count to get to: 0 curr count: 27183
2021/12/17 09:06:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 09:06:27 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:06:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 09:06:37 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/17 09:06:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationKVRebOut (189.32s)
=== RUN   TestDuplicateTimerCreationKVSwapReb
2021/12/17 09:06:47 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/17 09:06:47 App: TestDuplicateTimerCreationKVSwapReb got un-deployed
2021/12/17 09:06:47 No eventing-con process running
2021/12/17 09:06:47 Function: TestDuplicateTimerCreationKVSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationKVSwapReb not found"
 }
}
2021/12/17 09:06:49 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:06:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 09:06:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 09:06:56 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:07:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 09:07:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:07:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 09:07:06 Deploying app: TestDuplicateTimerCreationKVSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/17 09:07:07 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationKVSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/17 09:07:12 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get deployed
2021/12/17 09:07:17 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get deployed
2021/12/17 09:07:22 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get deployed
2021/12/17 09:07:27 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get deployed
2021/12/17 09:07:27 App: TestDuplicateTimerCreationKVSwapReb got deployed
2021/12/17 09:07:27 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/17 09:07:27 Starting bucket ops to source bucket
2021/12/17 09:07:27 Removing node(s): [] from the cluster
2021/12/17 09:07:32 Rebalance progress: 12.5
2021/12/17 09:07:37 Rebalance progress: 12.5
2021/12/17 09:07:42 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:07:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:07:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 09:07:42 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/17 09:07:43 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/17 09:07:48 Rebalance progress: 12.5
2021/12/17 09:07:53 Rebalance progress: 12.5
2021/12/17 09:07:58 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:07:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:07:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 09:07:58 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/17 09:08:03 Rebalance progress: 12.5
2021/12/17 09:08:08 Rebalance progress: 12.5
2021/12/17 09:08:13 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:08:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:08:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 09:08:13 Waiting for src bucket item count to get to: 50000 curr count: 49080
2021/12/17 09:08:18 Waiting for src bucket item count to get to: 50000 curr count: 49080
2021/12/17 09:08:28 Waiting for dst bucket item count to get to: 50000 curr count: 35117
2021/12/17 09:08:33 Waiting for dst bucket item count to get to: 50000 curr count: 35117
2021/12/17 09:08:38 Waiting for dst bucket item count to get to: 50000 curr count: 41448
2021/12/17 09:08:43 Waiting for dst bucket item count to get to: 50000 curr count: 41448
2021/12/17 09:08:43 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/17 09:08:43 Post SET ops. Expected item count: 50000 got 50000
2021/12/17 09:08:43 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 09:08:48 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/17 09:08:53 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/17 09:08:58 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/17 09:09:03 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/17 09:09:08 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/17 09:09:13 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/17 09:09:13 App: TestDuplicateTimerCreationKVSwapReb got un-deployed
2021/12/17 09:09:13 No eventing-con process running
2021/12/17 09:09:13 Function: TestDuplicateTimerCreationKVSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationKVSwapReb deleting in the background"
}
2021/12/17 09:09:15 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:09:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 09:09:25 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/17 09:09:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 09:09:27 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:09:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 09:09:37 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/17 09:09:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationKVSwapReb (179.79s)
=== RUN   TestDuplicateTimerFiringEventingRebIn
2021/12/17 09:09:47 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/17 09:09:47 App: TestDuplicateTimerFiringEventingRebIn got un-deployed
2021/12/17 09:09:47 No eventing-con process running
2021/12/17 09:09:47 Function: TestDuplicateTimerFiringEventingRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringEventingRebIn not found"
 }
}
2021/12/17 09:09:49 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:09:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 09:09:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 09:09:56 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:10:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 09:10:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:10:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 09:10:06 Deploying app: TestDuplicateTimerFiringEventingRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/17 09:10:07 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringEventingRebIn' in metakv",
  "warnings": null
 }
}
2021/12/17 09:10:12 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get deployed
2021/12/17 09:10:17 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get deployed
2021/12/17 09:10:22 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get deployed
2021/12/17 09:10:27 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get deployed
2021/12/17 09:10:27 App: TestDuplicateTimerFiringEventingRebIn got deployed
2021/12/17 09:10:27 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/17 09:10:27 Starting bucket ops to source bucket
2021/12/17 09:10:27 Removing node(s): [] from the cluster
2021/12/17 09:10:32 Rebalance progress: 12.5
2021/12/17 09:10:37 Rebalance progress: 12.5
2021/12/17 09:10:42 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:10:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:10:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 09:10:42 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/17 09:10:47 Rebalance progress: 12.5
2021/12/17 09:10:52 Rebalance progress: 12.5
2021/12/17 09:10:57 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:10:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:10:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 09:10:57 Waiting for src bucket item count to get to: 50000 curr count: 26890
2021/12/17 09:11:02 Waiting for src bucket item count to get to: 50000 curr count: 37573
2021/12/17 09:11:07 Waiting for src bucket item count to get to: 50000 curr count: 37573
2021/12/17 09:11:12 Waiting for src bucket item count to get to: 50000 curr count: 49041
2021/12/17 09:11:18 Waiting for src bucket item count to get to: 50000 curr count: 49041
2021/12/17 09:11:28 Waiting for dst bucket item count to get to: 50000 curr count: 45820
2021/12/17 09:11:33 Waiting for dst bucket item count to get to: 50000 curr count: 45820
2021/12/17 09:11:33 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/17 09:11:33 Post SET ops. Expected item count: 50000 got 50000
2021/12/17 09:11:33 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 09:11:38 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/17 09:11:43 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/17 09:11:48 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/17 09:11:53 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/17 09:11:58 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/17 09:12:03 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/17 09:12:03 App: TestDuplicateTimerFiringEventingRebIn got un-deployed
2021/12/17 09:12:03 No eventing-con process running
2021/12/17 09:12:03 Function: TestDuplicateTimerFiringEventingRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringEventingRebIn deleting in the background"
}
2021/12/17 09:12:05 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:12:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 09:12:15 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/17 09:12:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 09:12:16 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:12:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 09:12:27 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/17 09:12:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringEventingRebIn (169.38s)
=== RUN   TestDuplicateTimerFiringEventingRebOut
2021/12/17 09:12:32 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/17 09:12:32 Removing node(s): [] from the cluster
2021/12/17 09:12:37 Rebalance progress: 100
2021/12/17 09:12:42 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/17 09:12:42 App: TestDuplicateTimerFiringEventingRebOut got un-deployed
2021/12/17 09:12:42 No eventing-con process running
2021/12/17 09:12:42 Function: TestDuplicateTimerFiringEventingRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringEventingRebOut not found"
 }
}
2021/12/17 09:12:44 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:12:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 09:12:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 09:12:51 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:12:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 09:12:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:13:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 09:13:01 Deploying app: TestDuplicateTimerFiringEventingRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/17 09:13:02 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringEventingRebOut' in metakv",
  "warnings": null
 }
}
2021/12/17 09:13:07 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get deployed
2021/12/17 09:13:12 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get deployed
2021/12/17 09:13:17 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get deployed
2021/12/17 09:13:22 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get deployed
2021/12/17 09:13:22 App: TestDuplicateTimerFiringEventingRebOut got deployed
2021/12/17 09:13:22 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/17 09:13:22 Starting bucket ops to source bucket
2021/12/17 09:13:27 Rebalance progress: 12.5
2021/12/17 09:13:32 Rebalance progress: 12.5
2021/12/17 09:13:37 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:13:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:13:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 09:13:37 Waiting for src bucket item count to get to: 50000 curr count: 9552
2021/12/17 09:13:42 Waiting for src bucket item count to get to: 50000 curr count: 22290
2021/12/17 09:13:47 Waiting for src bucket item count to get to: 50000 curr count: 22290
2021/12/17 09:13:52 Waiting for src bucket item count to get to: 50000 curr count: 33500
2021/12/17 09:13:57 Waiting for src bucket item count to get to: 50000 curr count: 33500
2021/12/17 09:14:02 Waiting for src bucket item count to get to: 50000 curr count: 43472
2021/12/17 09:14:07 Waiting for src bucket item count to get to: 50000 curr count: 43472
2021/12/17 09:14:12 Waiting for src bucket item count to get to: 50000 curr count: 43472
2021/12/17 09:14:22 Waiting for dst bucket item count to get to: 50000 curr count: 38626
2021/12/17 09:14:27 Waiting for dst bucket item count to get to: 50000 curr count: 38626
2021/12/17 09:14:27 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/17 09:14:27 Post SET ops. Expected item count: 50000 got 50000
2021/12/17 09:14:27 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 09:14:32 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/17 09:14:37 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/17 09:14:42 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/17 09:14:47 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/17 09:14:52 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/17 09:14:57 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/17 09:14:57 App: TestDuplicateTimerFiringEventingRebOut got un-deployed
2021/12/17 09:14:57 No eventing-con process running
2021/12/17 09:14:57 Function: TestDuplicateTimerFiringEventingRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringEventingRebOut deleting in the background"
}
2021/12/17 09:14:59 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:15:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 09:15:09 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/17 09:15:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 09:15:11 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:15:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 09:15:21 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/17 09:15:26 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/17 09:15:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringEventingRebOut (179.54s)
=== RUN   TestDuplicateTimerFiringEventingSwapReb
2021/12/17 09:15:36 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/17 09:15:36 App: TestDuplicateTimerFiringEventingSwapReb got un-deployed
2021/12/17 09:15:36 No eventing-con process running
2021/12/17 09:15:36 Function: TestDuplicateTimerFiringEventingSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringEventingSwapReb not found"
 }
}
2021/12/17 09:15:38 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:15:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 09:15:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 09:15:45 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:15:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 09:15:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:15:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 09:15:55 Deploying app: TestDuplicateTimerFiringEventingSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/17 09:15:56 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringEventingSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/17 09:16:01 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get deployed
2021/12/17 09:16:06 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get deployed
2021/12/17 09:16:11 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get deployed
2021/12/17 09:16:16 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get deployed
2021/12/17 09:16:16 App: TestDuplicateTimerFiringEventingSwapReb got deployed
2021/12/17 09:16:16 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/17 09:16:16 Starting bucket ops to source bucket
2021/12/17 09:16:16 Removing node(s): [] from the cluster
2021/12/17 09:16:21 Rebalance progress: 12.5
2021/12/17 09:16:26 Rebalance progress: 12.5
2021/12/17 09:16:31 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:16:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:16:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 09:16:31 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/17 09:16:31 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/17 09:16:37 Rebalance progress: 12.5
2021/12/17 09:16:42 Rebalance progress: 12.5
2021/12/17 09:16:47 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:16:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:16:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 09:16:47 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/17 09:16:52 Rebalance progress: 12.5
2021/12/17 09:16:57 Rebalance progress: 12.5
2021/12/17 09:17:02 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:17:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:17:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 09:17:02 Waiting for src bucket item count to get to: 50000 curr count: 48736
2021/12/17 09:17:07 Waiting for src bucket item count to get to: 50000 curr count: 48736
2021/12/17 09:17:17 Waiting for dst bucket item count to get to: 50000 curr count: 46274
2021/12/17 09:17:22 Waiting for dst bucket item count to get to: 50000 curr count: 46274
2021/12/17 09:17:22 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/17 09:17:22 Post SET ops. Expected item count: 50000 got 50000
2021/12/17 09:17:22 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 09:17:27 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/17 09:17:32 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/17 09:17:37 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/17 09:17:42 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/17 09:17:47 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/17 09:17:52 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/17 09:17:52 App: TestDuplicateTimerFiringEventingSwapReb got un-deployed
2021/12/17 09:17:52 No eventing-con process running
2021/12/17 09:17:52 Function: TestDuplicateTimerFiringEventingSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringEventingSwapReb deleting in the background"
}
2021/12/17 09:17:54 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:17:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 09:18:04 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/17 09:18:09 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/17 09:18:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 09:18:11 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:18:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 09:18:21 Waiting for dst bucket item count to get to: 0 curr count: 34632
2021/12/17 09:18:26 Waiting for dst bucket item count to get to: 0 curr count: 34632
2021/12/17 09:18:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringEventingSwapReb (179.65s)
=== RUN   TestDuplicateTimerFiringKVRebIn
2021/12/17 09:18:36 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/17 09:18:36 App: TestDuplicateTimerFiringKVRebIn got un-deployed
2021/12/17 09:18:36 No eventing-con process running
2021/12/17 09:18:36 Function: TestDuplicateTimerFiringKVRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringKVRebIn not found"
 }
}
2021/12/17 09:18:38 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:18:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 09:18:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 09:18:44 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:18:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 09:18:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:18:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 09:18:54 Deploying app: TestDuplicateTimerFiringKVRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/17 09:18:55 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringKVRebIn' in metakv",
  "warnings": null
 }
}
2021/12/17 09:19:00 Waiting for app: TestDuplicateTimerFiringKVRebIn to get deployed
2021/12/17 09:19:05 Waiting for app: TestDuplicateTimerFiringKVRebIn to get deployed
2021/12/17 09:19:10 Waiting for app: TestDuplicateTimerFiringKVRebIn to get deployed
2021/12/17 09:19:15 Waiting for app: TestDuplicateTimerFiringKVRebIn to get deployed
2021/12/17 09:19:15 App: TestDuplicateTimerFiringKVRebIn got deployed
2021/12/17 09:19:15 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/17 09:19:15 Starting bucket ops to source bucket
2021/12/17 09:19:16 Removing node(s): [] from the cluster
2021/12/17 09:19:21 Rebalance progress: 12.5
2021/12/17 09:19:26 Rebalance progress: 12.5
2021/12/17 09:19:31 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:19:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:19:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 09:19:31 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/17 09:19:36 Rebalance progress: 12.5
2021/12/17 09:19:41 Rebalance progress: 12.5
2021/12/17 09:19:46 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:19:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:19:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 09:19:46 Waiting for src bucket item count to get to: 50000 curr count: 29359
2021/12/17 09:19:51 Waiting for src bucket item count to get to: 50000 curr count: 39965
2021/12/17 09:19:56 Waiting for src bucket item count to get to: 50000 curr count: 39965
2021/12/17 09:20:06 Waiting for dst bucket item count to get to: 50000 curr count: 33674
2021/12/17 09:20:11 Waiting for dst bucket item count to get to: 50000 curr count: 33674
2021/12/17 09:20:16 Waiting for dst bucket item count to get to: 50000 curr count: 40468
2021/12/17 09:20:21 Waiting for dst bucket item count to get to: 50000 curr count: 40468
2021/12/17 09:20:21 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/17 09:20:21 Post SET ops. Expected item count: 50000 got 50000
2021/12/17 09:20:21 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 09:20:26 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/17 09:20:31 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/17 09:20:36 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/17 09:20:41 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/17 09:20:46 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/17 09:20:51 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/17 09:20:51 App: TestDuplicateTimerFiringKVRebIn got un-deployed
2021/12/17 09:20:51 No eventing-con process running
2021/12/17 09:20:51 Function: TestDuplicateTimerFiringKVRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringKVRebIn deleting in the background"
}
2021/12/17 09:20:53 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:20:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 09:21:03 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/17 09:21:08 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/17 09:21:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 09:21:10 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:21:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 09:21:20 Waiting for dst bucket item count to get to: 0 curr count: 2344
2021/12/17 09:21:25 Waiting for dst bucket item count to get to: 0 curr count: 2344
2021/12/17 09:21:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringKVRebIn (179.28s)
=== RUN   TestDuplicateTimerFiringKVRebOut
2021/12/17 09:21:30 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/17 09:21:31 Removing node(s): [] from the cluster
2021/12/17 09:21:36 Rebalance progress: 100
2021/12/17 09:21:41 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/17 09:21:41 App: TestDuplicateTimerFiringKVRebOut got un-deployed
2021/12/17 09:21:41 No eventing-con process running
2021/12/17 09:21:41 Function: TestDuplicateTimerFiringKVRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringKVRebOut not found"
 }
}
2021/12/17 09:21:43 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:21:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 09:21:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 09:21:49 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:21:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 09:21:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:21:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 09:21:59 Deploying app: TestDuplicateTimerFiringKVRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/17 09:22:01 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringKVRebOut' in metakv",
  "warnings": null
 }
}
2021/12/17 09:22:06 Waiting for app: TestDuplicateTimerFiringKVRebOut to get deployed
2021/12/17 09:22:11 Waiting for app: TestDuplicateTimerFiringKVRebOut to get deployed
2021/12/17 09:22:16 Waiting for app: TestDuplicateTimerFiringKVRebOut to get deployed
2021/12/17 09:22:21 Waiting for app: TestDuplicateTimerFiringKVRebOut to get deployed
2021/12/17 09:22:21 App: TestDuplicateTimerFiringKVRebOut got deployed
2021/12/17 09:22:21 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/17 09:22:21 Starting bucket ops to source bucket
2021/12/17 09:22:26 Rebalance progress: 12.5
2021/12/17 09:22:31 Rebalance progress: 12.5
2021/12/17 09:22:36 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:22:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:22:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 09:22:36 Waiting for src bucket item count to get to: 50000 curr count: 11461
2021/12/17 09:22:41 Waiting for src bucket item count to get to: 50000 curr count: 23851
2021/12/17 09:22:46 Waiting for src bucket item count to get to: 50000 curr count: 23851
2021/12/17 09:22:51 Waiting for src bucket item count to get to: 50000 curr count: 33969
2021/12/17 09:22:56 Waiting for src bucket item count to get to: 50000 curr count: 33969
2021/12/17 09:23:01 Waiting for src bucket item count to get to: 50000 curr count: 33969
2021/12/17 09:23:06 Waiting for src bucket item count to get to: 50000 curr count: 45119
2021/12/17 09:23:11 Waiting for src bucket item count to get to: 50000 curr count: 45119
2021/12/17 09:23:21 Waiting for dst bucket item count to get to: 50000 curr count: 40482
2021/12/17 09:23:26 Waiting for dst bucket item count to get to: 50000 curr count: 40482
2021/12/17 09:23:26 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/17 09:23:26 Post SET ops. Expected item count: 50000 got 50000
2021/12/17 09:23:26 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 09:23:31 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/17 09:23:36 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/17 09:23:41 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/17 09:23:46 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/17 09:23:51 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/17 09:23:56 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/17 09:23:56 App: TestDuplicateTimerFiringKVRebOut got un-deployed
2021/12/17 09:23:56 No eventing-con process running
2021/12/17 09:23:56 Function: TestDuplicateTimerFiringKVRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringKVRebOut deleting in the background"
}
2021/12/17 09:23:58 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:24:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 09:24:08 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/17 09:24:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 09:24:10 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:24:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 09:24:20 Waiting for dst bucket item count to get to: 0 curr count: 1769
2021/12/17 09:24:25 Waiting for dst bucket item count to get to: 0 curr count: 1769
2021/12/17 09:24:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringKVRebOut (179.96s)
=== RUN   TestDuplicateTimerFiringKVSwapReb
2021/12/17 09:24:35 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/17 09:24:35 App: TestDuplicateTimerFiringKVSwapReb got un-deployed
2021/12/17 09:24:35 No eventing-con process running
2021/12/17 09:24:35 Function: TestDuplicateTimerFiringKVSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringKVSwapReb not found"
 }
}
2021/12/17 09:24:37 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:24:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 09:24:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 09:24:43 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:24:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 09:24:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:24:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 09:24:54 Deploying app: TestDuplicateTimerFiringKVSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/17 09:24:55 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringKVSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/17 09:25:00 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get deployed
2021/12/17 09:25:05 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get deployed
2021/12/17 09:25:10 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get deployed
2021/12/17 09:25:15 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get deployed
2021/12/17 09:25:15 App: TestDuplicateTimerFiringKVSwapReb got deployed
2021/12/17 09:25:15 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/17 09:25:15 Starting bucket ops to source bucket
2021/12/17 09:25:15 Removing node(s): [] from the cluster
2021/12/17 09:25:20 Rebalance progress: 12.5
2021/12/17 09:25:25 Rebalance progress: 12.5
2021/12/17 09:25:30 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:25:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:25:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 09:25:30 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/17 09:25:31 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/17 09:25:36 Rebalance progress: 12.5
2021/12/17 09:25:41 Rebalance progress: 12.5
2021/12/17 09:25:46 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:25:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:25:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 09:25:46 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/17 09:25:51 Rebalance progress: 12.5
2021/12/17 09:25:56 Rebalance progress: 12.5
2021/12/17 09:26:01 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:26:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:26:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 09:26:01 Waiting for src bucket item count to get to: 50000 curr count: 49347
2021/12/17 09:26:06 Waiting for src bucket item count to get to: 50000 curr count: 49347
2021/12/17 09:26:16 Waiting for dst bucket item count to get to: 50000 curr count: 45834
2021/12/17 09:26:21 Waiting for dst bucket item count to get to: 50000 curr count: 45834
2021/12/17 09:26:21 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/17 09:26:21 Post SET ops. Expected item count: 50000 got 50000
2021/12/17 09:26:21 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 09:26:26 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/17 09:26:31 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/17 09:26:36 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/17 09:26:41 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/17 09:26:46 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/17 09:26:51 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/17 09:26:51 App: TestDuplicateTimerFiringKVSwapReb got un-deployed
2021/12/17 09:26:51 No eventing-con process running
2021/12/17 09:26:51 Function: TestDuplicateTimerFiringKVSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringKVSwapReb deleting in the background"
}
2021/12/17 09:26:53 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:26:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 09:27:03 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/17 09:27:08 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/17 09:27:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 09:27:10 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:27:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 09:27:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringKVSwapReb (169.93s)
=== RUN   TestKVRebOnUpdateBucketOpOneByOne
2021/12/17 09:27:25 Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 09:27:25 App: bucket_op_on_update got un-deployed
2021/12/17 09:27:25 No eventing-con process running
2021/12/17 09:27:25 Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/17 09:27:27 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:27:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 09:27:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 09:27:33 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:27:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 09:27:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:27:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 09:27:43 Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/17 09:27:45 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/17 09:27:50 Starting bucket ops to source bucket
2021/12/17 09:27:55 Waiting for app: bucket_op_on_update to get deployed
2021/12/17 09:28:00 Waiting for app: bucket_op_on_update to get deployed
2021/12/17 09:28:05 Waiting for app: bucket_op_on_update to get deployed
2021/12/17 09:28:05 App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:28:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:28:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 09:28:05 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/17 09:28:05 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/17 09:28:10 Rebalance progress: 12.5
2021/12/17 09:28:15 Rebalance progress: 12.5
2021/12/17 09:28:20 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:28:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:28:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 09:28:20 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/17 09:28:21 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/17 09:28:26 Rebalance progress: 12.5
2021/12/17 09:28:31 Rebalance progress: 12.5
2021/12/17 09:28:36 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:28:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:28:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 09:28:36 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: kv to the cluster
2021/12/17 09:28:36 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/17 09:28:41 Rebalance progress: 12.5
2021/12/17 09:28:46 Rebalance progress: 12.5
2021/12/17 09:28:51 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:28:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:28:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 09:28:51 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/17 09:28:56 Rebalance progress: 12.5
2021/12/17 09:29:01 Rebalance progress: 12.5
2021/12/17 09:29:06 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:29:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:29:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 09:29:06 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/17 09:29:12 Rebalance progress: 12.5
2021/12/17 09:29:17 Rebalance progress: 12.5
2021/12/17 09:29:22 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:29:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:29:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 09:29:22 clusterSetup.go:253: Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/17 09:29:27 Rebalance progress: 12.5
2021/12/17 09:29:32 Rebalance progress: 12.5
2021/12/17 09:29:37 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:29:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:29:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 09:29:37 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 09:29:42 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 09:29:47 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 09:29:52 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 09:29:57 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 09:30:02 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 09:30:07 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 09:30:07 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/17 09:30:07 eventingSetup.go:46: No eventing-con process running
2021/12/17 09:30:07 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/17 09:30:09 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:30:14 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 09:30:19 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 10703
2021/12/17 09:30:19 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 09:30:21 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:30:26 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 09:30:31 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6896
2021/12/17 09:30:36 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6896
2021/12/17 09:30:36 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebOnUpdateBucketOpOneByOne (200.74s)
=== RUN   TestKVForceFailover
2021/12/17 09:30:51 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 09:30:51 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/17 09:30:51 eventingSetup.go:46: No eventing-con process running
2021/12/17 09:30:51 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/17 09:30:52 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:30:57 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 09:30:57 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 09:30:59 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:31:04 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 09:31:04 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 09:31:04 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/17 09:31:04 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/17 09:31:09 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:31:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 09:31:09 Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/17 09:31:11 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/17 09:31:11 Starting bucket ops to source bucket
2021/12/17 09:31:16 Waiting for app: bucket_op_on_update to get deployed
2021/12/17 09:31:21 Waiting for app: bucket_op_on_update to get deployed
2021/12/17 09:31:26 Waiting for app: bucket_op_on_update to get deployed
2021/12/17 09:31:31 Waiting for app: bucket_op_on_update to get deployed
2021/12/17 09:31:31 App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:31:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:31:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 09:31:31 clusterSetup.go:229: Failing over: [127.0.0.1:19001]
2021/12/17 09:31:51 clusterSetup.go:237: Kicking off failover recovery, type: full
2021/12/17 09:31:51 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/17 09:31:56 Rebalance progress: 12.5
2021/12/17 09:32:01 Rebalance progress: 12.5
2021/12/17 09:32:06 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:32:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:32:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 09:32:06 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002] from the cluster
2021/12/17 09:32:11 Rebalance progress: 12.5
2021/12/17 09:32:16 Rebalance progress: 12.5
2021/12/17 09:32:21 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:32:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:32:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 09:32:21 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 09:32:26 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 09:32:31 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 09:32:36 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 09:32:41 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 09:32:46 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 09:32:51 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 09:32:51 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/17 09:32:51 eventingSetup.go:46: No eventing-con process running
2021/12/17 09:32:52 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/17 09:32:53 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:32:58 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 09:33:03 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7038
2021/12/17 09:33:08 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7038
2021/12/17 09:33:08 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 09:33:10 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:33:15 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 09:33:20 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1045
2021/12/17 09:33:25 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1045
2021/12/17 09:33:25 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVForceFailover (169.47s)
=== RUN   TestKVHardFailoverRecovery
2021/12/17 09:33:35 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 09:33:35 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/17 09:33:35 eventingSetup.go:46: No eventing-con process running
2021/12/17 09:33:35 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/17 09:33:37 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:33:42 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 09:33:42 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 09:33:44 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:33:49 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 09:33:49 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:33:54 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 09:33:54 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/17 09:33:55 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/17 09:34:00 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/17 09:34:05 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/17 09:34:10 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/17 09:34:15 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/17 09:34:15 clusterSetup.go:354: App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:34:15 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:34:15 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 09:34:15 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/17 09:34:15 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/17 09:34:20 Rebalance progress: 12.5
2021/12/17 09:34:25 Rebalance progress: 12.5
2021/12/17 09:34:30 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:34:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:34:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 09:34:30 clusterSetup.go:229: Failing over: [127.0.0.1:19001]
2021/12/17 09:35:30 clusterSetup.go:237: Kicking off failover recovery, type: full
2021/12/17 09:35:36 Rebalance progress: 12.5
2021/12/17 09:35:41 Rebalance progress: 12.5
2021/12/17 09:35:46 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:35:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:35:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 09:35:46 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/17 09:35:51 Rebalance progress: 12.5
2021/12/17 09:35:56 Rebalance progress: 12.5
2021/12/17 09:36:01 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:36:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:36:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 09:36:01 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 09:36:06 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 09:36:11 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 09:36:16 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 09:36:21 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 09:36:26 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 09:36:31 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 09:36:31 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/17 09:36:31 eventingSetup.go:46: No eventing-con process running
2021/12/17 09:36:31 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/17 09:36:33 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:36:38 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 09:36:43 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12081
2021/12/17 09:36:48 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12081
2021/12/17 09:36:48 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 09:36:50 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:36:55 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 09:36:55 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVHardFailoverRecovery (209.62s)
=== RUN   TestKVRebOnUpdateBucketOpAllAtOnce
2021/12/17 09:37:05 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 09:37:05 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/17 09:37:05 eventingSetup.go:46: No eventing-con process running
2021/12/17 09:37:05 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/17 09:37:07 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:37:12 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 09:37:12 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 09:37:13 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:37:18 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 09:37:18 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:37:23 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 09:37:23 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/17 09:37:24 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/17 09:37:29 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/17 09:37:34 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/17 09:37:39 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/17 09:37:44 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/17 09:37:44 clusterSetup.go:354: App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:37:44 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:37:44 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 09:37:45 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/17 09:37:45 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/17 09:37:45 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: kv to the cluster
2021/12/17 09:37:45 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/17 09:37:50 Rebalance progress: 12.5
2021/12/17 09:37:55 Rebalance progress: 12.5
2021/12/17 09:38:00 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:38:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:38:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 09:38:00 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002 127.0.0.1:19003] from the cluster
2021/12/17 09:38:05 Rebalance progress: 12.5
2021/12/17 09:38:10 Rebalance progress: 12.5
2021/12/17 09:38:15 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:38:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:38:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 09:38:16 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 09:38:21 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 09:38:26 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 09:38:31 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 09:38:36 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 09:38:41 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 09:38:46 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 09:38:46 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/17 09:38:46 eventingSetup.go:46: No eventing-con process running
2021/12/17 09:38:46 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/17 09:38:48 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:38:53 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 09:38:58 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4591
2021/12/17 09:38:58 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 09:38:59 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:39:04 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 09:39:04 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebOnUpdateBucketOpAllAtOnce (129.70s)
=== RUN   TestKVSwapRebOnUpdateBucketOp
2021/12/17 09:39:14 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 09:39:14 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/17 09:39:14 eventingSetup.go:46: No eventing-con process running
2021/12/17 09:39:14 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/17 09:39:16 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:39:21 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 09:39:21 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 09:39:23 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:39:28 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 09:39:28 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:39:28 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 09:39:28 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/17 09:39:29 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/17 09:39:34 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/17 09:39:39 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/17 09:39:44 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/17 09:39:49 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/17 09:39:49 clusterSetup.go:354: App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:39:49 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:39:49 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 09:39:49 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/17 09:39:50 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/17 09:39:55 Rebalance progress: 12.5
2021/12/17 09:40:00 Rebalance progress: 12.5
2021/12/17 09:40:05 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:40:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:40:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 09:40:05 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/17 09:40:05 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/17 09:40:10 Rebalance progress: 12.5
2021/12/17 09:40:15 Rebalance progress: 12.5
2021/12/17 09:40:20 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:40:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:40:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 09:40:20 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/17 09:40:25 Rebalance progress: 12.5
2021/12/17 09:40:30 Rebalance progress: 12.5
2021/12/17 09:40:35 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:40:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:40:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 09:40:35 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 09:40:40 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 09:40:45 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 09:40:50 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 09:40:55 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 09:41:00 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 09:41:06 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 09:41:06 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/17 09:41:06 eventingSetup.go:46: No eventing-con process running
2021/12/17 09:41:06 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/17 09:41:07 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:41:12 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 09:41:17 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6100
2021/12/17 09:41:17 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 09:41:19 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:41:24 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 09:41:29 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6100
2021/12/17 09:41:29 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 09:41:34 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 09:41:34 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/17 09:41:34 eventingSetup.go:46: No eventing-con process running
2021/12/17 09:41:34 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/17 09:41:36 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:41:41 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 09:41:41 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 09:41:43 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:41:48 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 09:41:48 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVSwapRebOnUpdateBucketOp (163.31s)
=== RUN   TestKVRebalanceWithMultipleHandlers
2021/12/17 09:41:58 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 09:41:58 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/17 09:41:58 eventingSetup.go:46: No eventing-con process running
2021/12/17 09:41:58 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/17 09:41:59 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:42:04 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 09:42:04 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 09:42:06 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:42:11 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 09:42:11 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 09:42:16 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/17 09:42:16 clusterSetup.go:405: App: sys_test_bucket_op got un-deployed
2021/12/17 09:42:16 eventingSetup.go:46: No eventing-con process running
2021/12/17 09:42:16 eventingSetup.go:418: Function: sys_test_bucket_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: sys_test_bucket_op not found"
 }
}
2021/12/17 09:42:18 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:42:23 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 09:42:23 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 09:42:25 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:42:30 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 09:42:30 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:42:30 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 09:42:30 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/17 09:42:31 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:42:31 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 09:42:31 eventingSetup.go:125: Deploying app: sys_test_bucket_op. Item count src bucket: 0 dst bucket: 0
2021/12/17 09:42:32 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'sys_test_bucket_op' in metakv",
  "warnings": null
 }
}
2021/12/17 09:42:37 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/17 09:42:42 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/17 09:42:47 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/17 09:42:52 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/17 09:42:52 clusterSetup.go:354: App: bucket_op_on_update got deployed
2021/12/17 09:42:57 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/17 09:43:02 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/17 09:43:07 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/17 09:43:07 clusterSetup.go:354: App: sys_test_bucket_op got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:43:07 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:43:07 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 09:43:07 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/17 09:43:08 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/17 09:43:13 Rebalance progress: 12.5
2021/12/17 09:43:18 Rebalance progress: 12.5
2021/12/17 09:43:23 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:43:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:43:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 09:43:23 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/17 09:43:23 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/17 09:43:28 Rebalance progress: 12.5
2021/12/17 09:43:33 Rebalance progress: 12.5
2021/12/17 09:43:38 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:43:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:43:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 09:43:38 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: kv to the cluster
2021/12/17 09:43:39 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/17 09:43:44 Rebalance progress: 12.5
2021/12/17 09:43:49 Rebalance progress: 12.5
2021/12/17 09:43:54 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:43:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:43:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 09:43:54 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/17 09:43:59 Rebalance progress: 12.5
2021/12/17 09:44:04 Rebalance progress: 12.5
2021/12/17 09:44:09 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:44:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:44:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 09:44:09 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/17 09:44:14 Rebalance progress: 12.5
2021/12/17 09:44:19 Rebalance progress: 12.5
2021/12/17 09:44:24 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:44:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:44:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 09:44:24 clusterSetup.go:253: Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/17 09:44:29 Rebalance progress: 12.5
2021/12/17 09:44:34 Rebalance progress: 12.5
2021/12/17 09:44:39 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:44:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:44:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 09:44:39 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 09:44:44 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 09:44:49 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 09:44:54 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 09:44:59 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 09:45:04 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 09:45:09 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 09:45:14 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 09:45:14 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/17 09:45:14 eventingSetup.go:46: No eventing-con process running
2021/12/17 09:45:15 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/17 09:45:18 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:45:23 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 09:45:28 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12183
2021/12/17 09:45:28 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 09:45:30 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:45:35 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 09:45:35 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 09:45:35 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 09:45:40 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/17 09:45:45 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/17 09:45:50 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/17 09:45:55 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/17 09:46:00 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/17 09:46:05 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/17 09:46:05 clusterSetup.go:405: App: sys_test_bucket_op got un-deployed
2021/12/17 09:46:05 eventingSetup.go:46: No eventing-con process running
2021/12/17 09:46:05 eventingSetup.go:418: Function: sys_test_bucket_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: sys_test_bucket_op deleting in the background"
}
2021/12/17 09:46:07 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:46:12 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 09:46:12 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 09:46:14 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:46:19 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 09:46:19 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebalanceWithMultipleHandlers (271.12s)
=== RUN   TestKVSwapRebalanceWithMultipleHandlers
2021/12/17 09:46:29 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 09:46:29 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/17 09:46:29 eventingSetup.go:46: No eventing-con process running
2021/12/17 09:46:29 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/17 09:46:31 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:46:36 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 09:46:36 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 09:46:37 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:46:42 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 09:46:42 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 09:46:47 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/17 09:46:47 clusterSetup.go:405: App: sys_test_bucket_op got un-deployed
2021/12/17 09:46:47 eventingSetup.go:46: No eventing-con process running
2021/12/17 09:46:47 eventingSetup.go:418: Function: sys_test_bucket_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: sys_test_bucket_op not found"
 }
}
2021/12/17 09:46:49 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:46:54 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 09:46:54 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 09:46:56 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:47:01 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 09:47:01 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:47:01 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 09:47:01 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/17 09:47:02 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:47:02 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 09:47:02 eventingSetup.go:125: Deploying app: sys_test_bucket_op. Item count src bucket: 0 dst bucket: 0
2021/12/17 09:47:03 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'sys_test_bucket_op' in metakv",
  "warnings": null
 }
}
2021/12/17 09:47:08 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/17 09:47:13 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/17 09:47:18 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/17 09:47:23 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/17 09:47:23 clusterSetup.go:354: App: bucket_op_on_update got deployed
2021/12/17 09:47:28 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/17 09:47:33 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/17 09:47:38 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/17 09:47:38 clusterSetup.go:354: App: sys_test_bucket_op got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:47:38 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:47:38 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 09:47:38 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/17 09:47:39 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/17 09:47:44 Rebalance progress: 12.5
2021/12/17 09:47:49 Rebalance progress: 12.5
2021/12/17 09:47:54 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:47:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:47:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 09:47:54 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/17 09:47:55 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/17 09:48:00 Rebalance progress: 12.5
2021/12/17 09:48:05 Rebalance progress: 12.5
2021/12/17 09:48:10 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:48:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:48:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 09:48:10 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/17 09:48:15 Rebalance progress: 12.5
2021/12/17 09:48:20 Rebalance progress: 12.5
2021/12/17 09:48:25 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:48:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:48:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 09:48:25 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 09:48:30 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 09:48:35 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 09:48:40 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 09:48:45 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 09:48:50 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 09:48:55 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 09:48:55 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/17 09:48:55 eventingSetup.go:46: No eventing-con process running
2021/12/17 09:48:55 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/17 09:48:59 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:49:04 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 09:49:04 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 09:49:06 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:49:11 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 09:49:16 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7649
2021/12/17 09:49:16 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 09:49:16 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 09:49:21 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/17 09:49:26 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/17 09:49:31 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/17 09:49:37 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/17 09:49:42 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/17 09:49:42 clusterSetup.go:405: App: sys_test_bucket_op got un-deployed
2021/12/17 09:49:42 eventingSetup.go:46: No eventing-con process running
2021/12/17 09:49:42 eventingSetup.go:418: Function: sys_test_bucket_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: sys_test_bucket_op deleting in the background"
}
2021/12/17 09:49:43 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:49:48 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 09:49:48 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 09:49:50 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:49:55 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 09:49:55 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVSwapRebalanceWithMultipleHandlers (216.39s)
=== RUN   TestKVRebStopStartKVOpsOnUpdateBucketOpOneByOne
2021/12/17 09:50:05 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 09:50:05 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/17 09:50:05 eventingSetup.go:46: No eventing-con process running
2021/12/17 09:50:05 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/17 09:50:07 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:50:12 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 09:50:12 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 09:50:14 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:50:19 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 09:50:19 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:50:24 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 09:50:24 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/17 09:50:25 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/17 09:50:30 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/17 09:50:35 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/17 09:50:40 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/17 09:50:45 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/17 09:50:45 clusterSetup.go:354: App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:50:45 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:50:45 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 09:50:45 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/17 09:50:45 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/17 09:51:06 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:51:06 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:51:06 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:51:16 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:51:16 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 09:51:16 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/17 09:51:21 Rebalance progress: 12.5
2021/12/17 09:51:26 Rebalance progress: 12.5
2021/12/17 09:51:31 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:51:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:51:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 09:51:31 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/17 09:51:31 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/17 09:51:36 Rebalance progress: 12.5
2021/12/17 09:51:41 Rebalance progress: 12.5
2021/12/17 09:51:46 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:51:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:51:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 09:51:46 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/17 09:51:52 Rebalance progress: 12.5
2021/12/17 09:51:57 Rebalance progress: 12.5
2021/12/17 09:52:02 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:52:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:52:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 09:52:02 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 09:52:07 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 09:52:12 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 09:52:17 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 09:52:22 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 09:52:27 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 09:52:32 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 09:52:32 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/17 09:52:32 eventingSetup.go:46: No eventing-con process running
2021/12/17 09:52:32 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/17 09:52:34 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:52:39 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 09:52:44 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 9156
2021/12/17 09:52:49 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 9156
2021/12/17 09:52:49 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 09:52:50 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:52:55 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 09:53:00 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4849
2021/12/17 09:53:05 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4849
2021/12/17 09:53:05 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebStopStartKVOpsOnUpdateBucketOpOneByOne (190.26s)
=== RUN   TestKVFailoverOnUpdateBucketOp
2021/12/17 09:53:15 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 09:53:15 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/17 09:53:15 eventingSetup.go:46: No eventing-con process running
2021/12/17 09:53:15 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/17 09:53:17 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:53:22 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 09:53:22 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 09:53:24 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:53:29 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 09:53:29 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:53:34 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 09:53:34 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/17 09:53:35 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/17 09:53:40 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/17 09:53:45 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/17 09:53:50 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/17 09:53:55 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/17 09:53:55 clusterSetup.go:354: App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:53:55 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:53:55 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 09:53:55 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/17 09:53:56 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/17 09:54:01 Rebalance progress: 12.5
2021/12/17 09:54:06 Rebalance progress: 12.5
2021/12/17 09:54:11 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:54:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:54:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 09:54:11 clusterSetup.go:229: Failing over: [127.0.0.1:19001]
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:54:21 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:54:21 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 09:54:21 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/17 09:54:26 Rebalance progress: 12.5
2021/12/17 09:54:31 Rebalance progress: 12.5
2021/12/17 09:54:36 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:54:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:54:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 09:54:36 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/17 09:54:36 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/17 09:54:41 Rebalance progress: 12.5
2021/12/17 09:54:46 Rebalance progress: 12.5
2021/12/17 09:54:51 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:54:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:54:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 09:54:51 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/17 09:54:56 Rebalance progress: 12.5
2021/12/17 09:55:01 Rebalance progress: 12.5
2021/12/17 09:55:06 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:55:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:55:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 09:55:07 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 09:55:12 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 09:55:17 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 09:55:22 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 09:55:27 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 09:55:32 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 09:55:37 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 09:55:37 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/17 09:55:37 eventingSetup.go:46: No eventing-con process running
2021/12/17 09:55:37 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/17 09:55:39 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:55:44 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 09:55:49 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 8610
2021/12/17 09:55:49 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 09:55:50 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:55:55 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 09:56:00 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4535
2021/12/17 09:56:05 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4535
2021/12/17 09:56:05 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVFailoverOnUpdateBucketOp (180.00s)
=== RUN   TestBootstrapAfterKVHardFailover
2021/12/17 09:56:10 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/17 09:56:11 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/17 09:56:16 Rebalance progress: 100
2021/12/17 09:56:16 Failing over: [127.0.0.1:19001]
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:56:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 09:56:46 Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/17 09:56:47 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/17 09:56:52 Removing node(s): [] from the cluster
2021/12/17 09:56:52 Starting bucket ops to source bucket
2021/12/17 09:56:57 Rebalance failed. See logs for detailed reason. You can try again.
2021/12/17 09:57:02 Waiting for app: bucket_op_on_update to get deployed
2021/12/17 09:57:07 Waiting for app: bucket_op_on_update to get deployed
2021/12/17 09:57:07 App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:57:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:57:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 09:57:07 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 09:57:12 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 09:57:17 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 09:57:22 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 09:57:27 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 09:57:32 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 09:57:37 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 09:57:37 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/17 09:57:37 eventingSetup.go:46: No eventing-con process running
2021/12/17 09:57:37 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/17 09:57:39 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:57:44 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 09:57:49 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1506
2021/12/17 09:57:49 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 09:57:51 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 09:57:56 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 09:58:01 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1296
2021/12/17 09:58:06 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1296
2021/12/17 09:58:06 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestBootstrapAfterKVHardFailover (120.46s)
PASS
ok  	github.com/couchbase/eventing/tests/functional_tests	12114.713s

Rebalance tests took 3 hours, 21 minutes and 57 seconds.

Testing Succeeded


Version: versions-17.12.2021-02.11.cfg
Build Log: make-17.12.2021-02.11.log
Server Log: logs-17.12.2021-02.11.tar.gz

Finished