Building

Started building at 2021/12/15 02:27:25
Calculating base
Updating mirror
Basing run on 7.1.0-1910 57bbcbbc7
Updating tree for run 15.12.2021-02.27
gocb is at 635fa6c, changes since last good build: none
libcouchbase is at f6822fc, changes since last good build: none
indexing is at 8cc3b2c, changes since last good build: none
kv_engine is at 1fac684, changes since last good build: 
 1fac684b6 MB-49727: Turn vbstate check into assertion in VB::fetchValueForWrite
 3c4483a9c MB-49727: Remove double vbstate check in VBucket::fetchValidValue
ns_server is at abe4a02, changes since last good build: none
couchstore is at 6eea212, changes since last good build: 
 6eea212 Clang-Tidy: dbdump.cc
couchdb is at fca2e6b, changes since last good build: none
testrunner is at 842cf83, changes since last good build: 
 842cf8356 CBQE-7346: Pass arch and server type to executor job
 3d6768ced Fix az terminate command
Switching eventing to unstable
eventing is at 9fce322, changes since last good build: none
Switching eventing-ee to unstable
eventing-ee is at 3ae49c5, changes since last good build: none
Pulling in uncommitted change 160397 at refs/changes/97/160397/3
Total 23 (delta 15), reused 17 (delta 15)
[unstable 63368b4b] MB-100 : Do not fail n1ql timeout test if we are catching timeout exceptions successfully
 Author: abhijpes 
 Date: Mon Aug 30 08:24:38 2021 +0530
 1 file changed, 2 insertions(+), 2 deletions(-)
Pulling in uncommitted change 163449 at refs/changes/49/163449/1
Total 48 (delta 27), reused 39 (delta 27)
[unstable 713e32cb] MB-100: collect data when test fails
 Author: AnkitPrabhu 
 Date: Tue Oct 12 14:02:31 2021 +0530
 1 file changed, 2 insertions(+)
Pulling in uncommitted change 165976 at refs/changes/76/165976/17
Total 4 (delta 3), reused 4 (delta 3)
[unstable c23f9aba] MB-48538: Add new audit ids for different events
 Author: AnkitPrabhu 
 Date: Tue Nov 23 14:29:10 2021 +0530
 1 file changed, 337 insertions(+), 61 deletions(-)
Pulling in uncommitted change 166697 at refs/changes/97/166697/4
Total 10 (delta 9), reused 10 (delta 9)
[unstable c4aeefe7] MB-49547: Check bucket instance to get kv node for dcp request
 Author: AnkitPrabhu 
 Date: Thu Dec 2 12:22:17 2021 +0530
 6 files changed, 38 insertions(+), 109 deletions(-)
Pulling in uncommitted change 166698 at refs/changes/98/166698/3
Total 4 (delta 3), reused 4 (delta 3)
[unstable 99dbd577] MB-48334 : Fail rebalance unless all functions deployed on current encryption level
 Author: abhijpes 
 Date: Wed Dec 1 14:51:29 2021 +0530
 1 file changed, 9 insertions(+)
Pulling in uncommitted change 166812 at refs/changes/12/166812/6
Total 24 (delta 20), reused 24 (delta 20)
[unstable 46184458] MB-49547: Refresh bucket info from streaming endpoint instance
 Author: AnkitPrabhu 
 Date: Thu Dec 2 13:55:55 2021 +0530
 7 files changed, 147 insertions(+), 235 deletions(-)
Pulling in uncommitted change 166999 at refs/changes/99/166999/9
Total 56 (delta 47), reused 56 (delta 47)
[unstable 0a599cf2] MB-46304: Add functionality to disable curl feature
 Author: AnkitPrabhu 
 Date: Mon Dec 6 13:26:12 2021 +0530
 20 files changed, 193 insertions(+), 30 deletions(-)
Pulling in uncommitted change 167196 at refs/changes/96/167196/7
Total 67 (delta 57), reused 67 (delta 57)
[unstable 3a4a9652] MB-49720: Store failover log in checkpoint blob
 Author: AnkitPrabhu 
 Date: Wed Dec 8 15:16:22 2021 +0530
 5 files changed, 149 insertions(+), 74 deletions(-)
Pulling in uncommitted change 167542 at refs/changes/42/167542/4
Total 82 (delta 64), reused 74 (delta 64)
[unstable 3f8446ff] MB-49570: Start dcp stream for a single collection id
 Author: AnkitPrabhu 
 Date: Tue Dec 14 14:27:25 2021 +0530
 7 files changed, 52 insertions(+), 29 deletions(-)
Pulling in uncommitted change 167544 at refs/changes/44/167544/2
Total 90 (delta 69), reused 80 (delta 69)
[unstable b4958794] MB-49570: Add backlog stats on ui
 Author: AnkitPrabhu 
 Date: Tue Dec 14 14:28:01 2021 +0530
 2 files changed, 18 insertions(+), 1 deletion(-)
Pulling in uncommitted change 167183 at refs/changes/83/167183/3
Total 5 (delta 2), reused 5 (delta 2)
[unstable 5c1f654] MB-46304: Throw runtime exception when curl feature is disabled
 Author: AnkitPrabhu 
 Date: Wed Dec 8 10:39:19 2021 +0530
 1 file changed, 9 insertions(+), 3 deletions(-)
Building cmakefiles and deps
Building main product

Testing

Started testing at 2021/12/15 02:43:39
Testing mode: rebalance,n1ql,functional
Starting server
Starting server: attempt 1

Functional tests

2021/12/15 02:45:14 Started functional
Post "http://127.0.0.1:9000/pools/default": EOF
Quota setup Post "http://127.0.0.1:9000/pools/default": EOF
2021/12/15 02:45:25 Adding node: https://127.0.0.1:19001 with role: kv,index,n1ql to the cluster
2021/12/15 02:45:30 Error in adding nodes : , response: {"otpNode":"n_1@127.0.0.1"}
2021/12/15 02:45:30 Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/15 02:45:35 Removing node(s): [] from the cluster
2021/12/15 02:45:40 Rebalance progress: 0
2021/12/15 02:45:46 Rebalance progress: 0
2021/12/15 02:45:50 Rebalance progress: 4.521122685185185
2021/12/15 02:45:55 Rebalance progress: 9.223090277777777
2021/12/15 02:46:00 Rebalance progress: 12.53255208333333
2021/12/15 02:46:05 Rebalance progress: 17.08984375
2021/12/15 02:46:10 Rebalance progress: 20.45355902777778
2021/12/15 02:46:15 Rebalance progress: 25.06510416666667
2021/12/15 02:46:20 Rebalance progress: 100
2021/12/15 02:46:23 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/15 02:46:38 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/15 02:46:53 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/15 02:47:08 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/15 02:47:23 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/15 02:47:38 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/15 02:47:53 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/15 02:48:08 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/15 02:48:23 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/15 02:48:38 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/15 02:48:53 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/15 02:49:08 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/15 02:49:23 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/15 02:49:38 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/15 02:49:53 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
=== RUN   TestAdvancedGetOps
2021/12/15 02:50:11 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/15 02:50:26 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/15 02:50:41 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/15 02:50:56 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/15 02:51:11 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/15 02:51:26 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/15 02:51:41 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/15 02:51:56 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/15 02:52:11 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/15 02:52:26 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/15 02:52:41 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/15 02:52:56 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/15 02:53:11 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/15 02:53:26 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/15 02:53:41 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/15 02:53:56 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/15 02:54:11 Deploying app: TestAdvancedGetOps. Item count src bucket: 0 dst bucket: 0
2021/12/15 02:54:12 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedGetOps' in metakv",
  "warnings": null
 }
}
2021/12/15 02:54:17 Waiting for app: TestAdvancedGetOps to get deployed
2021/12/15 02:54:22 Waiting for app: TestAdvancedGetOps to get deployed
2021/12/15 02:54:27 Waiting for app: TestAdvancedGetOps to get deployed
2021/12/15 02:54:32 Waiting for app: TestAdvancedGetOps to get deployed
2021/12/15 02:54:32 App: TestAdvancedGetOps got deployed
2021/12/15 02:54:32 Starting bucket ops to source bucket
2021/12/15 02:54:37 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/15 02:54:42 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/15 02:54:42 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/15 02:54:42 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 02:54:42 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/15 02:54:42 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 02:54:47 Waiting for app: TestAdvancedGetOps to get un-deployed
2021/12/15 02:54:52 Waiting for app: TestAdvancedGetOps to get un-deployed
2021/12/15 02:54:57 Waiting for app: TestAdvancedGetOps to get un-deployed
2021/12/15 02:55:03 Waiting for app: TestAdvancedGetOps to get un-deployed
2021/12/15 02:55:03 App: TestAdvancedGetOps got un-deployed
2021/12/15 02:55:03 No eventing-con process running
2021/12/15 02:55:03 Function: TestAdvancedGetOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedGetOps deleting in the background"
}
2021/12/15 02:55:04 Context: Bucket flush, response code: 200 dump: 
2021/12/15 02:55:14 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/15 02:55:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 02:55:16 Context: Bucket flush, response code: 200 dump: 
2021/12/15 02:55:26 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/15 02:55:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedGetOps (315.20s)
=== RUN   TestAdvancedInsertOps
2021/12/15 02:55:26 Deploying app: TestAdvancedInsertOps. Item count src bucket: 0 dst bucket: 0
2021/12/15 02:55:27 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedInsertOps' in metakv",
  "warnings": null
 }
}
2021/12/15 02:55:32 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/15 02:55:37 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/15 02:55:42 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/15 02:55:47 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/15 02:55:47 App: TestAdvancedInsertOps got deployed
2021/12/15 02:55:47 Starting bucket ops to source bucket
2021/12/15 02:55:53 Waiting for dst bucket item count to get to: 1024 curr count: 0
2021/12/15 02:55:58 Waiting for dst bucket item count to get to: 1024 curr count: 512
2021/12/15 02:56:03 Waiting for dst bucket item count to get to: 1024 curr count: 512
2021/12/15 02:56:03 src & dst bucket item count matched up. src bucket count: 1024 dst bucket count: 1024
2021/12/15 02:56:03 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 02:56:03 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/15 02:56:03 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 02:56:08 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/15 02:56:13 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/15 02:56:18 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/15 02:56:23 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/15 02:56:23 App: TestAdvancedInsertOps got un-deployed
2021/12/15 02:56:23 No eventing-con process running
2021/12/15 02:56:23 Function: TestAdvancedInsertOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedInsertOps deleting in the background"
}
2021/12/15 02:56:25 Context: Bucket flush, response code: 200 dump: 
2021/12/15 02:56:35 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/15 02:56:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 02:56:37 Context: Bucket flush, response code: 200 dump: 
2021/12/15 02:56:47 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/15 02:56:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 02:56:47 Testing insert operation on source bucket
2021/12/15 02:56:47 Deploying app: TestAdvancedInsertOps. Item count src bucket: 0 dst bucket: 0
2021/12/15 02:56:48 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedInsertOps' in metakv",
  "warnings": null
 }
}
2021/12/15 02:56:53 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/15 02:56:58 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/15 02:57:03 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/15 02:57:08 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/15 02:57:08 App: TestAdvancedInsertOps got deployed
2021/12/15 02:57:08 Starting bucket ops to source bucket
2021/12/15 02:57:14 Waiting for dst bucket item count to get to: 2048 curr count: 0
2021/12/15 02:57:19 Waiting for dst bucket item count to get to: 2048 curr count: 1024
2021/12/15 02:57:24 Waiting for dst bucket item count to get to: 2048 curr count: 1024
2021/12/15 02:57:24 src & dst bucket item count matched up. src bucket count: 2048 dst bucket count: 2048
2021/12/15 02:57:24 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 02:57:24 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/15 02:57:24 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 02:57:29 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/15 02:57:34 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/15 02:57:39 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/15 02:57:44 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/15 02:57:44 App: TestAdvancedInsertOps got un-deployed
2021/12/15 02:57:44 No eventing-con process running
2021/12/15 02:57:44 Function: TestAdvancedInsertOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedInsertOps deleting in the background"
}
2021/12/15 02:57:46 Context: Bucket flush, response code: 200 dump: 
2021/12/15 02:57:56 Waiting for dst bucket item count to get to: 0 curr count: 2048
2021/12/15 02:57:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 02:57:58 Context: Bucket flush, response code: 200 dump: 
2021/12/15 02:58:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedInsertOps (156.64s)
=== RUN   TestAdvancedUpsertOps
2021/12/15 02:58:03 Starting bucket ops to source bucket
2021/12/15 02:58:03 Deploying app: TestAdvancedUpsertOps. Item count src bucket: 0 dst bucket: 0
2021/12/15 02:58:05 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedUpsertOps' in metakv",
  "warnings": null
 }
}
2021/12/15 02:58:10 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/15 02:58:15 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/15 02:58:20 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/15 02:58:25 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/15 02:58:25 App: TestAdvancedUpsertOps got deployed
2021/12/15 02:58:25 Starting bucket ops to source bucket
2021/12/15 02:58:31 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/15 02:58:36 Waiting for dst bucket item count to get to: 0 curr count: 512
2021/12/15 02:58:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 02:58:36 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 02:58:36 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/15 02:58:36 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 02:58:41 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/15 02:58:46 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/15 02:58:51 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/15 02:58:56 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/15 02:58:56 App: TestAdvancedUpsertOps got un-deployed
2021/12/15 02:58:56 No eventing-con process running
2021/12/15 02:58:56 Function: TestAdvancedUpsertOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedUpsertOps deleting in the background"
}
2021/12/15 02:58:57 Context: Bucket flush, response code: 200 dump: 
2021/12/15 02:59:07 Waiting for dst bucket item count to get to: 0 curr count: 512
2021/12/15 02:59:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 02:59:09 Context: Bucket flush, response code: 200 dump: 
2021/12/15 02:59:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 02:59:14 Testing upsert operation on source bucket
2021/12/15 02:59:14 Deploying app: TestAdvancedUpsertOps. Item count src bucket: 0 dst bucket: 0
2021/12/15 02:59:15 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedUpsertOps' in metakv",
  "warnings": null
 }
}
2021/12/15 02:59:20 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/15 02:59:25 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/15 02:59:30 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/15 02:59:35 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/15 02:59:35 App: TestAdvancedUpsertOps got deployed
2021/12/15 02:59:35 Starting bucket ops to source bucket
2021/12/15 02:59:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 02:59:37 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 02:59:37 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/15 02:59:37 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 02:59:42 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/15 02:59:47 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/15 02:59:52 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/15 02:59:57 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/15 02:59:57 App: TestAdvancedUpsertOps got un-deployed
2021/12/15 02:59:57 No eventing-con process running
2021/12/15 02:59:57 Function: TestAdvancedUpsertOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedUpsertOps deleting in the background"
}
2021/12/15 02:59:58 Context: Bucket flush, response code: 200 dump: 
2021/12/15 03:00:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 03:00:06 Context: Bucket flush, response code: 200 dump: 
2021/12/15 03:00:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedUpsertOps (128.03s)
=== RUN   TestAdvancedReplaceOps
2021/12/15 03:00:11 Starting bucket ops to source bucket
2021/12/15 03:00:12 Deploying app: TestAdvancedReplaceOps. Item count src bucket: 0 dst bucket: 0
2021/12/15 03:00:13 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedReplaceOps' in metakv",
  "warnings": null
 }
}
2021/12/15 03:00:18 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/15 03:00:23 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/15 03:00:28 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/15 03:00:33 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/15 03:00:33 App: TestAdvancedReplaceOps got deployed
2021/12/15 03:00:33 Starting bucket ops to source bucket
2021/12/15 03:00:39 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/15 03:00:44 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/15 03:00:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 03:00:44 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 03:00:44 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/15 03:00:44 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 03:00:49 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/15 03:00:54 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/15 03:00:59 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/15 03:01:04 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/15 03:01:04 App: TestAdvancedReplaceOps got un-deployed
2021/12/15 03:01:04 No eventing-con process running
2021/12/15 03:01:04 Function: TestAdvancedReplaceOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedReplaceOps deleting in the background"
}
2021/12/15 03:01:06 Context: Bucket flush, response code: 200 dump: 
2021/12/15 03:01:16 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/15 03:01:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 03:01:18 Context: Bucket flush, response code: 200 dump: 
2021/12/15 03:01:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 03:01:23 Testing upsert operation on source bucket
2021/12/15 03:01:23 Deploying app: TestAdvancedReplaceOps. Item count src bucket: 0 dst bucket: 0
2021/12/15 03:01:24 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedReplaceOps' in metakv",
  "warnings": null
 }
}
2021/12/15 03:01:29 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/15 03:01:34 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/15 03:01:39 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/15 03:01:44 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/15 03:01:44 App: TestAdvancedReplaceOps got deployed
2021/12/15 03:01:44 Starting bucket ops to source bucket
2021/12/15 03:01:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 03:01:45 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 03:01:45 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/15 03:01:45 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 03:01:50 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/15 03:01:55 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/15 03:02:00 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/15 03:02:05 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/15 03:02:05 App: TestAdvancedReplaceOps got un-deployed
2021/12/15 03:02:05 No eventing-con process running
2021/12/15 03:02:05 Function: TestAdvancedReplaceOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedReplaceOps deleting in the background"
}
2021/12/15 03:02:07 Context: Bucket flush, response code: 200 dump: 
2021/12/15 03:02:17 Waiting for dst bucket item count to get to: 0 curr count: 59
2021/12/15 03:02:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 03:02:19 Context: Bucket flush, response code: 200 dump: 
2021/12/15 03:02:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedReplaceOps (133.22s)
=== RUN   TestAdvancedDeleteOps
2021/12/15 03:02:24 Starting bucket ops to source bucket
2021/12/15 03:02:24 Deploying app: TestAdvancedDeleteOps. Item count src bucket: 0 dst bucket: 0
2021/12/15 03:02:25 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedDeleteOps' in metakv",
  "warnings": null
 }
}
2021/12/15 03:02:30 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/15 03:02:35 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/15 03:02:40 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/15 03:02:45 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/15 03:02:45 App: TestAdvancedDeleteOps got deployed
2021/12/15 03:02:45 Starting bucket ops to source bucket
2021/12/15 03:02:50 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/15 03:02:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 03:02:50 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 03:02:50 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/15 03:02:51 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 03:02:56 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/15 03:03:01 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/15 03:03:06 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/15 03:03:11 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/15 03:03:11 App: TestAdvancedDeleteOps got un-deployed
2021/12/15 03:03:11 No eventing-con process running
2021/12/15 03:03:11 Function: TestAdvancedDeleteOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedDeleteOps deleting in the background"
}
2021/12/15 03:03:12 Context: Bucket flush, response code: 200 dump: 
2021/12/15 03:03:22 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/15 03:03:27 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/15 03:03:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 03:03:29 Context: Bucket flush, response code: 200 dump: 
2021/12/15 03:03:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 03:03:34 Testing delete operation on source bucket
2021/12/15 03:03:34 Deploying app: TestAdvancedDeleteOps. Item count src bucket: 0 dst bucket: 0
2021/12/15 03:03:35 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedDeleteOps' in metakv",
  "warnings": null
 }
}
2021/12/15 03:03:40 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/15 03:03:45 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/15 03:03:50 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/15 03:03:55 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/15 03:03:55 App: TestAdvancedDeleteOps got deployed
2021/12/15 03:03:55 Starting bucket ops to source bucket
2021/12/15 03:03:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 03:03:55 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 03:03:55 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/15 03:03:56 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 03:04:01 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/15 03:04:06 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/15 03:04:11 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/15 03:04:16 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/15 03:04:16 App: TestAdvancedDeleteOps got un-deployed
2021/12/15 03:04:16 No eventing-con process running
2021/12/15 03:04:16 Function: TestAdvancedDeleteOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedDeleteOps deleting in the background"
}
2021/12/15 03:04:17 Context: Bucket flush, response code: 200 dump: 
2021/12/15 03:04:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 03:04:25 Context: Bucket flush, response code: 200 dump: 
2021/12/15 03:04:30 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedDeleteOps (125.64s)
=== RUN   TestEnoentAdvancedGet
2021/12/15 03:04:30 Deploying app: TestEnoentAdvancedGet. Item count src bucket: 0 dst bucket: 0
2021/12/15 03:04:31 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedGet' in metakv",
  "warnings": null
 }
}
2021/12/15 03:04:36 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/15 03:04:41 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/15 03:04:46 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/15 03:04:51 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/15 03:04:51 App: TestEnoentAdvancedGet got deployed
2021/12/15 03:04:51 Starting bucket ops to source bucket
2021/12/15 03:04:56 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/15 03:05:01 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/15 03:05:01 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/15 03:05:01 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 03:05:01 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/15 03:05:01 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 03:05:06 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/15 03:05:11 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/15 03:05:16 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/15 03:05:21 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/15 03:05:21 App: TestEnoentAdvancedGet got un-deployed
2021/12/15 03:05:21 No eventing-con process running
2021/12/15 03:05:21 Function: TestEnoentAdvancedGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedGet deleting in the background"
}
2021/12/15 03:05:23 Context: Bucket flush, response code: 200 dump: 
2021/12/15 03:05:33 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/15 03:05:38 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/15 03:05:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 03:05:40 Context: Bucket flush, response code: 200 dump: 
2021/12/15 03:05:50 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/15 03:05:55 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/15 03:05:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 03:05:55 Testing get enoent operation on source bucket
2021/12/15 03:05:55 Deploying app: TestEnoentAdvancedGet. Item count src bucket: 0 dst bucket: 0
2021/12/15 03:05:56 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedGet' in metakv",
  "warnings": null
 }
}
2021/12/15 03:06:01 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/15 03:06:06 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/15 03:06:11 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/15 03:06:16 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/15 03:06:16 App: TestEnoentAdvancedGet got deployed
2021/12/15 03:06:16 Starting bucket ops to source bucket
2021/12/15 03:06:21 Waiting for dst bucket item count to get to: 200 curr count: 0
2021/12/15 03:06:21 src & dst bucket item count matched up. src bucket count: 200 dst bucket count: 200
2021/12/15 03:06:21 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 03:06:21 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/15 03:06:21 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 03:06:26 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/15 03:06:31 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/15 03:06:36 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/15 03:06:41 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/15 03:06:41 App: TestEnoentAdvancedGet got un-deployed
2021/12/15 03:06:41 No eventing-con process running
2021/12/15 03:06:41 Function: TestEnoentAdvancedGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedGet deleting in the background"
}
2021/12/15 03:06:43 Context: Bucket flush, response code: 200 dump: 
2021/12/15 03:06:53 Waiting for dst bucket item count to get to: 0 curr count: 200
2021/12/15 03:06:58 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/15 03:06:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 03:07:00 Context: Bucket flush, response code: 200 dump: 
2021/12/15 03:07:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEnoentAdvancedGet (155.22s)
=== RUN   TestEnoentAdvancedReplace
2021/12/15 03:07:05 Deploying app: TestEnoentAdvancedReplace. Item count src bucket: 0 dst bucket: 0
2021/12/15 03:07:06 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedReplace' in metakv",
  "warnings": null
 }
}
2021/12/15 03:07:11 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/15 03:07:16 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/15 03:07:21 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/15 03:07:26 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/15 03:07:26 App: TestEnoentAdvancedReplace got deployed
2021/12/15 03:07:26 Starting bucket ops to source bucket
2021/12/15 03:07:31 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/15 03:07:31 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/15 03:07:31 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 03:07:31 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/15 03:07:31 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 03:07:36 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/15 03:07:41 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/15 03:07:46 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/15 03:07:51 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/15 03:07:51 App: TestEnoentAdvancedReplace got un-deployed
2021/12/15 03:07:51 No eventing-con process running
2021/12/15 03:07:51 Function: TestEnoentAdvancedReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedReplace deleting in the background"
}
2021/12/15 03:07:53 Context: Bucket flush, response code: 200 dump: 
2021/12/15 03:08:03 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/15 03:08:08 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/15 03:08:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 03:08:10 Context: Bucket flush, response code: 200 dump: 
2021/12/15 03:08:20 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/15 03:08:25 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/15 03:08:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 03:08:25 Testing get enoent operation on source bucket
2021/12/15 03:08:25 Deploying app: TestEnoentAdvancedReplace. Item count src bucket: 0 dst bucket: 0
2021/12/15 03:08:26 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedReplace' in metakv",
  "warnings": null
 }
}
2021/12/15 03:08:31 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/15 03:08:36 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/15 03:08:41 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/15 03:08:46 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/15 03:08:46 App: TestEnoentAdvancedReplace got deployed
2021/12/15 03:08:46 Starting bucket ops to source bucket
2021/12/15 03:08:51 Waiting for dst bucket item count to get to: 200 curr count: 0
2021/12/15 03:08:51 src & dst bucket item count matched up. src bucket count: 200 dst bucket count: 200
2021/12/15 03:08:51 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 03:08:51 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/15 03:08:51 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 03:08:56 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/15 03:09:01 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/15 03:09:06 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/15 03:09:11 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/15 03:09:11 App: TestEnoentAdvancedReplace got un-deployed
2021/12/15 03:09:11 No eventing-con process running
2021/12/15 03:09:11 Function: TestEnoentAdvancedReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedReplace deleting in the background"
}
2021/12/15 03:09:13 Context: Bucket flush, response code: 200 dump: 
2021/12/15 03:09:23 Waiting for dst bucket item count to get to: 0 curr count: 200
2021/12/15 03:09:28 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/15 03:09:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 03:09:30 Context: Bucket flush, response code: 200 dump: 
2021/12/15 03:09:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEnoentAdvancedReplace (149.93s)
=== RUN   TestEnoentAdvancedDelete
2021/12/15 03:09:35 Deploying app: TestEnoentAdvancedDelete. Item count src bucket: 0 dst bucket: 0
2021/12/15 03:09:36 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedDelete' in metakv",
  "warnings": null
 }
}
2021/12/15 03:09:41 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/15 03:09:46 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/15 03:09:51 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/15 03:09:56 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/15 03:09:56 App: TestEnoentAdvancedDelete got deployed
2021/12/15 03:09:56 Starting bucket ops to source bucket
2021/12/15 03:10:01 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/15 03:10:01 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/15 03:10:01 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 03:10:01 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/15 03:10:01 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 03:10:06 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/15 03:10:11 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/15 03:10:16 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/15 03:10:21 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/15 03:10:21 App: TestEnoentAdvancedDelete got un-deployed
2021/12/15 03:10:21 No eventing-con process running
2021/12/15 03:10:21 Function: TestEnoentAdvancedDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedDelete deleting in the background"
}
2021/12/15 03:10:23 Context: Bucket flush, response code: 200 dump: 
2021/12/15 03:10:33 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/15 03:10:38 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/15 03:10:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 03:10:40 Context: Bucket flush, response code: 200 dump: 
2021/12/15 03:10:50 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/15 03:10:55 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/15 03:10:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 03:10:55 Testing get enoent operation on source bucket
2021/12/15 03:10:55 Deploying app: TestEnoentAdvancedDelete. Item count src bucket: 0 dst bucket: 0
2021/12/15 03:10:56 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedDelete' in metakv",
  "warnings": null
 }
}
2021/12/15 03:11:01 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/15 03:11:06 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/15 03:11:11 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/15 03:11:16 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/15 03:11:16 App: TestEnoentAdvancedDelete got deployed
2021/12/15 03:11:16 Starting bucket ops to source bucket
2021/12/15 03:11:21 Waiting for dst bucket item count to get to: 200 curr count: 0
2021/12/15 03:11:21 src & dst bucket item count matched up. src bucket count: 200 dst bucket count: 200
2021/12/15 03:11:21 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 03:11:21 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/15 03:11:21 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 03:11:26 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/15 03:11:31 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/15 03:11:36 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/15 03:11:41 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/15 03:11:41 App: TestEnoentAdvancedDelete got un-deployed
2021/12/15 03:11:41 No eventing-con process running
2021/12/15 03:11:41 Function: TestEnoentAdvancedDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedDelete deleting in the background"
}
2021/12/15 03:11:43 Context: Bucket flush, response code: 200 dump: 
2021/12/15 03:11:53 Waiting for dst bucket item count to get to: 0 curr count: 200
2021/12/15 03:11:58 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/15 03:11:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 03:12:00 Context: Bucket flush, response code: 200 dump: 
2021/12/15 03:12:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEnoentAdvancedDelete (149.98s)
=== RUN   TestAdvancedInsertKeyExist
2021/12/15 03:12:05 Deploying app: TestAdvancedInsertKeyExist. Item count src bucket: 0 dst bucket: 0
2021/12/15 03:12:06 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedInsertKeyExist' in metakv",
  "warnings": null
 }
}
2021/12/15 03:12:11 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/15 03:12:16 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/15 03:12:21 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/15 03:12:26 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/15 03:12:26 App: TestAdvancedInsertKeyExist got deployed
2021/12/15 03:12:26 Starting bucket ops to source bucket
2021/12/15 03:12:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 03:12:26 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 03:12:26 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/15 03:12:26 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 03:12:31 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/15 03:12:36 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/15 03:12:41 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/15 03:12:46 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/15 03:12:46 App: TestAdvancedInsertKeyExist got un-deployed
2021/12/15 03:12:46 No eventing-con process running
2021/12/15 03:12:46 Function: TestAdvancedInsertKeyExist delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedInsertKeyExist deleting in the background"
}
2021/12/15 03:12:48 Context: Bucket flush, response code: 200 dump: 
2021/12/15 03:12:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 03:12:55 Context: Bucket flush, response code: 200 dump: 
2021/12/15 03:13:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 03:13:00 Pumping document in the destination bucket
2021/12/15 03:13:00 Starting bucket ops to source bucket
2021/12/15 03:13:01 Deploying app: TestAdvancedInsertKeyExist. Item count src bucket: 0 dst bucket: 0
2021/12/15 03:13:02 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedInsertKeyExist' in metakv",
  "warnings": null
 }
}
2021/12/15 03:13:07 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/15 03:13:12 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/15 03:13:17 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/15 03:13:22 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/15 03:13:22 App: TestAdvancedInsertKeyExist got deployed
2021/12/15 03:13:22 Starting bucket ops to source bucket
2021/12/15 03:13:27 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/15 03:13:32 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/15 03:13:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 03:13:32 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 03:13:32 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/15 03:13:32 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 03:13:37 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/15 03:13:42 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/15 03:13:47 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/15 03:13:52 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/15 03:13:52 App: TestAdvancedInsertKeyExist got un-deployed
2021/12/15 03:13:52 No eventing-con process running
2021/12/15 03:13:52 Function: TestAdvancedInsertKeyExist delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedInsertKeyExist deleting in the background"
}
2021/12/15 03:13:54 Context: Bucket flush, response code: 200 dump: 
2021/12/15 03:14:04 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/15 03:14:09 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/15 03:14:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 03:14:10 Context: Bucket flush, response code: 200 dump: 
2021/12/15 03:14:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedInsertKeyExist (130.76s)
=== RUN   TestExipryGet
2021/12/15 03:14:16 Deploying app: TestExipryGet. Item count src bucket: 0 dst bucket: 0
2021/12/15 03:14:17 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryGet' in metakv",
  "warnings": null
 }
}
2021/12/15 03:14:22 Waiting for app: TestExipryGet to get deployed
2021/12/15 03:14:27 Waiting for app: TestExipryGet to get deployed
2021/12/15 03:14:32 Waiting for app: TestExipryGet to get deployed
2021/12/15 03:14:37 Waiting for app: TestExipryGet to get deployed
2021/12/15 03:14:37 App: TestExipryGet got deployed
2021/12/15 03:14:37 Starting bucket ops to source bucket
2021/12/15 03:15:22 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/15 03:15:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 03:15:22 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 03:15:22 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/15 03:15:22 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 03:15:27 Waiting for app: TestExipryGet to get un-deployed
2021/12/15 03:15:32 Waiting for app: TestExipryGet to get un-deployed
2021/12/15 03:15:37 Waiting for app: TestExipryGet to get un-deployed
2021/12/15 03:15:42 Waiting for app: TestExipryGet to get un-deployed
2021/12/15 03:15:42 App: TestExipryGet got un-deployed
2021/12/15 03:15:42 No eventing-con process running
2021/12/15 03:15:42 Function: TestExipryGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryGet deleting in the background"
}
2021/12/15 03:15:44 Context: Bucket flush, response code: 200 dump: 
2021/12/15 03:15:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 03:15:51 Context: Bucket flush, response code: 200 dump: 
2021/12/15 03:15:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 03:15:56 Pumping document in the destination bucket for expiry get
2021/12/15 03:15:56 Starting bucket ops to source bucket
2021/12/15 03:15:56 Deploying app: TestExipryGet. Item count src bucket: 0 dst bucket: 0
2021/12/15 03:15:58 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryGet' in metakv",
  "warnings": null
 }
}
2021/12/15 03:16:03 Waiting for app: TestExipryGet to get deployed
2021/12/15 03:16:08 Waiting for app: TestExipryGet to get deployed
2021/12/15 03:16:13 Waiting for app: TestExipryGet to get deployed
2021/12/15 03:16:18 Waiting for app: TestExipryGet to get deployed
2021/12/15 03:16:18 App: TestExipryGet got deployed
2021/12/15 03:16:18 Starting bucket ops to source bucket
2021/12/15 03:17:03 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/15 03:17:08 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/15 03:17:13 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/15 03:17:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 03:17:13 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 03:17:13 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/15 03:17:13 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 03:17:18 Waiting for app: TestExipryGet to get un-deployed
2021/12/15 03:17:23 Waiting for app: TestExipryGet to get un-deployed
2021/12/15 03:17:28 Waiting for app: TestExipryGet to get un-deployed
2021/12/15 03:17:33 Waiting for app: TestExipryGet to get un-deployed
2021/12/15 03:17:33 App: TestExipryGet got un-deployed
2021/12/15 03:17:33 No eventing-con process running
2021/12/15 03:17:33 Function: TestExipryGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryGet deleting in the background"
}
2021/12/15 03:17:35 Context: Bucket flush, response code: 200 dump: 
2021/12/15 03:17:45 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/15 03:17:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 03:17:46 Context: Bucket flush, response code: 200 dump: 
2021/12/15 03:17:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestExipryGet (215.95s)
=== RUN   TestExipryInsert
2021/12/15 03:17:51 Deploying app: TestExipryInsert. Item count src bucket: 0 dst bucket: 0
2021/12/15 03:17:53 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryInsert' in metakv",
  "warnings": null
 }
}
2021/12/15 03:17:58 Waiting for app: TestExipryInsert to get deployed
2021/12/15 03:18:03 Waiting for app: TestExipryInsert to get deployed
2021/12/15 03:18:08 Waiting for app: TestExipryInsert to get deployed
2021/12/15 03:18:13 Waiting for app: TestExipryInsert to get deployed
2021/12/15 03:18:13 App: TestExipryInsert got deployed
2021/12/15 03:18:13 Starting bucket ops to source bucket
2021/12/15 03:18:58 Waiting for dst bucket item count to get to: 0 curr count: 200
2021/12/15 03:19:03 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/15 03:19:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 03:19:03 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 03:19:03 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/15 03:19:03 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 03:19:08 Waiting for app: TestExipryInsert to get un-deployed
2021/12/15 03:19:13 Waiting for app: TestExipryInsert to get un-deployed
2021/12/15 03:19:18 Waiting for app: TestExipryInsert to get un-deployed
2021/12/15 03:19:23 Waiting for app: TestExipryInsert to get un-deployed
2021/12/15 03:19:23 App: TestExipryInsert got un-deployed
2021/12/15 03:19:23 No eventing-con process running
2021/12/15 03:19:23 Function: TestExipryInsert delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryInsert deleting in the background"
}
2021/12/15 03:19:25 Context: Bucket flush, response code: 200 dump: 
2021/12/15 03:19:30 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 03:19:32 Context: Bucket flush, response code: 200 dump: 
2021/12/15 03:19:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 03:19:37 Testing on the destination bucket for expiry insert
2021/12/15 03:19:37 Deploying app: TestExipryInsert. Item count src bucket: 0 dst bucket: 0
2021/12/15 03:19:38 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryInsert' in metakv",
  "warnings": null
 }
}
2021/12/15 03:19:43 Waiting for app: TestExipryInsert to get deployed
2021/12/15 03:19:48 Waiting for app: TestExipryInsert to get deployed
2021/12/15 03:19:53 Waiting for app: TestExipryInsert to get deployed
2021/12/15 03:19:58 Waiting for app: TestExipryInsert to get deployed
2021/12/15 03:19:58 App: TestExipryInsert got deployed
2021/12/15 03:19:58 Starting bucket ops to source bucket
2021/12/15 03:20:44 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/15 03:20:49 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/15 03:20:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 03:20:49 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 03:20:49 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/15 03:20:49 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 03:20:54 Waiting for app: TestExipryInsert to get un-deployed
2021/12/15 03:20:59 Waiting for app: TestExipryInsert to get un-deployed
2021/12/15 03:21:04 Waiting for app: TestExipryInsert to get un-deployed
2021/12/15 03:21:09 Waiting for app: TestExipryInsert to get un-deployed
2021/12/15 03:21:09 App: TestExipryInsert got un-deployed
2021/12/15 03:21:09 No eventing-con process running
2021/12/15 03:21:09 Function: TestExipryInsert delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryInsert deleting in the background"
}
2021/12/15 03:21:10 Context: Bucket flush, response code: 200 dump: 
2021/12/15 03:21:20 Waiting for dst bucket item count to get to: 0 curr count: 79
2021/12/15 03:21:25 Waiting for dst bucket item count to get to: 0 curr count: 79
2021/12/15 03:21:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 03:21:27 Context: Bucket flush, response code: 200 dump: 
2021/12/15 03:21:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestExipryInsert (220.79s)
=== RUN   TestExipryReplace
2021/12/15 03:21:32 Deploying app: TestExipryReplace. Item count src bucket: 0 dst bucket: 0
2021/12/15 03:21:33 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryReplace' in metakv",
  "warnings": null
 }
}
2021/12/15 03:21:38 Waiting for app: TestExipryReplace to get deployed
2021/12/15 03:21:43 Waiting for app: TestExipryReplace to get deployed
2021/12/15 03:21:48 Waiting for app: TestExipryReplace to get deployed
2021/12/15 03:21:53 Waiting for app: TestExipryReplace to get deployed
2021/12/15 03:21:53 App: TestExipryReplace got deployed
2021/12/15 03:21:53 Starting bucket ops to source bucket
2021/12/15 03:22:39 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/15 03:22:44 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/15 03:22:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 03:22:44 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 03:22:44 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/15 03:22:44 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 03:22:49 Waiting for app: TestExipryReplace to get un-deployed
2021/12/15 03:22:54 Waiting for app: TestExipryReplace to get un-deployed
2021/12/15 03:22:59 Waiting for app: TestExipryReplace to get un-deployed
2021/12/15 03:23:04 Waiting for app: TestExipryReplace to get un-deployed
2021/12/15 03:23:04 App: TestExipryReplace got un-deployed
2021/12/15 03:23:04 No eventing-con process running
2021/12/15 03:23:04 Function: TestExipryReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryReplace deleting in the background"
}
2021/12/15 03:23:06 Context: Bucket flush, response code: 200 dump: 
2021/12/15 03:23:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 03:23:13 Context: Bucket flush, response code: 200 dump: 
2021/12/15 03:23:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 03:23:18 Testing on the destination bucket for expiry insert
2021/12/15 03:23:18 Deploying app: TestExipryReplace. Item count src bucket: 0 dst bucket: 0
2021/12/15 03:23:19 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryReplace' in metakv",
  "warnings": null
 }
}
2021/12/15 03:23:24 Waiting for app: TestExipryReplace to get deployed
2021/12/15 03:23:29 Waiting for app: TestExipryReplace to get deployed
2021/12/15 03:23:34 Waiting for app: TestExipryReplace to get deployed
2021/12/15 03:23:39 Waiting for app: TestExipryReplace to get deployed
2021/12/15 03:23:39 App: TestExipryReplace got deployed
2021/12/15 03:23:39 Starting bucket ops to source bucket
2021/12/15 03:24:24 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/15 03:24:29 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/15 03:24:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 03:24:29 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 03:24:29 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/15 03:24:29 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 03:24:34 Waiting for app: TestExipryReplace to get un-deployed
2021/12/15 03:24:39 Waiting for app: TestExipryReplace to get un-deployed
2021/12/15 03:24:44 Waiting for app: TestExipryReplace to get un-deployed
2021/12/15 03:24:49 Waiting for app: TestExipryReplace to get un-deployed
2021/12/15 03:24:49 App: TestExipryReplace got un-deployed
2021/12/15 03:24:49 No eventing-con process running
2021/12/15 03:24:49 Function: TestExipryReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryReplace deleting in the background"
}
2021/12/15 03:24:51 Context: Bucket flush, response code: 200 dump: 
2021/12/15 03:25:01 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/15 03:25:06 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/15 03:25:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 03:25:08 Context: Bucket flush, response code: 200 dump: 
2021/12/15 03:25:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestExipryReplace (220.59s)
=== RUN   TestExipryUpsert
2021/12/15 03:25:13 Deploying app: TestExipryUpsert. Item count src bucket: 0 dst bucket: 0
2021/12/15 03:25:14 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryUpsert' in metakv",
  "warnings": null
 }
}
2021/12/15 03:25:19 Waiting for app: TestExipryUpsert to get deployed
2021/12/15 03:25:24 Waiting for app: TestExipryUpsert to get deployed
2021/12/15 03:25:29 Waiting for app: TestExipryUpsert to get deployed
2021/12/15 03:25:34 Waiting for app: TestExipryUpsert to get deployed
2021/12/15 03:25:34 App: TestExipryUpsert got deployed
2021/12/15 03:25:34 Starting bucket ops to source bucket
2021/12/15 03:26:19 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/15 03:26:24 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/15 03:26:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 03:26:24 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 03:26:24 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/15 03:26:24 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 03:26:29 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/15 03:26:34 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/15 03:26:39 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/15 03:26:44 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/15 03:26:44 App: TestExipryUpsert got un-deployed
2021/12/15 03:26:44 No eventing-con process running
2021/12/15 03:26:44 Function: TestExipryUpsert delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryUpsert deleting in the background"
}
2021/12/15 03:26:46 Context: Bucket flush, response code: 200 dump: 
2021/12/15 03:26:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 03:26:53 Context: Bucket flush, response code: 200 dump: 
2021/12/15 03:26:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 03:26:58 Testing on the destination bucket for expiry insert
2021/12/15 03:26:58 Deploying app: TestExipryUpsert. Item count src bucket: 0 dst bucket: 0
2021/12/15 03:27:00 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryUpsert' in metakv",
  "warnings": null
 }
}
2021/12/15 03:27:05 Waiting for app: TestExipryUpsert to get deployed
2021/12/15 03:27:10 Waiting for app: TestExipryUpsert to get deployed
2021/12/15 03:27:15 Waiting for app: TestExipryUpsert to get deployed
2021/12/15 03:27:20 Waiting for app: TestExipryUpsert to get deployed
2021/12/15 03:27:20 App: TestExipryUpsert got deployed
2021/12/15 03:27:20 Starting bucket ops to source bucket
2021/12/15 03:28:05 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/15 03:28:10 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/15 03:28:10 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 03:28:10 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 03:28:10 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/15 03:28:10 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 03:28:15 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/15 03:28:20 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/15 03:28:25 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/15 03:28:30 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/15 03:28:30 App: TestExipryUpsert got un-deployed
2021/12/15 03:28:30 No eventing-con process running
2021/12/15 03:28:30 Function: TestExipryUpsert delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryUpsert deleting in the background"
}
2021/12/15 03:28:32 Context: Bucket flush, response code: 200 dump: 
2021/12/15 03:28:42 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/15 03:28:47 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/15 03:28:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 03:28:49 Context: Bucket flush, response code: 200 dump: 
2021/12/15 03:28:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestExipryUpsert (220.74s)
=== RUN   TestCasReplace
2021/12/15 03:28:54 Deploying app: TestCasReplace. Item count src bucket: 0 dst bucket: 0
2021/12/15 03:28:55 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCasReplace' in metakv",
  "warnings": null
 }
}
2021/12/15 03:29:00 Waiting for app: TestCasReplace to get deployed
2021/12/15 03:29:05 Waiting for app: TestCasReplace to get deployed
2021/12/15 03:29:10 Waiting for app: TestCasReplace to get deployed
2021/12/15 03:29:15 Waiting for app: TestCasReplace to get deployed
2021/12/15 03:29:15 App: TestCasReplace got deployed
2021/12/15 03:29:15 Starting bucket ops to source bucket
2021/12/15 03:29:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 03:29:15 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 03:29:15 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/15 03:29:15 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 03:29:20 Waiting for app: TestCasReplace to get un-deployed
2021/12/15 03:29:25 Waiting for app: TestCasReplace to get un-deployed
2021/12/15 03:29:30 Waiting for app: TestCasReplace to get un-deployed
2021/12/15 03:29:35 Waiting for app: TestCasReplace to get un-deployed
2021/12/15 03:29:35 App: TestCasReplace got un-deployed
2021/12/15 03:29:35 No eventing-con process running
2021/12/15 03:29:35 Function: TestCasReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCasReplace deleting in the background"
}
2021/12/15 03:29:37 Context: Bucket flush, response code: 200 dump: 
2021/12/15 03:29:47 Waiting for dst bucket item count to get to: 0 curr count: 7
2021/12/15 03:29:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 03:29:49 Context: Bucket flush, response code: 200 dump: 
2021/12/15 03:29:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 03:29:54 Testing on the destination bucket for upsert with cas
2021/12/15 03:29:54 Starting bucket ops to source bucket
2021/12/15 03:29:54 Deploying app: TestCasReplace. Item count src bucket: 0 dst bucket: 0
2021/12/15 03:29:55 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCasReplace' in metakv",
  "warnings": null
 }
}
2021/12/15 03:30:00 Waiting for app: TestCasReplace to get deployed
2021/12/15 03:30:05 Waiting for app: TestCasReplace to get deployed
2021/12/15 03:30:10 Waiting for app: TestCasReplace to get deployed
2021/12/15 03:30:15 Waiting for app: TestCasReplace to get deployed
2021/12/15 03:30:15 App: TestCasReplace got deployed
2021/12/15 03:30:15 Starting bucket ops to source bucket
2021/12/15 03:30:20 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/15 03:30:25 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/15 03:30:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 03:30:25 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 03:30:25 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/15 03:30:25 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 03:30:30 Waiting for app: TestCasReplace to get un-deployed
2021/12/15 03:30:35 Waiting for app: TestCasReplace to get un-deployed
2021/12/15 03:30:40 Waiting for app: TestCasReplace to get un-deployed
2021/12/15 03:30:45 Waiting for app: TestCasReplace to get un-deployed
2021/12/15 03:30:45 App: TestCasReplace got un-deployed
2021/12/15 03:30:45 No eventing-con process running
2021/12/15 03:30:45 Function: TestCasReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCasReplace deleting in the background"
}
2021/12/15 03:30:47 Context: Bucket flush, response code: 200 dump: 
2021/12/15 03:30:57 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/15 03:30:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 03:30:59 Context: Bucket flush, response code: 200 dump: 
2021/12/15 03:31:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCasReplace (130.42s)
=== RUN   TestCasDelete
2021/12/15 03:31:04 Deploying app: TestCasDelete. Item count src bucket: 0 dst bucket: 0
2021/12/15 03:31:05 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCasDelete' in metakv",
  "warnings": null
 }
}
2021/12/15 03:31:10 Waiting for app: TestCasDelete to get deployed
2021/12/15 03:31:15 Waiting for app: TestCasDelete to get deployed
2021/12/15 03:31:20 Waiting for app: TestCasDelete to get deployed
2021/12/15 03:31:25 Waiting for app: TestCasDelete to get deployed
2021/12/15 03:31:25 App: TestCasDelete got deployed
2021/12/15 03:31:25 Starting bucket ops to source bucket
2021/12/15 03:31:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 03:31:25 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 03:31:25 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/15 03:31:26 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 03:31:31 Waiting for app: TestCasDelete to get un-deployed
2021/12/15 03:31:36 Waiting for app: TestCasDelete to get un-deployed
2021/12/15 03:31:41 Waiting for app: TestCasDelete to get un-deployed
2021/12/15 03:31:46 Waiting for app: TestCasDelete to get un-deployed
2021/12/15 03:31:46 App: TestCasDelete got un-deployed
2021/12/15 03:31:46 No eventing-con process running
2021/12/15 03:31:46 Function: TestCasDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCasDelete deleting in the background"
}
2021/12/15 03:31:47 Context: Bucket flush, response code: 200 dump: 
2021/12/15 03:31:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 03:31:55 Context: Bucket flush, response code: 200 dump: 
2021/12/15 03:32:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 03:32:00 Testing on the destination bucket for delete with cas
2021/12/15 03:32:00 Starting bucket ops to source bucket
2021/12/15 03:32:00 Deploying app: TestCasDelete. Item count src bucket: 0 dst bucket: 0
2021/12/15 03:32:01 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCasDelete' in metakv",
  "warnings": null
 }
}
2021/12/15 03:32:06 Waiting for app: TestCasDelete to get deployed
2021/12/15 03:32:11 Waiting for app: TestCasDelete to get deployed
2021/12/15 03:32:16 Waiting for app: TestCasDelete to get deployed
2021/12/15 03:32:21 Waiting for app: TestCasDelete to get deployed
2021/12/15 03:32:21 App: TestCasDelete got deployed
2021/12/15 03:32:21 Starting bucket ops to source bucket
2021/12/15 03:32:26 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/15 03:32:31 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/15 03:32:31 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 03:32:31 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 03:32:31 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/15 03:32:31 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 03:32:36 Waiting for app: TestCasDelete to get un-deployed
2021/12/15 03:32:41 Waiting for app: TestCasDelete to get un-deployed
2021/12/15 03:32:46 Waiting for app: TestCasDelete to get un-deployed
2021/12/15 03:32:51 Waiting for app: TestCasDelete to get un-deployed
2021/12/15 03:32:51 App: TestCasDelete got un-deployed
2021/12/15 03:32:51 No eventing-con process running
2021/12/15 03:32:51 Function: TestCasDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCasDelete deleting in the background"
}
2021/12/15 03:32:53 Context: Bucket flush, response code: 200 dump: 
2021/12/15 03:33:03 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/15 03:33:08 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/15 03:33:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 03:33:10 Context: Bucket flush, response code: 200 dump: 
2021/12/15 03:33:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCasDelete (130.84s)
=== RUN   TestCountersIncrement
2021/12/15 03:33:15 Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/15 03:33:21 Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/15 03:33:21 Removing node(s): [] from the cluster
2021/12/15 03:33:26 Rebalance progress: 100
2021/12/15 03:33:26 Deploying app: TestCountersIncrement. Item count src bucket: 0 dst bucket: 0
2021/12/15 03:33:27 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCountersIncrement' in metakv",
  "warnings": null
 }
}
2021/12/15 03:33:32 Waiting for app: TestCountersIncrement to get deployed
2021/12/15 03:33:37 Waiting for app: TestCountersIncrement to get deployed
2021/12/15 03:33:42 Waiting for app: TestCountersIncrement to get deployed
2021/12/15 03:33:47 Waiting for app: TestCountersIncrement to get deployed
2021/12/15 03:33:47 App: TestCountersIncrement got deployed
2021/12/15 03:33:47 Starting bucket ops to source bucket
2021/12/15 03:33:54 Waiting for dst bucket item count to get to: 1024 curr count: 0
2021/12/15 03:33:59 Waiting for dst bucket item count to get to: 1024 curr count: 512
2021/12/15 03:34:04 Waiting for dst bucket item count to get to: 1024 curr count: 512
2021/12/15 03:34:04 src & dst bucket item count matched up. src bucket count: 1024 dst bucket count: 1024
2021/12/15 03:34:04 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 03:34:04 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 03:34:09 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/15 03:34:14 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/15 03:34:19 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/15 03:34:24 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/15 03:34:24 App: TestCountersIncrement got un-deployed
2021/12/15 03:34:24 No eventing-con process running
2021/12/15 03:34:24 Function: TestCountersIncrement delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCountersIncrement deleting in the background"
}
2021/12/15 03:34:26 Context: Bucket flush, response code: 200 dump: 
2021/12/15 03:34:36 Waiting for dst bucket item count to get to: 0 curr count: 1025
2021/12/15 03:34:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 03:34:37 Context: Bucket flush, response code: 200 dump: 
2021/12/15 03:34:47 Waiting for dst bucket item count to get to: 0 curr count: 512
2021/12/15 03:34:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 03:34:47 Deploying app: TestCountersIncrement. Item count src bucket: 0 dst bucket: 0
2021/12/15 03:34:49 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCountersIncrement' in metakv",
  "warnings": null
 }
}
2021/12/15 03:34:54 Waiting for app: TestCountersIncrement to get deployed
2021/12/15 03:34:59 Waiting for app: TestCountersIncrement to get deployed
2021/12/15 03:35:04 Waiting for app: TestCountersIncrement to get deployed
2021/12/15 03:35:09 Waiting for app: TestCountersIncrement to get deployed
2021/12/15 03:35:09 App: TestCountersIncrement got deployed
2021/12/15 03:35:09 Starting bucket ops to source bucket
2021/12/15 03:35:15 Waiting for dst bucket item count to get to: 2048 curr count: 0
2021/12/15 03:35:20 Waiting for dst bucket item count to get to: 2048 curr count: 525
2021/12/15 03:35:25 Waiting for dst bucket item count to get to: 2048 curr count: 1549
2021/12/15 03:35:25 src & dst bucket item count matched up. src bucket count: 2048 dst bucket count: 2048
2021/12/15 03:35:25 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 03:35:25 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 03:35:30 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/15 03:35:35 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/15 03:35:40 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/15 03:35:45 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/15 03:35:45 App: TestCountersIncrement got un-deployed
2021/12/15 03:35:45 No eventing-con process running
2021/12/15 03:35:45 Function: TestCountersIncrement delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCountersIncrement deleting in the background"
}
2021/12/15 03:35:47 Context: Bucket flush, response code: 200 dump: 
2021/12/15 03:35:57 Waiting for dst bucket item count to get to: 0 curr count: 2048
2021/12/15 03:35:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 03:35:59 Context: Bucket flush, response code: 200 dump: 
2021/12/15 03:36:09 Waiting for dst bucket item count to get to: 0 curr count: 1
2021/12/15 03:36:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 03:36:09 Removing node(s): [http://127.0.0.1:19002 http://127.0.0.1:19003] from the cluster
2021/12/15 03:36:14 Rebalance progress: 100
--- PASS: TestCountersIncrement (178.84s)
=== RUN   TestCountersDecrement
2021/12/15 03:36:14 Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/15 03:36:14 Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/15 03:36:14 Removing node(s): [] from the cluster
2021/12/15 03:36:19 Rebalance progress: 100
2021/12/15 03:36:20 Deploying app: TestCountersDecrement. Item count src bucket: 0 dst bucket: 0
2021/12/15 03:36:21 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCountersDecrement' in metakv",
  "warnings": null
 }
}
2021/12/15 03:36:26 Waiting for app: TestCountersDecrement to get deployed
2021/12/15 03:36:31 Waiting for app: TestCountersDecrement to get deployed
2021/12/15 03:36:36 Waiting for app: TestCountersDecrement to get deployed
2021/12/15 03:36:41 Waiting for app: TestCountersDecrement to get deployed
2021/12/15 03:36:41 App: TestCountersDecrement got deployed
2021/12/15 03:36:41 Starting bucket ops to source bucket
2021/12/15 03:36:47 Waiting for dst bucket item count to get to: 1024 curr count: 0
2021/12/15 03:36:52 Waiting for dst bucket item count to get to: 1024 curr count: 0
2021/12/15 03:36:57 Waiting for dst bucket item count to get to: 1024 curr count: 512
2021/12/15 03:36:57 src & dst bucket item count matched up. src bucket count: 1024 dst bucket count: 1024
2021/12/15 03:36:57 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 03:36:57 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 03:37:02 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/15 03:37:07 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/15 03:37:12 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/15 03:37:17 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/15 03:37:17 App: TestCountersDecrement got un-deployed
2021/12/15 03:37:17 No eventing-con process running
2021/12/15 03:37:17 Function: TestCountersDecrement delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCountersDecrement deleting in the background"
}
2021/12/15 03:37:19 Context: Bucket flush, response code: 200 dump: 
2021/12/15 03:37:29 Waiting for dst bucket item count to get to: 0 curr count: 1025
2021/12/15 03:37:34 Waiting for dst bucket item count to get to: 0 curr count: 512
2021/12/15 03:37:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 03:37:36 Context: Bucket flush, response code: 200 dump: 
2021/12/15 03:37:46 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/15 03:37:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 03:37:46 Deploying app: TestCountersDecrement. Item count src bucket: 0 dst bucket: 0
2021/12/15 03:37:47 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCountersDecrement' in metakv",
  "warnings": null
 }
}
2021/12/15 03:37:52 Waiting for app: TestCountersDecrement to get deployed
2021/12/15 03:37:57 Waiting for app: TestCountersDecrement to get deployed
2021/12/15 03:38:02 Waiting for app: TestCountersDecrement to get deployed
2021/12/15 03:38:07 Waiting for app: TestCountersDecrement to get deployed
2021/12/15 03:38:07 App: TestCountersDecrement got deployed
2021/12/15 03:38:07 Starting bucket ops to source bucket
2021/12/15 03:38:13 Waiting for dst bucket item count to get to: 2048 curr count: 0
2021/12/15 03:38:18 Waiting for dst bucket item count to get to: 2048 curr count: 1202
2021/12/15 03:38:23 Waiting for dst bucket item count to get to: 2048 curr count: 1202
2021/12/15 03:38:23 src & dst bucket item count matched up. src bucket count: 2048 dst bucket count: 2048
2021/12/15 03:38:23 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 03:38:23 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 03:38:28 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/15 03:38:33 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/15 03:38:38 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/15 03:38:43 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/15 03:38:43 App: TestCountersDecrement got un-deployed
2021/12/15 03:38:43 No eventing-con process running
2021/12/15 03:38:43 Function: TestCountersDecrement delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCountersDecrement deleting in the background"
}
2021/12/15 03:38:45 Context: Bucket flush, response code: 200 dump: 
2021/12/15 03:38:55 Waiting for dst bucket item count to get to: 0 curr count: 2048
2021/12/15 03:38:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 03:38:57 Context: Bucket flush, response code: 200 dump: 
2021/12/15 03:39:07 Waiting for dst bucket item count to get to: 0 curr count: 1
2021/12/15 03:39:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 03:39:07 Removing node(s): [http://127.0.0.1:19002 http://127.0.0.1:19003] from the cluster
2021/12/15 03:39:12 Rebalance progress: 100
--- PASS: TestCountersDecrement (178.27s)
=== RUN   TestCacheGet
2021/12/15 03:39:12 Deploying app: TestCacheGet. Item count src bucket: 0 dst bucket: 0
2021/12/15 03:39:13 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCacheGet' in metakv",
  "warnings": null
 }
}
2021/12/15 03:39:18 Waiting for app: TestCacheGet to get deployed
2021/12/15 03:39:23 Waiting for app: TestCacheGet to get deployed
2021/12/15 03:39:28 Waiting for app: TestCacheGet to get deployed
2021/12/15 03:39:33 Waiting for app: TestCacheGet to get deployed
2021/12/15 03:39:33 App: TestCacheGet got deployed
2021/12/15 03:39:33 Starting bucket ops to source bucket
2021/12/15 03:39:38 Waiting for dst bucket item count to get to: 20 curr count: 0
2021/12/15 03:39:43 Waiting for dst bucket item count to get to: 20 curr count: 0
2021/12/15 03:39:48 Waiting for dst bucket item count to get to: 20 curr count: 0
2021/12/15 03:39:53 Waiting for dst bucket item count to get to: 20 curr count: 8
2021/12/15 03:39:58 Waiting for dst bucket item count to get to: 20 curr count: 16
2021/12/15 03:40:03 Waiting for dst bucket item count to get to: 20 curr count: 18
2021/12/15 03:40:03 src & dst bucket item count matched up. src bucket count: 20 dst bucket count: 20
2021/12/15 03:40:03 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 03:40:03 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 03:40:08 Waiting for app: TestCacheGet to get un-deployed
2021/12/15 03:40:14 Waiting for app: TestCacheGet to get un-deployed
2021/12/15 03:40:19 Waiting for app: TestCacheGet to get un-deployed
2021/12/15 03:40:24 Waiting for app: TestCacheGet to get un-deployed
2021/12/15 03:40:24 App: TestCacheGet got un-deployed
2021/12/15 03:40:24 No eventing-con process running
2021/12/15 03:40:24 Function: TestCacheGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCacheGet deleting in the background"
}
2021/12/15 03:40:25 Context: Bucket flush, response code: 200 dump: 
2021/12/15 03:40:35 Waiting for dst bucket item count to get to: 0 curr count: 20
2021/12/15 03:40:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 03:40:37 Context: Bucket flush, response code: 200 dump: 
2021/12/15 03:40:47 Waiting for dst bucket item count to get to: 0 curr count: 20
2021/12/15 03:40:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCacheGet (95.34s)
=== RUN   TestCacheRYOW
2021/12/15 03:40:47 Deploying app: TestCacheRYOW. Item count src bucket: 0 dst bucket: 0
2021/12/15 03:40:48 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCacheRYOW' in metakv",
  "warnings": null
 }
}
2021/12/15 03:40:53 Waiting for app: TestCacheRYOW to get deployed
2021/12/15 03:40:58 Waiting for app: TestCacheRYOW to get deployed
2021/12/15 03:41:03 Waiting for app: TestCacheRYOW to get deployed
2021/12/15 03:41:09 Waiting for app: TestCacheRYOW to get deployed
2021/12/15 03:41:09 App: TestCacheRYOW got deployed
2021/12/15 03:41:09 Starting bucket ops to source bucket
2021/12/15 03:41:14 Waiting for dst bucket item count to get to: 50 curr count: 0
2021/12/15 03:41:19 Waiting for dst bucket item count to get to: 50 curr count: 25
2021/12/15 03:41:19 src & dst bucket item count matched up. src bucket count: 50 dst bucket count: 50
2021/12/15 03:41:19 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 03:41:19 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 03:41:24 Waiting for app: TestCacheRYOW to get un-deployed
2021/12/15 03:41:29 Waiting for app: TestCacheRYOW to get un-deployed
2021/12/15 03:41:34 Waiting for app: TestCacheRYOW to get un-deployed
2021/12/15 03:41:39 Waiting for app: TestCacheRYOW to get un-deployed
2021/12/15 03:41:39 App: TestCacheRYOW got un-deployed
2021/12/15 03:41:39 No eventing-con process running
2021/12/15 03:41:39 Function: TestCacheRYOW delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCacheRYOW deleting in the background"
}
2021/12/15 03:41:41 Context: Bucket flush, response code: 200 dump: 
2021/12/15 03:41:51 Waiting for dst bucket item count to get to: 0 curr count: 42
2021/12/15 03:41:56 Waiting for dst bucket item count to get to: 0 curr count: 42
2021/12/15 03:41:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 03:41:57 Context: Bucket flush, response code: 200 dump: 
2021/12/15 03:42:07 Waiting for dst bucket item count to get to: 0 curr count: 25
2021/12/15 03:42:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCacheRYOW (80.19s)
=== RUN   TestCacheExpiry
2021/12/15 03:42:07 Deploying app: TestCacheExpiry. Item count src bucket: 0 dst bucket: 0
2021/12/15 03:42:09 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCacheExpiry' in metakv",
  "warnings": null
 }
}
2021/12/15 03:42:14 Waiting for app: TestCacheExpiry to get deployed
2021/12/15 03:42:19 Waiting for app: TestCacheExpiry to get deployed
2021/12/15 03:42:24 Waiting for app: TestCacheExpiry to get deployed
2021/12/15 03:42:29 Waiting for app: TestCacheExpiry to get deployed
2021/12/15 03:42:29 App: TestCacheExpiry got deployed
2021/12/15 03:42:29 Starting bucket ops to source bucket
2021/12/15 03:42:34 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/15 03:42:39 Waiting for dst bucket item count to get to: 100 curr count: 3
2021/12/15 03:42:44 Waiting for dst bucket item count to get to: 100 curr count: 42
2021/12/15 03:42:49 Waiting for dst bucket item count to get to: 100 curr count: 86
2021/12/15 03:42:54 Waiting for dst bucket item count to get to: 100 curr count: 97
2021/12/15 03:42:54 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/15 03:42:54 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 03:42:54 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 03:42:59 Waiting for app: TestCacheExpiry to get un-deployed
2021/12/15 03:43:04 Waiting for app: TestCacheExpiry to get un-deployed
2021/12/15 03:43:09 Waiting for app: TestCacheExpiry to get un-deployed
2021/12/15 03:43:14 Waiting for app: TestCacheExpiry to get un-deployed
2021/12/15 03:43:14 App: TestCacheExpiry got un-deployed
2021/12/15 03:43:14 No eventing-con process running
2021/12/15 03:43:14 Function: TestCacheExpiry delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCacheExpiry deleting in the background"
}
2021/12/15 03:43:16 Context: Bucket flush, response code: 200 dump: 
2021/12/15 03:43:26 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/15 03:43:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 03:43:28 Context: Bucket flush, response code: 200 dump: 
2021/12/15 03:43:38 Waiting for dst bucket item count to get to: 0 curr count: 52
2021/12/15 03:43:43 Waiting for dst bucket item count to get to: 0 curr count: 2
2021/12/15 03:43:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCacheExpiry (95.23s)
=== RUN   TestCacheExpiryCustom
2021/12/15 03:43:43 Deploying app: TestCacheExpiryCustom. Item count src bucket: 0 dst bucket: 0
2021/12/15 03:43:44 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCacheExpiryCustom' in metakv",
  "warnings": null
 }
}
2021/12/15 03:43:49 Waiting for app: TestCacheExpiryCustom to get deployed
2021/12/15 03:43:54 Waiting for app: TestCacheExpiryCustom to get deployed
2021/12/15 03:43:59 Waiting for app: TestCacheExpiryCustom to get deployed
2021/12/15 03:44:04 Waiting for app: TestCacheExpiryCustom to get deployed
2021/12/15 03:44:04 App: TestCacheExpiryCustom got deployed
2021/12/15 03:44:04 Starting bucket ops to source bucket
2021/12/15 03:44:09 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/15 03:44:14 Waiting for dst bucket item count to get to: 100 curr count: 14
2021/12/15 03:44:19 Waiting for dst bucket item count to get to: 100 curr count: 37
2021/12/15 03:44:24 Waiting for dst bucket item count to get to: 100 curr count: 72
2021/12/15 03:44:29 Waiting for dst bucket item count to get to: 100 curr count: 99
2021/12/15 03:44:29 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/15 03:44:29 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 03:44:29 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 03:44:34 Waiting for app: TestCacheExpiryCustom to get un-deployed
2021/12/15 03:44:39 Waiting for app: TestCacheExpiryCustom to get un-deployed
2021/12/15 03:44:44 Waiting for app: TestCacheExpiryCustom to get un-deployed
2021/12/15 03:44:49 Waiting for app: TestCacheExpiryCustom to get un-deployed
2021/12/15 03:44:49 App: TestCacheExpiryCustom got un-deployed
2021/12/15 03:44:49 No eventing-con process running
2021/12/15 03:44:49 Function: TestCacheExpiryCustom delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCacheExpiryCustom deleting in the background"
}
2021/12/15 03:44:51 Context: Bucket flush, response code: 200 dump: 
2021/12/15 03:45:01 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/15 03:45:06 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/15 03:45:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 03:45:08 Context: Bucket flush, response code: 200 dump: 
2021/12/15 03:45:18 Waiting for dst bucket item count to get to: 0 curr count: 63
2021/12/15 03:45:23 Waiting for dst bucket item count to get to: 0 curr count: 13
2021/12/15 03:45:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCacheExpiryCustom (100.19s)
=== RUN   TestCacheOverflowCustom
2021/12/15 03:45:23 Deploying app: TestCacheOverflowCustom. Item count src bucket: 0 dst bucket: 0
2021/12/15 03:45:24 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCacheOverflowCustom' in metakv",
  "warnings": null
 }
}
2021/12/15 03:45:29 Waiting for app: TestCacheOverflowCustom to get deployed
2021/12/15 03:45:34 Waiting for app: TestCacheOverflowCustom to get deployed
2021/12/15 03:45:39 Waiting for app: TestCacheOverflowCustom to get deployed
2021/12/15 03:45:44 Waiting for app: TestCacheOverflowCustom to get deployed
2021/12/15 03:45:44 App: TestCacheOverflowCustom got deployed
2021/12/15 03:45:44 Starting bucket ops to source bucket
2021/12/15 03:45:49 Waiting for dst bucket item count to get to: 35 curr count: 0
2021/12/15 03:45:54 Waiting for dst bucket item count to get to: 35 curr count: 9
2021/12/15 03:45:59 Waiting for dst bucket item count to get to: 35 curr count: 18
2021/12/15 03:46:04 Waiting for dst bucket item count to get to: 35 curr count: 23
2021/12/15 03:46:09 Waiting for dst bucket item count to get to: 35 curr count: 27
2021/12/15 03:46:14 Waiting for dst bucket item count to get to: 35 curr count: 30
2021/12/15 03:46:14 src & dst bucket item count matched up. src bucket count: 35 dst bucket count: 35
2021/12/15 03:46:14 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 03:46:14 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 03:46:19 Waiting for app: TestCacheOverflowCustom to get un-deployed
2021/12/15 03:46:24 Waiting for app: TestCacheOverflowCustom to get un-deployed
2021/12/15 03:46:29 Waiting for app: TestCacheOverflowCustom to get un-deployed
2021/12/15 03:46:34 Waiting for app: TestCacheOverflowCustom to get un-deployed
2021/12/15 03:46:34 App: TestCacheOverflowCustom got un-deployed
2021/12/15 03:46:34 No eventing-con process running
2021/12/15 03:46:34 Function: TestCacheOverflowCustom delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCacheOverflowCustom deleting in the background"
}
2021/12/15 03:46:36 Context: Bucket flush, response code: 200 dump: 
2021/12/15 03:46:46 Waiting for dst bucket item count to get to: 0 curr count: 5
2021/12/15 03:46:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 03:46:48 Context: Bucket flush, response code: 200 dump: 
2021/12/15 03:46:58 Waiting for dst bucket item count to get to: 0 curr count: 26
2021/12/15 03:47:03 Waiting for dst bucket item count to get to: 0 curr count: 8
2021/12/15 03:47:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCacheOverflowCustom (100.24s)
=== RUN   TestStrictMode
2021/12/15 03:47:03 Deploying app: TestStrictMode. Item count src bucket: 0 dst bucket: 0
2021/12/15 03:47:04 Context: Post to main store, response code: 422 dump: {
 "name": "ERR_HANDLER_COMPILATION",
 "code": 27,
 "description": "Function compilation failed",
 "attributes": null,
 "runtime_info": {
  "code": 27,
  "info": {
   "area": "handlerCode",
   "column_number": 63,
   "compile_success": false,
   "description": "        destination[meta.id] = 00000000000000000000000000000101;\n                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nSyntaxError: Octal literals are not allowed in strict mode.\n",
   "index": 132,
   "language": "JavaScript",
   "line_number": 4
  }
 }
}
--- PASS: TestStrictMode (1.06s)
=== RUN   TestDataTypes
2021/12/15 03:47:04 Deploying app: TestDataTypes. Item count src bucket: 0 dst bucket: 0
2021/12/15 03:47:05 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDataTypes' in metakv",
  "warnings": null
 }
}
2021/12/15 03:47:10 Waiting for app: TestDataTypes to get deployed
2021/12/15 03:47:15 Waiting for app: TestDataTypes to get deployed
2021/12/15 03:47:20 Waiting for app: TestDataTypes to get deployed
2021/12/15 03:47:25 Waiting for app: TestDataTypes to get deployed
2021/12/15 03:47:25 App: TestDataTypes got deployed
2021/12/15 03:47:25 Starting bucket ops to source bucket
2021/12/15 03:47:30 Waiting for dst bucket item count to get to: 18 curr count: 0
2021/12/15 03:47:35 Waiting for dst bucket item count to get to: 18 curr count: 8
2021/12/15 03:47:36 src & dst bucket item count matched up. src bucket count: 18 dst bucket count: 18
2021/12/15 03:47:36 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 03:47:36 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 03:47:41 Waiting for app: TestDataTypes to get un-deployed
2021/12/15 03:47:46 Waiting for app: TestDataTypes to get un-deployed
2021/12/15 03:47:51 Waiting for app: TestDataTypes to get un-deployed
2021/12/15 03:47:56 Waiting for app: TestDataTypes to get un-deployed
2021/12/15 03:47:56 App: TestDataTypes got un-deployed
2021/12/15 03:47:56 No eventing-con process running
2021/12/15 03:47:56 Function: TestDataTypes delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDataTypes deleting in the background"
}
2021/12/15 03:47:57 Context: Bucket flush, response code: 200 dump: 
2021/12/15 03:48:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 03:48:05 Context: Bucket flush, response code: 200 dump: 
2021/12/15 03:48:15 Waiting for dst bucket item count to get to: 0 curr count: 18
2021/12/15 03:48:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDataTypes (70.75s)
=== RUN   TestEnoentGet
2021/12/15 03:48:15 Deploying app: TestEnoentGet. Item count src bucket: 0 dst bucket: 0
2021/12/15 03:48:16 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentGet' in metakv",
  "warnings": null
 }
}
2021/12/15 03:48:21 Waiting for app: TestEnoentGet to get deployed
2021/12/15 03:48:26 Waiting for app: TestEnoentGet to get deployed
2021/12/15 03:48:31 Waiting for app: TestEnoentGet to get deployed
2021/12/15 03:48:36 Waiting for app: TestEnoentGet to get deployed
2021/12/15 03:48:36 App: TestEnoentGet got deployed
2021/12/15 03:48:36 Starting bucket ops to source bucket
2021/12/15 03:48:41 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/15 03:48:41 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/15 03:48:41 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 03:48:41 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 03:48:46 Waiting for app: TestEnoentGet to get un-deployed
2021/12/15 03:48:51 Waiting for app: TestEnoentGet to get un-deployed
2021/12/15 03:48:56 Waiting for app: TestEnoentGet to get un-deployed
2021/12/15 03:49:01 Waiting for app: TestEnoentGet to get un-deployed
2021/12/15 03:49:01 App: TestEnoentGet got un-deployed
2021/12/15 03:49:01 No eventing-con process running
2021/12/15 03:49:01 Function: TestEnoentGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentGet deleting in the background"
}
2021/12/15 03:49:03 Context: Bucket flush, response code: 200 dump: 
2021/12/15 03:49:13 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/15 03:49:18 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/15 03:49:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 03:49:20 Context: Bucket flush, response code: 200 dump: 
2021/12/15 03:49:30 Waiting for dst bucket item count to get to: 0 curr count: 67
2021/12/15 03:49:35 Waiting for dst bucket item count to get to: 0 curr count: 67
2021/12/15 03:49:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 03:49:35 Deploying app: TestEnoentGet. Item count src bucket: 0 dst bucket: 0
2021/12/15 03:49:36 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentGet' in metakv",
  "warnings": null
 }
}
2021/12/15 03:49:41 Waiting for app: TestEnoentGet to get deployed
2021/12/15 03:49:46 Waiting for app: TestEnoentGet to get deployed
2021/12/15 03:49:51 Waiting for app: TestEnoentGet to get deployed
2021/12/15 03:49:56 Waiting for app: TestEnoentGet to get deployed
2021/12/15 03:49:56 App: TestEnoentGet got deployed
2021/12/15 03:49:56 Starting bucket ops to source bucket
2021/12/15 03:50:01 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/15 03:50:01 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/15 03:50:01 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 03:50:02 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 03:50:07 Waiting for app: TestEnoentGet to get un-deployed
2021/12/15 03:50:12 Waiting for app: TestEnoentGet to get un-deployed
2021/12/15 03:50:17 Waiting for app: TestEnoentGet to get un-deployed
2021/12/15 03:50:22 Waiting for app: TestEnoentGet to get un-deployed
2021/12/15 03:50:22 App: TestEnoentGet got un-deployed
2021/12/15 03:50:22 No eventing-con process running
2021/12/15 03:50:22 Function: TestEnoentGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentGet deleting in the background"
}
2021/12/15 03:50:23 Context: Bucket flush, response code: 200 dump: 
2021/12/15 03:50:33 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/15 03:50:38 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/15 03:50:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 03:50:40 Context: Bucket flush, response code: 200 dump: 
2021/12/15 03:50:50 Waiting for dst bucket item count to get to: 0 curr count: 75
2021/12/15 03:50:55 Waiting for dst bucket item count to get to: 0 curr count: 75
2021/12/15 03:50:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEnoentGet (160.35s)
=== RUN   TestEnoentDelete
2021/12/15 03:50:55 Deploying app: TestEnoentDelete. Item count src bucket: 0 dst bucket: 0
2021/12/15 03:50:56 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentDelete' in metakv",
  "warnings": null
 }
}
2021/12/15 03:51:01 Waiting for app: TestEnoentDelete to get deployed
2021/12/15 03:51:06 Waiting for app: TestEnoentDelete to get deployed
2021/12/15 03:51:11 Waiting for app: TestEnoentDelete to get deployed
2021/12/15 03:51:16 Waiting for app: TestEnoentDelete to get deployed
2021/12/15 03:51:16 App: TestEnoentDelete got deployed
2021/12/15 03:51:16 Starting bucket ops to source bucket
2021/12/15 03:51:22 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/15 03:51:22 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/15 03:51:22 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 03:51:22 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 03:51:27 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/15 03:51:32 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/15 03:51:37 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/15 03:51:42 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/15 03:51:42 App: TestEnoentDelete got un-deployed
2021/12/15 03:51:42 No eventing-con process running
2021/12/15 03:51:42 Function: TestEnoentDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentDelete deleting in the background"
}
2021/12/15 03:51:44 Context: Bucket flush, response code: 200 dump: 
2021/12/15 03:51:54 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/15 03:51:59 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/15 03:51:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 03:52:00 Context: Bucket flush, response code: 200 dump: 
2021/12/15 03:52:10 Waiting for dst bucket item count to get to: 0 curr count: 80
2021/12/15 03:52:15 Waiting for dst bucket item count to get to: 0 curr count: 80
2021/12/15 03:52:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 03:52:15 Deploying app: TestEnoentDelete. Item count src bucket: 0 dst bucket: 0
2021/12/15 03:52:17 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentDelete' in metakv",
  "warnings": null
 }
}
2021/12/15 03:52:22 Waiting for app: TestEnoentDelete to get deployed
2021/12/15 03:52:27 Waiting for app: TestEnoentDelete to get deployed
2021/12/15 03:52:32 Waiting for app: TestEnoentDelete to get deployed
2021/12/15 03:52:37 Waiting for app: TestEnoentDelete to get deployed
2021/12/15 03:52:37 App: TestEnoentDelete got deployed
2021/12/15 03:52:37 Starting bucket ops to source bucket
2021/12/15 03:52:42 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/15 03:52:42 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/15 03:52:42 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 03:52:42 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 03:52:47 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/15 03:52:52 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/15 03:52:57 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/15 03:53:02 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/15 03:53:02 App: TestEnoentDelete got un-deployed
2021/12/15 03:53:02 No eventing-con process running
2021/12/15 03:53:02 Function: TestEnoentDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentDelete deleting in the background"
}
2021/12/15 03:53:04 Context: Bucket flush, response code: 200 dump: 
2021/12/15 03:53:14 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/15 03:53:19 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/15 03:53:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 03:53:21 Context: Bucket flush, response code: 200 dump: 
2021/12/15 03:53:31 Waiting for dst bucket item count to get to: 0 curr count: 83
2021/12/15 03:53:36 Waiting for dst bucket item count to get to: 0 curr count: 83
2021/12/15 03:53:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEnoentDelete (160.39s)
=== RUN   TestError
2021/12/15 03:53:46 Waiting for app: TestError to get un-deployed
2021/12/15 03:53:46 App: TestError got un-deployed
2021/12/15 03:53:46 No eventing-con process running
2021/12/15 03:53:46 Function: TestError delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestError not found"
 }
}
2021/12/15 03:53:47 Context: Bucket flush, response code: 200 dump: 
2021/12/15 03:53:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 03:53:54 Context: Bucket flush, response code: 200 dump: 
2021/12/15 03:53:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 03:53:59 Deploying app: TestError. Item count src bucket: 0 dst bucket: 0
2021/12/15 03:54:00 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestError' in metakv",
  "warnings": null
 }
}
2021/12/15 03:54:05 Waiting for app: TestError to get deployed
2021/12/15 03:54:10 Waiting for app: TestError to get deployed
2021/12/15 03:54:15 Waiting for app: TestError to get deployed
2021/12/15 03:54:20 Waiting for app: TestError to get deployed
2021/12/15 03:54:20 App: TestError got deployed
2021/12/15 03:54:20 Starting bucket ops to source bucket
2021/12/15 03:54:26 Waiting for dst bucket item count to get to: 300 curr count: 0
2021/12/15 03:54:31 Waiting for dst bucket item count to get to: 300 curr count: 0
2021/12/15 03:54:31 src & dst bucket item count matched up. src bucket count: 300 dst bucket count: 300
2021/12/15 03:54:31 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 03:54:31 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 03:54:36 Waiting for app: TestError to get un-deployed
2021/12/15 03:54:41 Waiting for app: TestError to get un-deployed
2021/12/15 03:54:46 Waiting for app: TestError to get un-deployed
2021/12/15 03:54:51 Waiting for app: TestError to get un-deployed
2021/12/15 03:54:51 App: TestError got un-deployed
2021/12/15 03:54:51 No eventing-con process running
2021/12/15 03:54:51 Function: TestError delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestError deleting in the background"
}
2021/12/15 03:54:53 Context: Bucket flush, response code: 200 dump: 
2021/12/15 03:55:03 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/15 03:55:08 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/15 03:55:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 03:55:09 Context: Bucket flush, response code: 200 dump: 
2021/12/15 03:55:19 Waiting for dst bucket item count to get to: 0 curr count: 156
2021/12/15 03:55:24 Waiting for dst bucket item count to get to: 0 curr count: 156
2021/12/15 03:55:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestError (108.77s)
=== RUN   TestCRLF
2021/12/15 03:55:34 Waiting for app: TestCRLF to get un-deployed
2021/12/15 03:55:34 App: TestCRLF got un-deployed
2021/12/15 03:55:34 No eventing-con process running
2021/12/15 03:55:34 Function: TestCRLF delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestCRLF not found"
 }
}
2021/12/15 03:55:36 Context: Bucket flush, response code: 200 dump: 
2021/12/15 03:55:41 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 03:55:43 Context: Bucket flush, response code: 200 dump: 
2021/12/15 03:55:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 03:55:48 Deploying app: TestCRLF. Item count src bucket: 0 dst bucket: 0
2021/12/15 03:55:49 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCRLF' in metakv",
  "warnings": null
 }
}
2021/12/15 03:55:54 Waiting for app: TestCRLF to get deployed
2021/12/15 03:55:59 Waiting for app: TestCRLF to get deployed
2021/12/15 03:56:04 Waiting for app: TestCRLF to get deployed
2021/12/15 03:56:09 Waiting for app: TestCRLF to get deployed
2021/12/15 03:56:09 App: TestCRLF got deployed
2021/12/15 03:56:09 Starting bucket ops to source bucket
2021/12/15 03:56:14 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/15 03:56:19 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/15 03:56:24 Waiting for dst bucket item count to get to: 100 curr count: 39
2021/12/15 03:56:24 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/15 03:56:24 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 03:56:25 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 03:56:30 Waiting for app: TestCRLF to get un-deployed
2021/12/15 03:56:35 Waiting for app: TestCRLF to get un-deployed
2021/12/15 03:56:40 Waiting for app: TestCRLF to get un-deployed
2021/12/15 03:56:45 Waiting for app: TestCRLF to get un-deployed
2021/12/15 03:56:45 App: TestCRLF got un-deployed
2021/12/15 03:56:45 No eventing-con process running
2021/12/15 03:56:45 Function: TestCRLF delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCRLF deleting in the background"
}
2021/12/15 03:56:47 Context: Bucket flush, response code: 200 dump: 
2021/12/15 03:56:57 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/15 03:56:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 03:56:58 Context: Bucket flush, response code: 200 dump: 
2021/12/15 03:57:08 Waiting for dst bucket item count to get to: 0 curr count: 90
2021/12/15 03:57:13 Waiting for dst bucket item count to get to: 0 curr count: 29
2021/12/15 03:57:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCRLF (108.89s)
=== RUN   TestImportExport
2021/12/15 03:57:23 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/15 03:57:23 App: TestImportExport_n1ql_insert_on_update got un-deployed
2021/12/15 03:57:23 No eventing-con process running
2021/12/15 03:57:23 Function: TestImportExport_n1ql_insert_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestImportExport_n1ql_insert_on_update not found"
 }
}
2021/12/15 03:57:25 Context: Bucket flush, response code: 200 dump: 
2021/12/15 03:57:30 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 03:57:32 Context: Bucket flush, response code: 200 dump: 
2021/12/15 03:57:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 03:57:42 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/15 03:57:42 App: TestImportExport_bucket_op_on_update got un-deployed
2021/12/15 03:57:42 No eventing-con process running
2021/12/15 03:57:42 Function: TestImportExport_bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestImportExport_bucket_op_on_update not found"
 }
}
2021/12/15 03:57:44 Context: Bucket flush, response code: 200 dump: 
2021/12/15 03:57:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 03:57:51 Context: Bucket flush, response code: 200 dump: 
2021/12/15 03:57:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 03:57:56 Deploying app: TestImportExport_n1ql_insert_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/15 03:57:57 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestImportExport_n1ql_insert_on_update' in metakv",
  "warnings": null
 }
}
2021/12/15 03:57:57 Deploying app: TestImportExport_bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/15 03:57:58 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestImportExport_bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/15 03:58:03 Waiting for app: TestImportExport_n1ql_insert_on_update to get deployed
2021/12/15 03:58:08 Waiting for app: TestImportExport_n1ql_insert_on_update to get deployed
2021/12/15 03:58:13 Waiting for app: TestImportExport_n1ql_insert_on_update to get deployed
2021/12/15 03:58:18 Waiting for app: TestImportExport_n1ql_insert_on_update to get deployed
2021/12/15 03:58:18 App: TestImportExport_n1ql_insert_on_update got deployed
2021/12/15 03:58:23 Waiting for app: TestImportExport_bucket_op_on_update to get deployed
2021/12/15 03:58:28 Waiting for app: TestImportExport_bucket_op_on_update to get deployed
2021/12/15 03:58:33 Waiting for app: TestImportExport_bucket_op_on_update to get deployed
2021/12/15 03:58:33 App: TestImportExport_bucket_op_on_update got deployed
2021/12/15 03:58:33 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 03:58:38 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/15 03:58:43 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/15 03:58:48 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/15 03:58:53 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/15 03:58:53 App: TestImportExport_n1ql_insert_on_update got un-deployed
2021/12/15 03:58:53 No eventing-con process running
2021/12/15 03:58:53 Function: TestImportExport_n1ql_insert_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestImportExport_n1ql_insert_on_update deleting in the background"
}
2021/12/15 03:58:53 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 03:58:58 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/15 03:59:03 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/15 03:59:08 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/15 03:59:13 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/15 03:59:13 App: TestImportExport_bucket_op_on_update got un-deployed
2021/12/15 03:59:13 No eventing-con process running
2021/12/15 03:59:13 Function: TestImportExport_bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestImportExport_bucket_op_on_update deleting in the background"
}
2021/12/15 03:59:31 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/15 03:59:31 App: TestImportExport_bucket_op_on_update got un-deployed
2021/12/15 03:59:31 No eventing-con process running
2021/12/15 03:59:31 Function: TestImportExport_bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestImportExport_bucket_op_on_update deleting in the background"
}
2021/12/15 03:59:33 Context: Bucket flush, response code: 200 dump: 
2021/12/15 03:59:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 03:59:39 Context: Bucket flush, response code: 200 dump: 
2021/12/15 03:59:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 03:59:49 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/15 03:59:49 App: TestImportExport_n1ql_insert_on_update got un-deployed
2021/12/15 03:59:49 No eventing-con process running
2021/12/15 03:59:49 Function: TestImportExport_n1ql_insert_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestImportExport_n1ql_insert_on_update deleting in the background"
}
2021/12/15 03:59:51 Context: Bucket flush, response code: 200 dump: 
2021/12/15 03:59:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 03:59:58 Context: Bucket flush, response code: 200 dump: 
2021/12/15 04:00:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestImportExport (169.62s)
=== RUN   TestDeployUndeployLoopNonDefaultSettings
2021/12/15 04:00:13 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/15 04:00:13 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/15 04:00:13 No eventing-con process running
2021/12/15 04:00:13 Function: TestDeployUndeployLoopNonDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDeployUndeployLoopNonDefaultSettings not found"
 }
}
2021/12/15 04:00:15 Context: Bucket flush, response code: 200 dump: 
2021/12/15 04:00:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 04:00:21 Context: Bucket flush, response code: 200 dump: 
2021/12/15 04:00:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 04:00:26 Deploying app: TestDeployUndeployLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/15 04:00:28 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/15 04:00:28 Starting bucket ops to source bucket
2021/12/15 04:00:36 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 04:00:41 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 04:00:46 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 04:00:51 Waiting for dst bucket item count to get to: 5000 curr count: 401
2021/12/15 04:00:56 Waiting for dst bucket item count to get to: 5000 curr count: 401
2021/12/15 04:00:56 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/15 04:01:01 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get deployed
2021/12/15 04:01:01 App: TestDeployUndeployLoopNonDefaultSettings got deployed
2021/12/15 04:01:01 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 04:01:01 Undeploying app: bucket_op_on_update
2021/12/15 04:01:01 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 04:01:06 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/15 04:01:11 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/15 04:01:16 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/15 04:01:21 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/15 04:01:21 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/15 04:01:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 04:01:21 No eventing-con process running
2021/12/15 04:01:23 Context: Bucket flush, response code: 200 dump: 
2021/12/15 04:01:30 Context: Bucket flush, response code: 200 dump: 
2021/12/15 04:02:05 Deploying app: TestDeployUndeployLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/15 04:02:06 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/15 04:02:06 Starting bucket ops to source bucket
2021/12/15 04:02:14 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 04:02:19 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 04:02:24 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 04:02:29 Waiting for dst bucket item count to get to: 5000 curr count: 2135
2021/12/15 04:02:34 Waiting for dst bucket item count to get to: 5000 curr count: 2135
2021/12/15 04:02:34 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/15 04:02:39 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get deployed
2021/12/15 04:02:39 App: TestDeployUndeployLoopNonDefaultSettings got deployed
2021/12/15 04:02:39 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 04:02:39 Undeploying app: bucket_op_on_update
2021/12/15 04:02:39 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 04:02:44 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/15 04:02:49 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/15 04:02:54 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/15 04:02:59 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/15 04:02:59 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/15 04:02:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 04:02:59 No eventing-con process running
2021/12/15 04:03:01 Context: Bucket flush, response code: 200 dump: 
2021/12/15 04:03:08 Context: Bucket flush, response code: 200 dump: 
2021/12/15 04:03:43 Deploying app: TestDeployUndeployLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/15 04:03:44 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/15 04:03:44 Starting bucket ops to source bucket
2021/12/15 04:03:52 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 04:03:57 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 04:04:02 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 04:04:07 Waiting for dst bucket item count to get to: 5000 curr count: 1272
2021/12/15 04:04:12 Waiting for dst bucket item count to get to: 5000 curr count: 3772
2021/12/15 04:04:12 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/15 04:04:17 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get deployed
2021/12/15 04:04:17 App: TestDeployUndeployLoopNonDefaultSettings got deployed
2021/12/15 04:04:17 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 04:04:17 Undeploying app: bucket_op_on_update
2021/12/15 04:04:17 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 04:04:22 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/15 04:04:27 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/15 04:04:32 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/15 04:04:37 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/15 04:04:37 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/15 04:04:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 04:04:37 No eventing-con process running
2021/12/15 04:04:39 Context: Bucket flush, response code: 200 dump: 
2021/12/15 04:04:46 Context: Bucket flush, response code: 200 dump: 
2021/12/15 04:05:21 Deploying app: TestDeployUndeployLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/15 04:05:23 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/15 04:05:23 Starting bucket ops to source bucket
2021/12/15 04:05:31 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 04:05:36 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 04:05:41 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 04:05:46 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 04:05:46 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/15 04:05:51 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get deployed
2021/12/15 04:05:51 App: TestDeployUndeployLoopNonDefaultSettings got deployed
2021/12/15 04:05:51 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 04:05:51 Undeploying app: bucket_op_on_update
2021/12/15 04:05:51 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 04:05:56 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/15 04:06:01 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/15 04:06:06 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/15 04:06:11 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/15 04:06:11 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/15 04:06:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 04:06:11 No eventing-con process running
2021/12/15 04:06:12 Context: Bucket flush, response code: 200 dump: 
2021/12/15 04:06:20 Context: Bucket flush, response code: 200 dump: 
2021/12/15 04:06:55 Deploying app: TestDeployUndeployLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/15 04:06:56 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/15 04:06:56 Starting bucket ops to source bucket
2021/12/15 04:07:04 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 04:07:09 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 04:07:14 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 04:07:19 Waiting for dst bucket item count to get to: 5000 curr count: 2440
2021/12/15 04:07:24 Waiting for dst bucket item count to get to: 5000 curr count: 2440
2021/12/15 04:07:24 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/15 04:07:29 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get deployed
2021/12/15 04:07:29 App: TestDeployUndeployLoopNonDefaultSettings got deployed
2021/12/15 04:07:29 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 04:07:29 Undeploying app: bucket_op_on_update
2021/12/15 04:07:29 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 04:07:34 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/15 04:07:39 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/15 04:07:44 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/15 04:07:49 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/15 04:07:49 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/15 04:07:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 04:07:49 No eventing-con process running
2021/12/15 04:07:51 Context: Bucket flush, response code: 200 dump: 
2021/12/15 04:07:58 Context: Bucket flush, response code: 200 dump: 
2021/12/15 04:08:33 Function: TestDeployUndeployLoopNonDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDeployUndeployLoopNonDefaultSettings deleting in the background"
}
--- PASS: TestDeployUndeployLoopNonDefaultSettings (510.39s)
=== RUN   TestOnUpdateN1QLOp
2021/12/15 04:08:43 Waiting for app: TestOnUpdateN1QLOp to get un-deployed
2021/12/15 04:08:43 App: TestOnUpdateN1QLOp got un-deployed
2021/12/15 04:08:43 No eventing-con process running
2021/12/15 04:08:43 Function: TestOnUpdateN1QLOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateN1QLOp not found"
 }
}
2021/12/15 04:08:45 Context: Bucket flush, response code: 200 dump: 
2021/12/15 04:08:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 04:08:52 Context: Bucket flush, response code: 200 dump: 
2021/12/15 04:08:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 04:08:57 Deploying app: TestOnUpdateN1QLOp. Item count src bucket: 0 dst bucket: 0
2021/12/15 04:08:58 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateN1QLOp' in metakv",
  "warnings": null
 }
}
2021/12/15 04:08:58 Starting bucket ops to source bucket
2021/12/15 04:09:06 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 04:09:11 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 04:09:16 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 04:09:21 Waiting for dst bucket item count to get to: 5000 curr count: 48
2021/12/15 04:09:26 Waiting for dst bucket item count to get to: 5000 curr count: 48
2021/12/15 04:09:26 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/15 04:09:26 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 04:09:26 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 04:09:31 Waiting for app: TestOnUpdateN1QLOp to get un-deployed
2021/12/15 04:09:36 Waiting for app: TestOnUpdateN1QLOp to get un-deployed
2021/12/15 04:09:41 Waiting for app: TestOnUpdateN1QLOp to get un-deployed
2021/12/15 04:09:46 Waiting for app: TestOnUpdateN1QLOp to get un-deployed
2021/12/15 04:09:46 App: TestOnUpdateN1QLOp got un-deployed
2021/12/15 04:09:46 No eventing-con process running
2021/12/15 04:09:46 Function: TestOnUpdateN1QLOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateN1QLOp deleting in the background"
}
2021/12/15 04:09:48 Context: Bucket flush, response code: 200 dump: 
2021/12/15 04:09:58 Waiting for dst bucket item count to get to: 0 curr count: 3369
2021/12/15 04:10:03 Waiting for dst bucket item count to get to: 0 curr count: 869
2021/12/15 04:10:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 04:10:05 Context: Bucket flush, response code: 200 dump: 
2021/12/15 04:10:15 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/15 04:10:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateN1QLOp (101.64s)
=== RUN   TestOnUpdateBucketOpDefaultSettings
2021/12/15 04:10:25 Waiting for app: TestOnUpdateBucketOpDefaultSettings to get un-deployed
2021/12/15 04:10:25 App: TestOnUpdateBucketOpDefaultSettings got un-deployed
2021/12/15 04:10:25 No eventing-con process running
2021/12/15 04:10:25 Function: TestOnUpdateBucketOpDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateBucketOpDefaultSettings not found"
 }
}
2021/12/15 04:10:27 Context: Bucket flush, response code: 200 dump: 
2021/12/15 04:10:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 04:10:34 Context: Bucket flush, response code: 200 dump: 
2021/12/15 04:10:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 04:10:39 Deploying app: TestOnUpdateBucketOpDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/15 04:10:40 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateBucketOpDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/15 04:10:40 Starting bucket ops to source bucket
2021/12/15 04:10:48 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 04:10:53 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 04:10:58 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 04:11:03 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 04:11:08 Waiting for dst bucket item count to get to: 5000 curr count: 2500
2021/12/15 04:11:08 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/15 04:11:08 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 04:11:08 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 04:11:13 Waiting for app: TestOnUpdateBucketOpDefaultSettings to get un-deployed
2021/12/15 04:11:18 Waiting for app: TestOnUpdateBucketOpDefaultSettings to get un-deployed
2021/12/15 04:11:23 Waiting for app: TestOnUpdateBucketOpDefaultSettings to get un-deployed
2021/12/15 04:11:28 Waiting for app: TestOnUpdateBucketOpDefaultSettings to get un-deployed
2021/12/15 04:11:28 App: TestOnUpdateBucketOpDefaultSettings got un-deployed
2021/12/15 04:11:28 No eventing-con process running
2021/12/15 04:11:28 Function: TestOnUpdateBucketOpDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateBucketOpDefaultSettings deleting in the background"
}
2021/12/15 04:11:30 Context: Bucket flush, response code: 200 dump: 
2021/12/15 04:11:40 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/15 04:11:45 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/15 04:11:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 04:11:47 Context: Bucket flush, response code: 200 dump: 
2021/12/15 04:11:57 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/15 04:11:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateBucketOpDefaultSettings (101.59s)
=== RUN   TestOnUpdateBucketOpNonDefaultSettings
2021/12/15 04:12:07 Waiting for app: TestOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/15 04:12:07 App: TestOnUpdateBucketOpNonDefaultSettings got un-deployed
2021/12/15 04:12:07 No eventing-con process running
2021/12/15 04:12:07 Function: TestOnUpdateBucketOpNonDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateBucketOpNonDefaultSettings not found"
 }
}
2021/12/15 04:12:08 Context: Bucket flush, response code: 200 dump: 
2021/12/15 04:12:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 04:12:15 Context: Bucket flush, response code: 200 dump: 
2021/12/15 04:12:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 04:12:20 Deploying app: TestOnUpdateBucketOpNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/15 04:12:21 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateBucketOpNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/15 04:12:21 Starting bucket ops to source bucket
2021/12/15 04:12:29 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 04:12:34 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 04:12:39 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 04:12:44 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 04:12:44 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/15 04:12:44 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 04:12:44 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 04:12:49 Waiting for app: TestOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/15 04:12:55 Waiting for app: TestOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/15 04:13:00 Waiting for app: TestOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/15 04:13:05 Waiting for app: TestOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/15 04:13:05 App: TestOnUpdateBucketOpNonDefaultSettings got un-deployed
2021/12/15 04:13:05 No eventing-con process running
2021/12/15 04:13:05 Function: TestOnUpdateBucketOpNonDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateBucketOpNonDefaultSettings deleting in the background"
}
2021/12/15 04:13:06 Context: Bucket flush, response code: 200 dump: 
2021/12/15 04:13:16 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/15 04:13:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 04:13:18 Context: Bucket flush, response code: 200 dump: 
2021/12/15 04:13:28 Waiting for dst bucket item count to get to: 0 curr count: 3958
2021/12/15 04:13:33 Waiting for dst bucket item count to get to: 0 curr count: 1458
2021/12/15 04:13:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateBucketOpNonDefaultSettings (96.69s)
=== RUN   TestOnUpdateBucketOpDefaultSettings10K
2021/12/15 04:13:43 Waiting for app: TestOnUpdateBucketOpDefaultSettings10K to get un-deployed
2021/12/15 04:13:43 App: TestOnUpdateBucketOpDefaultSettings10K got un-deployed
2021/12/15 04:13:43 No eventing-con process running
2021/12/15 04:13:43 Function: TestOnUpdateBucketOpDefaultSettings10K delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateBucketOpDefaultSettings10K not found"
 }
}
2021/12/15 04:13:45 Context: Bucket flush, response code: 200 dump: 
2021/12/15 04:13:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 04:13:52 Context: Bucket flush, response code: 200 dump: 
2021/12/15 04:13:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 04:13:57 Deploying app: TestOnUpdateBucketOpDefaultSettings10K. Item count src bucket: 0 dst bucket: 0
2021/12/15 04:13:58 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateBucketOpDefaultSettings10K' in metakv",
  "warnings": null
 }
}
2021/12/15 04:13:58 Starting bucket ops to source bucket
2021/12/15 04:14:06 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 04:14:11 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 04:14:16 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 04:14:21 Waiting for dst bucket item count to get to: 5000 curr count: 220
2021/12/15 04:14:26 Waiting for dst bucket item count to get to: 5000 curr count: 220
2021/12/15 04:14:26 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/15 04:14:26 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 04:14:26 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 04:14:31 Waiting for app: TestOnUpdateBucketOpDefaultSettings10K to get un-deployed
2021/12/15 04:14:36 Waiting for app: TestOnUpdateBucketOpDefaultSettings10K to get un-deployed
2021/12/15 04:14:41 Waiting for app: TestOnUpdateBucketOpDefaultSettings10K to get un-deployed
2021/12/15 04:14:46 Waiting for app: TestOnUpdateBucketOpDefaultSettings10K to get un-deployed
2021/12/15 04:14:46 App: TestOnUpdateBucketOpDefaultSettings10K got un-deployed
2021/12/15 04:14:46 No eventing-con process running
2021/12/15 04:14:46 Function: TestOnUpdateBucketOpDefaultSettings10K delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateBucketOpDefaultSettings10K deleting in the background"
}
2021/12/15 04:14:48 Context: Bucket flush, response code: 200 dump: 
2021/12/15 04:14:58 Waiting for dst bucket item count to get to: 0 curr count: 3619
2021/12/15 04:15:03 Waiting for dst bucket item count to get to: 0 curr count: 1119
2021/12/15 04:15:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 04:15:05 Context: Bucket flush, response code: 200 dump: 
2021/12/15 04:15:15 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/15 04:15:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateBucketOpDefaultSettings10K (101.74s)
=== RUN   TestOnUpdateBucketOpDefaultSettings100K
2021/12/15 04:15:25 Waiting for app: TestOnUpdateBucketOpDefaultSettings100K to get un-deployed
2021/12/15 04:15:25 App: TestOnUpdateBucketOpDefaultSettings100K got un-deployed
2021/12/15 04:15:25 No eventing-con process running
2021/12/15 04:15:25 Function: TestOnUpdateBucketOpDefaultSettings100K delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateBucketOpDefaultSettings100K not found"
 }
}
2021/12/15 04:15:27 Context: Bucket flush, response code: 200 dump: 
2021/12/15 04:15:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 04:15:34 Context: Bucket flush, response code: 200 dump: 
2021/12/15 04:15:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 04:15:39 Deploying app: TestOnUpdateBucketOpDefaultSettings100K. Item count src bucket: 0 dst bucket: 0
2021/12/15 04:15:40 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateBucketOpDefaultSettings100K' in metakv",
  "warnings": null
 }
}
2021/12/15 04:15:40 Starting bucket ops to source bucket
2021/12/15 04:15:48 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 04:15:53 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 04:15:58 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 04:16:03 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 04:16:08 Waiting for dst bucket item count to get to: 5000 curr count: 2500
2021/12/15 04:16:08 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/15 04:16:08 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 04:16:08 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 04:16:13 Waiting for app: TestOnUpdateBucketOpDefaultSettings100K to get un-deployed
2021/12/15 04:16:18 Waiting for app: TestOnUpdateBucketOpDefaultSettings100K to get un-deployed
2021/12/15 04:16:23 Waiting for app: TestOnUpdateBucketOpDefaultSettings100K to get un-deployed
2021/12/15 04:16:28 Waiting for app: TestOnUpdateBucketOpDefaultSettings100K to get un-deployed
2021/12/15 04:16:29 App: TestOnUpdateBucketOpDefaultSettings100K got un-deployed
2021/12/15 04:16:29 No eventing-con process running
2021/12/15 04:16:29 Function: TestOnUpdateBucketOpDefaultSettings100K delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateBucketOpDefaultSettings100K deleting in the background"
}
2021/12/15 04:16:30 Context: Bucket flush, response code: 200 dump: 
2021/12/15 04:16:40 Waiting for dst bucket item count to get to: 0 curr count: 3650
2021/12/15 04:16:45 Waiting for dst bucket item count to get to: 0 curr count: 3650
2021/12/15 04:16:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 04:16:47 Context: Bucket flush, response code: 200 dump: 
2021/12/15 04:16:57 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/15 04:16:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateBucketOpDefaultSettings100K (102.17s)
=== RUN   TestOnDeleteBucketOp
2021/12/15 04:17:07 Waiting for app: TestOnDeleteBucketOp to get un-deployed
2021/12/15 04:17:07 App: TestOnDeleteBucketOp got un-deployed
2021/12/15 04:17:07 No eventing-con process running
2021/12/15 04:17:07 Function: TestOnDeleteBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnDeleteBucketOp not found"
 }
}
2021/12/15 04:17:09 Context: Bucket flush, response code: 200 dump: 
2021/12/15 04:17:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 04:17:16 Context: Bucket flush, response code: 200 dump: 
2021/12/15 04:17:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 04:17:21 Deploying app: TestOnDeleteBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/15 04:17:22 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnDeleteBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/15 04:17:22 Starting bucket ops to source bucket
2021/12/15 04:17:33 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 04:17:38 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 04:17:43 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 04:17:43 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/15 04:17:43 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 04:17:43 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 04:17:48 Waiting for app: TestOnDeleteBucketOp to get un-deployed
2021/12/15 04:17:53 Waiting for app: TestOnDeleteBucketOp to get un-deployed
2021/12/15 04:17:58 Waiting for app: TestOnDeleteBucketOp to get un-deployed
2021/12/15 04:18:03 Waiting for app: TestOnDeleteBucketOp to get un-deployed
2021/12/15 04:18:03 App: TestOnDeleteBucketOp got un-deployed
2021/12/15 04:18:03 No eventing-con process running
2021/12/15 04:18:03 Function: TestOnDeleteBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnDeleteBucketOp deleting in the background"
}
2021/12/15 04:18:05 Context: Bucket flush, response code: 200 dump: 
2021/12/15 04:18:10 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 04:18:12 Context: Bucket flush, response code: 200 dump: 
2021/12/15 04:18:22 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/15 04:18:27 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/15 04:18:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnDeleteBucketOp (90.04s)
=== RUN   TestOnDeleteBucketOp5K
2021/12/15 04:18:37 Waiting for app: TestOnDeleteBucketOp5K to get un-deployed
2021/12/15 04:18:37 App: TestOnDeleteBucketOp5K got un-deployed
2021/12/15 04:18:37 No eventing-con process running
2021/12/15 04:18:37 Function: TestOnDeleteBucketOp5K delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnDeleteBucketOp5K not found"
 }
}
2021/12/15 04:18:39 Context: Bucket flush, response code: 200 dump: 
2021/12/15 04:18:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 04:18:46 Context: Bucket flush, response code: 200 dump: 
2021/12/15 04:18:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 04:18:51 Deploying app: TestOnDeleteBucketOp5K. Item count src bucket: 0 dst bucket: 0
2021/12/15 04:18:52 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnDeleteBucketOp5K' in metakv",
  "warnings": null
 }
}
2021/12/15 04:18:52 Starting bucket ops to source bucket
2021/12/15 04:19:03 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 04:19:08 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 04:19:13 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 04:19:13 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/15 04:19:13 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 04:19:13 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 04:19:18 Waiting for app: TestOnDeleteBucketOp5K to get un-deployed
2021/12/15 04:19:23 Waiting for app: TestOnDeleteBucketOp5K to get un-deployed
2021/12/15 04:19:28 Waiting for app: TestOnDeleteBucketOp5K to get un-deployed
2021/12/15 04:19:33 Waiting for app: TestOnDeleteBucketOp5K to get un-deployed
2021/12/15 04:19:33 App: TestOnDeleteBucketOp5K got un-deployed
2021/12/15 04:19:33 No eventing-con process running
2021/12/15 04:19:33 Function: TestOnDeleteBucketOp5K delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnDeleteBucketOp5K deleting in the background"
}
2021/12/15 04:19:35 Context: Bucket flush, response code: 200 dump: 
2021/12/15 04:19:40 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 04:19:42 Context: Bucket flush, response code: 200 dump: 
2021/12/15 04:19:52 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/15 04:19:57 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/15 04:19:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnDeleteBucketOp5K (90.05s)
=== RUN   TestTimerBucketOp
2021/12/15 04:20:07 Waiting for app: TestTimerBucketOp to get un-deployed
2021/12/15 04:20:07 App: TestTimerBucketOp got un-deployed
2021/12/15 04:20:07 No eventing-con process running
2021/12/15 04:20:07 Function: TestTimerBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestTimerBucketOp not found"
 }
}
2021/12/15 04:20:09 Context: Bucket flush, response code: 200 dump: 
2021/12/15 04:20:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 04:20:16 Context: Bucket flush, response code: 200 dump: 
2021/12/15 04:20:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 04:20:21 Deploying app: TestTimerBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/15 04:20:22 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestTimerBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/15 04:20:22 Starting bucket ops to source bucket
2021/12/15 04:20:30 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 04:20:35 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 04:20:40 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 04:20:45 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 04:20:50 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 04:20:55 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 04:21:00 Waiting for dst bucket item count to get to: 5000 curr count: 2712
2021/12/15 04:21:05 Waiting for dst bucket item count to get to: 5000 curr count: 2712
2021/12/15 04:21:05 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/15 04:21:05 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 04:21:05 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 04:21:10 Waiting for app: TestTimerBucketOp to get un-deployed
2021/12/15 04:21:15 Waiting for app: TestTimerBucketOp to get un-deployed
2021/12/15 04:21:20 Waiting for app: TestTimerBucketOp to get un-deployed
2021/12/15 04:21:25 Waiting for app: TestTimerBucketOp to get un-deployed
2021/12/15 04:21:25 App: TestTimerBucketOp got un-deployed
2021/12/15 04:21:25 No eventing-con process running
2021/12/15 04:21:25 Function: TestTimerBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestTimerBucketOp deleting in the background"
}
2021/12/15 04:21:27 Context: Bucket flush, response code: 200 dump: 
2021/12/15 04:21:37 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/15 04:21:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 04:21:39 Context: Bucket flush, response code: 200 dump: 
2021/12/15 04:21:49 Waiting for dst bucket item count to get to: 0 curr count: 4737
2021/12/15 04:21:54 Waiting for dst bucket item count to get to: 0 curr count: 2237
2021/12/15 04:21:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestTimerBucketOp (116.48s)
=== RUN   TestTimerInPastBucketOp
2021/12/15 04:22:04 Waiting for app: TestTimerInPastBucketOp to get un-deployed
2021/12/15 04:22:04 App: TestTimerInPastBucketOp got un-deployed
2021/12/15 04:22:04 No eventing-con process running
2021/12/15 04:22:04 Function: TestTimerInPastBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestTimerInPastBucketOp not found"
 }
}
2021/12/15 04:22:06 Context: Bucket flush, response code: 200 dump: 
2021/12/15 04:22:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 04:22:12 Context: Bucket flush, response code: 200 dump: 
2021/12/15 04:22:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 04:22:17 Deploying app: TestTimerInPastBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/15 04:22:18 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestTimerInPastBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/15 04:22:18 Starting bucket ops to source bucket
2021/12/15 04:22:26 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 04:22:31 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 04:22:36 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 04:22:41 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 04:22:46 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 04:22:51 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 04:22:56 Waiting for dst bucket item count to get to: 5000 curr count: 2416
2021/12/15 04:23:01 Waiting for dst bucket item count to get to: 5000 curr count: 4916
2021/12/15 04:23:01 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/15 04:23:02 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 04:23:02 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 04:23:07 Waiting for app: TestTimerInPastBucketOp to get un-deployed
2021/12/15 04:23:12 Waiting for app: TestTimerInPastBucketOp to get un-deployed
2021/12/15 04:23:17 Waiting for app: TestTimerInPastBucketOp to get un-deployed
2021/12/15 04:23:22 Waiting for app: TestTimerInPastBucketOp to get un-deployed
2021/12/15 04:23:22 App: TestTimerInPastBucketOp got un-deployed
2021/12/15 04:23:22 No eventing-con process running
2021/12/15 04:23:22 Function: TestTimerInPastBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestTimerInPastBucketOp deleting in the background"
}
2021/12/15 04:23:23 Context: Bucket flush, response code: 200 dump: 
2021/12/15 04:23:33 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/15 04:23:38 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/15 04:23:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 04:23:40 Context: Bucket flush, response code: 200 dump: 
2021/12/15 04:23:50 Waiting for dst bucket item count to get to: 0 curr count: 3586
2021/12/15 04:23:55 Waiting for dst bucket item count to get to: 0 curr count: 3586
2021/12/15 04:23:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestTimerInPastBucketOp (121.47s)
=== RUN   TestDeployUndeployLoopTimer
2021/12/15 04:24:05 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/15 04:24:05 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/15 04:24:05 No eventing-con process running
2021/12/15 04:24:05 Function: TestDeployUndeployLoopTimer delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDeployUndeployLoopTimer not found"
 }
}
2021/12/15 04:24:07 Context: Bucket flush, response code: 200 dump: 
2021/12/15 04:24:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 04:24:14 Context: Bucket flush, response code: 200 dump: 
2021/12/15 04:24:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 04:24:19 Deploying app: TestDeployUndeployLoopTimer. Item count src bucket: 0 dst bucket: 0
2021/12/15 04:24:20 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopTimer' in metakv",
  "warnings": null
 }
}
2021/12/15 04:24:20 Starting bucket ops to source bucket
2021/12/15 04:24:28 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 04:24:33 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 04:24:38 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 04:24:43 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 04:24:48 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 04:24:53 Waiting for dst bucket item count to get to: 5000 curr count: 142
2021/12/15 04:24:58 Waiting for dst bucket item count to get to: 5000 curr count: 2009
2021/12/15 04:25:03 Waiting for dst bucket item count to get to: 5000 curr count: 4367
2021/12/15 04:25:03 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/15 04:25:08 Waiting for app: TestDeployUndeployLoopTimer to get deployed
2021/12/15 04:25:08 App: TestDeployUndeployLoopTimer got deployed
2021/12/15 04:25:08 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 04:25:08 Undeploying app: bucket_op_with_timer
2021/12/15 04:25:08 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 04:25:13 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/15 04:25:18 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/15 04:25:23 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/15 04:25:28 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/15 04:25:28 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/15 04:25:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 04:25:28 No eventing-con process running
2021/12/15 04:25:30 Context: Bucket flush, response code: 200 dump: 
2021/12/15 04:25:37 Context: Bucket flush, response code: 200 dump: 
2021/12/15 04:26:12 Deploying app: TestDeployUndeployLoopTimer. Item count src bucket: 0 dst bucket: 0
2021/12/15 04:26:14 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopTimer' in metakv",
  "warnings": null
 }
}
2021/12/15 04:26:14 Starting bucket ops to source bucket
2021/12/15 04:26:21 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 04:26:26 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 04:26:31 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 04:26:36 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 04:26:41 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 04:26:46 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 04:26:52 Waiting for dst bucket item count to get to: 5000 curr count: 17
2021/12/15 04:26:57 Waiting for dst bucket item count to get to: 5000 curr count: 2517
2021/12/15 04:26:57 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/15 04:27:02 Waiting for app: TestDeployUndeployLoopTimer to get deployed
2021/12/15 04:27:02 App: TestDeployUndeployLoopTimer got deployed
2021/12/15 04:27:02 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 04:27:02 Undeploying app: bucket_op_with_timer
2021/12/15 04:27:02 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 04:27:07 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/15 04:27:12 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/15 04:27:17 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/15 04:27:22 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/15 04:27:22 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/15 04:27:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 04:27:22 No eventing-con process running
2021/12/15 04:27:23 Context: Bucket flush, response code: 200 dump: 
2021/12/15 04:27:31 Context: Bucket flush, response code: 200 dump: 
2021/12/15 04:28:06 Deploying app: TestDeployUndeployLoopTimer. Item count src bucket: 0 dst bucket: 0
2021/12/15 04:28:07 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopTimer' in metakv",
  "warnings": null
 }
}
2021/12/15 04:28:07 Starting bucket ops to source bucket
2021/12/15 04:28:15 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 04:28:20 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 04:28:25 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 04:28:30 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 04:28:35 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 04:28:40 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 04:28:45 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 04:28:45 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/15 04:28:50 Waiting for app: TestDeployUndeployLoopTimer to get deployed
2021/12/15 04:28:50 App: TestDeployUndeployLoopTimer got deployed
2021/12/15 04:28:50 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 04:28:50 Undeploying app: bucket_op_with_timer
2021/12/15 04:28:50 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 04:28:55 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/15 04:29:00 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/15 04:29:05 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/15 04:29:10 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/15 04:29:10 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/15 04:29:10 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 04:29:10 No eventing-con process running
2021/12/15 04:29:12 Context: Bucket flush, response code: 200 dump: 
2021/12/15 04:29:19 Context: Bucket flush, response code: 200 dump: 
2021/12/15 04:29:54 Deploying app: TestDeployUndeployLoopTimer. Item count src bucket: 0 dst bucket: 0
2021/12/15 04:29:55 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopTimer' in metakv",
  "warnings": null
 }
}
2021/12/15 04:29:55 Starting bucket ops to source bucket
2021/12/15 04:30:03 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 04:30:08 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 04:30:13 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 04:30:18 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 04:30:23 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 04:30:28 Waiting for dst bucket item count to get to: 5000 curr count: 1787
2021/12/15 04:30:33 Waiting for dst bucket item count to get to: 5000 curr count: 1787
2021/12/15 04:30:33 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/15 04:30:38 Waiting for app: TestDeployUndeployLoopTimer to get deployed
2021/12/15 04:30:38 App: TestDeployUndeployLoopTimer got deployed
2021/12/15 04:30:38 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 04:30:38 Undeploying app: bucket_op_with_timer
2021/12/15 04:30:38 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 04:30:43 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/15 04:30:49 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/15 04:30:54 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/15 04:30:59 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/15 04:30:59 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/15 04:30:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 04:30:59 No eventing-con process running
2021/12/15 04:31:00 Context: Bucket flush, response code: 200 dump: 
2021/12/15 04:31:08 Context: Bucket flush, response code: 200 dump: 
2021/12/15 04:31:43 Deploying app: TestDeployUndeployLoopTimer. Item count src bucket: 0 dst bucket: 0
2021/12/15 04:31:44 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopTimer' in metakv",
  "warnings": null
 }
}
2021/12/15 04:31:44 Starting bucket ops to source bucket
2021/12/15 04:31:52 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 04:31:57 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 04:32:02 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 04:32:07 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 04:32:12 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 04:32:17 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 04:32:22 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 04:32:27 Waiting for dst bucket item count to get to: 5000 curr count: 2500
2021/12/15 04:32:27 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/15 04:32:32 Waiting for app: TestDeployUndeployLoopTimer to get deployed
2021/12/15 04:32:32 App: TestDeployUndeployLoopTimer got deployed
2021/12/15 04:32:32 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 04:32:32 Undeploying app: bucket_op_with_timer
2021/12/15 04:32:32 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 04:32:37 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/15 04:32:42 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/15 04:32:47 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/15 04:32:52 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/15 04:32:52 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/15 04:32:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 04:32:52 No eventing-con process running
2021/12/15 04:32:54 Context: Bucket flush, response code: 200 dump: 
2021/12/15 04:33:01 Context: Bucket flush, response code: 200 dump: 
2021/12/15 04:33:36 Function: TestDeployUndeployLoopTimer delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDeployUndeployLoopTimer deleting in the background"
}
--- PASS: TestDeployUndeployLoopTimer (580.76s)
=== RUN   TestMultipleHandlers
2021/12/15 04:33:46 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get un-deployed
2021/12/15 04:33:46 App: TestMultipleHandlers_bucket_op_on_update got un-deployed
2021/12/15 04:33:46 No eventing-con process running
2021/12/15 04:33:46 Function: TestMultipleHandlers_bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMultipleHandlers_bucket_op_on_update not found"
 }
}
2021/12/15 04:33:48 Context: Bucket flush, response code: 200 dump: 
2021/12/15 04:33:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 04:33:55 Context: Bucket flush, response code: 200 dump: 
2021/12/15 04:34:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 04:34:05 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get un-deployed
2021/12/15 04:34:05 App: TestMultipleHandlers_bucket_op_on_delete got un-deployed
2021/12/15 04:34:05 No eventing-con process running
2021/12/15 04:34:05 Function: TestMultipleHandlers_bucket_op_on_delete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMultipleHandlers_bucket_op_on_delete not found"
 }
}
2021/12/15 04:34:06 Context: Bucket flush, response code: 200 dump: 
2021/12/15 04:34:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 04:34:13 Context: Bucket flush, response code: 200 dump: 
2021/12/15 04:34:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 04:34:18 Deploying app: TestMultipleHandlers_bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/15 04:34:19 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMultipleHandlers_bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/15 04:34:19 Deploying app: TestMultipleHandlers_bucket_op_on_delete. Item count src bucket: 0 dst bucket: 0
2021/12/15 04:34:21 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMultipleHandlers_bucket_op_on_delete' in metakv",
  "warnings": null
 }
}
2021/12/15 04:34:26 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get deployed
2021/12/15 04:34:31 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get deployed
2021/12/15 04:34:36 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get deployed
2021/12/15 04:34:41 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get deployed
2021/12/15 04:34:41 App: TestMultipleHandlers_bucket_op_on_update got deployed
2021/12/15 04:34:46 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get deployed
2021/12/15 04:34:51 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get deployed
2021/12/15 04:34:56 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get deployed
2021/12/15 04:34:56 App: TestMultipleHandlers_bucket_op_on_delete got deployed
2021/12/15 04:34:56 Starting bucket ops to source bucket
2021/12/15 04:35:06 Waiting for dst bucket item count to get to: 5000 curr count: 680
2021/12/15 04:35:11 Waiting for dst bucket item count to get to: 5000 curr count: 2429
2021/12/15 04:35:16 Waiting for dst bucket item count to get to: 5000 curr count: 4249
2021/12/15 04:35:16 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/15 04:35:16 Starting bucket ops to source bucket
2021/12/15 04:35:31 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/15 04:35:36 Waiting for dst bucket item count to get to: 10000 curr count: 7504
2021/12/15 04:35:36 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/15 04:35:36 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 04:35:36 Function: TestMultipleHandlers_bucket_op_on_update update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/15 04:35:36 Function: TestMultipleHandlers_bucket_op_on_delete update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/15 04:35:36 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 04:35:41 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get un-deployed
2021/12/15 04:35:46 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get un-deployed
2021/12/15 04:35:52 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get un-deployed
2021/12/15 04:35:57 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get un-deployed
2021/12/15 04:35:57 App: TestMultipleHandlers_bucket_op_on_update got un-deployed
2021/12/15 04:35:57 No eventing-con process running
2021/12/15 04:35:57 Function: TestMultipleHandlers_bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMultipleHandlers_bucket_op_on_update deleting in the background"
}
2021/12/15 04:35:58 Context: Bucket flush, response code: 200 dump: 
2021/12/15 04:36:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 04:36:06 Context: Bucket flush, response code: 200 dump: 
2021/12/15 04:36:16 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/15 04:36:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 04:36:16 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 04:36:21 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get un-deployed
2021/12/15 04:36:26 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get un-deployed
2021/12/15 04:36:31 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get un-deployed
2021/12/15 04:36:36 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get un-deployed
2021/12/15 04:36:36 App: TestMultipleHandlers_bucket_op_on_delete got un-deployed
2021/12/15 04:36:36 No eventing-con process running
2021/12/15 04:36:36 Function: TestMultipleHandlers_bucket_op_on_delete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMultipleHandlers_bucket_op_on_delete deleting in the background"
}
2021/12/15 04:36:38 Context: Bucket flush, response code: 200 dump: 
2021/12/15 04:36:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 04:36:44 Context: Bucket flush, response code: 200 dump: 
2021/12/15 04:36:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestMultipleHandlers (193.49s)
=== RUN   TestPauseResumeLoopDefaultSettings
2021/12/15 04:36:59 Waiting for app: TestPauseResumeLoopDefaultSettings to get un-deployed
2021/12/15 04:36:59 App: TestPauseResumeLoopDefaultSettings got un-deployed
2021/12/15 04:37:00 No eventing-con process running
2021/12/15 04:37:00 Function: TestPauseResumeLoopDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestPauseResumeLoopDefaultSettings not found"
 }
}
2021/12/15 04:37:01 Context: Bucket flush, response code: 200 dump: 
2021/12/15 04:37:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 04:37:08 Context: Bucket flush, response code: 200 dump: 
2021/12/15 04:37:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 04:37:13 Deploying app: TestPauseResumeLoopDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/15 04:37:14 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestPauseResumeLoopDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/15 04:37:14 Starting bucket ops to source bucket
2021/12/15 04:37:22 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 04:37:27 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 04:37:32 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 04:37:37 Waiting for dst bucket item count to get to: 5000 curr count: 1714
2021/12/15 04:37:42 Waiting for dst bucket item count to get to: 5000 curr count: 4214
2021/12/15 04:37:42 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/15 04:37:42 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 04:37:47 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to deployed
2021/12/15 04:37:47 Function: TestPauseResumeLoopDefaultSettings status changed to deployed
2021/12/15 04:37:47 Pausing app: bucket_op_on_update
2021/12/15 04:37:47 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/15 04:37:52 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to paused
2021/12/15 04:37:52 Function: TestPauseResumeLoopDefaultSettings status changed to paused
2021/12/15 04:37:52 Resuming app: bucket_op_on_update
2021/12/15 04:37:54 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/15 04:37:54 Starting bucket ops to source bucket
2021/12/15 04:38:02 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/15 04:38:07 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/15 04:38:12 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/15 04:38:17 Waiting for dst bucket item count to get to: 10000 curr count: 7500
2021/12/15 04:38:17 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/15 04:38:17 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 04:38:22 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to deployed
2021/12/15 04:38:22 Function: TestPauseResumeLoopDefaultSettings status changed to deployed
2021/12/15 04:38:22 Pausing app: bucket_op_on_update
2021/12/15 04:38:22 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/15 04:38:27 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to paused
2021/12/15 04:38:27 Function: TestPauseResumeLoopDefaultSettings status changed to paused
2021/12/15 04:38:27 Resuming app: bucket_op_on_update
2021/12/15 04:38:28 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/15 04:38:28 Starting bucket ops to source bucket
2021/12/15 04:38:36 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/15 04:38:41 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/15 04:38:41 src & dst bucket item count matched up. src bucket count: 15000 dst bucket count: 15000
2021/12/15 04:38:41 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 04:38:46 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to deployed
2021/12/15 04:38:46 Function: TestPauseResumeLoopDefaultSettings status changed to deployed
2021/12/15 04:38:46 Pausing app: bucket_op_on_update
2021/12/15 04:38:46 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/15 04:38:51 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to paused
2021/12/15 04:38:51 Function: TestPauseResumeLoopDefaultSettings status changed to paused
2021/12/15 04:38:51 Resuming app: bucket_op_on_update
2021/12/15 04:38:52 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/15 04:38:52 Starting bucket ops to source bucket
2021/12/15 04:39:00 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/15 04:39:05 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/15 04:39:10 Waiting for dst bucket item count to get to: 20000 curr count: 15366
2021/12/15 04:39:15 Waiting for dst bucket item count to get to: 20000 curr count: 15366
2021/12/15 04:39:15 src & dst bucket item count matched up. src bucket count: 20000 dst bucket count: 20000
2021/12/15 04:39:15 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 04:39:20 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to deployed
2021/12/15 04:39:20 Function: TestPauseResumeLoopDefaultSettings status changed to deployed
2021/12/15 04:39:20 Pausing app: bucket_op_on_update
2021/12/15 04:39:20 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/15 04:39:25 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to paused
2021/12/15 04:39:25 Function: TestPauseResumeLoopDefaultSettings status changed to paused
2021/12/15 04:39:25 Resuming app: bucket_op_on_update
2021/12/15 04:39:27 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/15 04:39:27 Starting bucket ops to source bucket
2021/12/15 04:39:34 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/15 04:39:40 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/15 04:39:45 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/15 04:39:45 src & dst bucket item count matched up. src bucket count: 25000 dst bucket count: 25000
2021/12/15 04:39:45 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 04:39:50 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to deployed
2021/12/15 04:39:50 Function: TestPauseResumeLoopDefaultSettings status changed to deployed
2021/12/15 04:39:50 Pausing app: bucket_op_on_update
2021/12/15 04:39:50 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/15 04:39:55 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to paused
2021/12/15 04:39:55 Function: TestPauseResumeLoopDefaultSettings status changed to paused
2021/12/15 04:39:55 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 04:40:00 Waiting for app: TestPauseResumeLoopDefaultSettings to get un-deployed
2021/12/15 04:40:05 Waiting for app: TestPauseResumeLoopDefaultSettings to get un-deployed
2021/12/15 04:40:10 Waiting for app: TestPauseResumeLoopDefaultSettings to get un-deployed
2021/12/15 04:40:15 Waiting for app: TestPauseResumeLoopDefaultSettings to get un-deployed
2021/12/15 04:40:15 App: TestPauseResumeLoopDefaultSettings got un-deployed
2021/12/15 04:40:15 No eventing-con process running
2021/12/15 04:40:15 Function: TestPauseResumeLoopDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestPauseResumeLoopDefaultSettings deleting in the background"
}
2021/12/15 04:40:17 Context: Bucket flush, response code: 200 dump: 
2021/12/15 04:40:27 Waiting for dst bucket item count to get to: 0 curr count: 25000
2021/12/15 04:40:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 04:40:28 Context: Bucket flush, response code: 200 dump: 
2021/12/15 04:40:38 Waiting for dst bucket item count to get to: 0 curr count: 21394
2021/12/15 04:40:43 Waiting for dst bucket item count to get to: 0 curr count: 8894
2021/12/15 04:40:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestPauseResumeLoopDefaultSettings (233.94s)
=== RUN   TestPauseResumeLoopNonDefaultSettings
2021/12/15 04:40:53 Waiting for app: TestPauseResumeLoopNonDefaultSettings to get un-deployed
2021/12/15 04:40:53 App: TestPauseResumeLoopNonDefaultSettings got un-deployed
2021/12/15 04:40:53 No eventing-con process running
2021/12/15 04:40:53 Function: TestPauseResumeLoopNonDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestPauseResumeLoopNonDefaultSettings not found"
 }
}
2021/12/15 04:40:55 Context: Bucket flush, response code: 200 dump: 
2021/12/15 04:41:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 04:41:02 Context: Bucket flush, response code: 200 dump: 
2021/12/15 04:41:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 04:41:07 Deploying app: TestPauseResumeLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/15 04:41:08 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestPauseResumeLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/15 04:41:08 Starting bucket ops to source bucket
2021/12/15 04:41:16 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 04:41:21 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 04:41:26 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 04:41:31 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 04:41:36 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 04:41:41 Waiting for dst bucket item count to get to: 5000 curr count: 807
2021/12/15 04:41:46 Waiting for dst bucket item count to get to: 5000 curr count: 3307
2021/12/15 04:41:46 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/15 04:41:46 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 04:41:51 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to deployed
2021/12/15 04:41:51 Function: TestPauseResumeLoopNonDefaultSettings status changed to deployed
2021/12/15 04:41:51 Pausing app: bucket_op_with_timer
2021/12/15 04:41:51 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/15 04:41:56 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to paused
2021/12/15 04:41:56 Function: TestPauseResumeLoopNonDefaultSettings status changed to paused
2021/12/15 04:41:56 Resuming app: bucket_op_with_timer
2021/12/15 04:41:58 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/15 04:41:58 Starting bucket ops to source bucket
2021/12/15 04:42:06 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/15 04:42:11 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/15 04:42:16 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/15 04:42:21 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/15 04:42:26 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/15 04:42:31 Waiting for dst bucket item count to get to: 10000 curr count: 5683
2021/12/15 04:42:36 Waiting for dst bucket item count to get to: 10000 curr count: 8183
2021/12/15 04:42:36 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/15 04:42:36 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 04:42:41 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to deployed
2021/12/15 04:42:41 Function: TestPauseResumeLoopNonDefaultSettings status changed to deployed
2021/12/15 04:42:41 Pausing app: bucket_op_with_timer
2021/12/15 04:42:41 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/15 04:42:46 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to paused
2021/12/15 04:42:46 Function: TestPauseResumeLoopNonDefaultSettings status changed to paused
2021/12/15 04:42:46 Resuming app: bucket_op_with_timer
2021/12/15 04:42:47 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/15 04:42:47 Starting bucket ops to source bucket
2021/12/15 04:42:55 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/15 04:43:00 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/15 04:43:05 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/15 04:43:10 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/15 04:43:15 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/15 04:43:20 Waiting for dst bucket item count to get to: 15000 curr count: 11478
2021/12/15 04:43:25 Waiting for dst bucket item count to get to: 15000 curr count: 13978
2021/12/15 04:43:25 src & dst bucket item count matched up. src bucket count: 15000 dst bucket count: 15000
2021/12/15 04:43:25 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 04:43:30 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to deployed
2021/12/15 04:43:30 Function: TestPauseResumeLoopNonDefaultSettings status changed to deployed
2021/12/15 04:43:30 Pausing app: bucket_op_with_timer
2021/12/15 04:43:30 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/15 04:43:35 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to paused
2021/12/15 04:43:35 Function: TestPauseResumeLoopNonDefaultSettings status changed to paused
2021/12/15 04:43:35 Resuming app: bucket_op_with_timer
2021/12/15 04:43:36 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/15 04:43:36 Starting bucket ops to source bucket
2021/12/15 04:43:44 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/15 04:43:49 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/15 04:43:54 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/15 04:43:59 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/15 04:44:04 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/15 04:44:09 Waiting for dst bucket item count to get to: 20000 curr count: 17340
2021/12/15 04:44:14 Waiting for dst bucket item count to get to: 20000 curr count: 17340
2021/12/15 04:44:14 src & dst bucket item count matched up. src bucket count: 20000 dst bucket count: 20000
2021/12/15 04:44:14 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 04:44:19 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to deployed
2021/12/15 04:44:19 Function: TestPauseResumeLoopNonDefaultSettings status changed to deployed
2021/12/15 04:44:19 Pausing app: bucket_op_with_timer
2021/12/15 04:44:20 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/15 04:44:25 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to paused
2021/12/15 04:44:25 Function: TestPauseResumeLoopNonDefaultSettings status changed to paused
2021/12/15 04:44:25 Resuming app: bucket_op_with_timer
2021/12/15 04:44:26 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/15 04:44:26 Starting bucket ops to source bucket
2021/12/15 04:44:34 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/15 04:44:39 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/15 04:44:44 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/15 04:44:49 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/15 04:44:54 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/15 04:44:59 Waiting for dst bucket item count to get to: 25000 curr count: 22729
2021/12/15 04:45:04 Waiting for dst bucket item count to get to: 25000 curr count: 22729
2021/12/15 04:45:04 src & dst bucket item count matched up. src bucket count: 25000 dst bucket count: 25000
2021/12/15 04:45:04 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 04:45:09 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to deployed
2021/12/15 04:45:09 Function: TestPauseResumeLoopNonDefaultSettings status changed to deployed
2021/12/15 04:45:09 Pausing app: bucket_op_with_timer
2021/12/15 04:45:09 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/15 04:45:14 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to paused
2021/12/15 04:45:14 Function: TestPauseResumeLoopNonDefaultSettings status changed to paused
2021/12/15 04:45:14 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 04:45:19 Waiting for app: TestPauseResumeLoopNonDefaultSettings to get un-deployed
2021/12/15 04:45:24 Waiting for app: TestPauseResumeLoopNonDefaultSettings to get un-deployed
2021/12/15 04:45:29 Waiting for app: TestPauseResumeLoopNonDefaultSettings to get un-deployed
2021/12/15 04:45:34 Waiting for app: TestPauseResumeLoopNonDefaultSettings to get un-deployed
2021/12/15 04:45:34 App: TestPauseResumeLoopNonDefaultSettings got un-deployed
2021/12/15 04:45:34 No eventing-con process running
2021/12/15 04:45:34 Function: TestPauseResumeLoopNonDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestPauseResumeLoopNonDefaultSettings deleting in the background"
}
2021/12/15 04:45:36 Context: Bucket flush, response code: 200 dump: 
2021/12/15 04:45:46 Waiting for dst bucket item count to get to: 0 curr count: 25000
2021/12/15 04:45:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 04:45:48 Context: Bucket flush, response code: 200 dump: 
2021/12/15 04:45:58 Waiting for dst bucket item count to get to: 0 curr count: 14298
2021/12/15 04:46:03 Waiting for dst bucket item count to get to: 0 curr count: 1798
2021/12/15 04:46:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestPauseResumeLoopNonDefaultSettings (319.33s)
=== RUN   TestPauseAndResumeWithWorkerCountChange
2021/12/15 04:46:13 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get un-deployed
2021/12/15 04:46:13 App: TestPauseAndResumeWithWorkerCountChange got un-deployed
2021/12/15 04:46:13 No eventing-con process running
2021/12/15 04:46:13 Function: TestPauseAndResumeWithWorkerCountChange delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestPauseAndResumeWithWorkerCountChange not found"
 }
}
2021/12/15 04:46:15 Context: Bucket flush, response code: 200 dump: 
2021/12/15 04:46:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 04:46:21 Context: Bucket flush, response code: 200 dump: 
2021/12/15 04:46:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 04:46:26 Deploying app: TestPauseAndResumeWithWorkerCountChange. Item count src bucket: 0 dst bucket: 0
2021/12/15 04:46:27 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestPauseAndResumeWithWorkerCountChange' in metakv",
  "warnings": null
 }
}
2021/12/15 04:46:32 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get deployed
2021/12/15 04:46:38 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get deployed
2021/12/15 04:46:43 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get deployed
2021/12/15 04:46:48 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get deployed
2021/12/15 04:46:48 App: TestPauseAndResumeWithWorkerCountChange got deployed
2021/12/15 04:46:48 Starting bucket ops to source bucket
2021/12/15 04:46:57 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 04:47:02 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 04:47:07 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 04:47:12 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 04:47:17 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 04:47:22 Waiting for dst bucket item count to get to: 5000 curr count: 3656
2021/12/15 04:47:27 Waiting for dst bucket item count to get to: 5000 curr count: 3656
2021/12/15 04:47:27 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/15 04:47:27 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 04:47:27 Pausing app: bucket_op_with_timer
2021/12/15 04:47:27 Function: TestPauseAndResumeWithWorkerCountChange update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/15 04:47:32 Waiting for function: TestPauseAndResumeWithWorkerCountChange status to change to paused
2021/12/15 04:47:32 Function: TestPauseAndResumeWithWorkerCountChange status changed to paused
2021/12/15 04:47:32 Resuming app: bucket_op_with_timer
2021/12/15 04:47:33 Function: TestPauseAndResumeWithWorkerCountChange update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:6] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/15 04:47:33 Starting bucket ops to source bucket
2021/12/15 04:47:45 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/15 04:47:50 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/15 04:47:55 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/15 04:48:00 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/15 04:48:05 Waiting for dst bucket item count to get to: 10000 curr count: 6706
2021/12/15 04:48:10 Waiting for dst bucket item count to get to: 10000 curr count: 9206
2021/12/15 04:48:10 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/15 04:48:10 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 04:48:15 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get un-deployed
2021/12/15 04:48:20 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get un-deployed
2021/12/15 04:48:25 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get un-deployed
2021/12/15 04:48:30 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get un-deployed
2021/12/15 04:48:30 App: TestPauseAndResumeWithWorkerCountChange got un-deployed
2021/12/15 04:48:30 No eventing-con process running
2021/12/15 04:48:30 Function: TestPauseAndResumeWithWorkerCountChange delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestPauseAndResumeWithWorkerCountChange deleting in the background"
}
2021/12/15 04:48:32 Context: Bucket flush, response code: 200 dump: 
2021/12/15 04:48:42 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/15 04:48:47 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/15 04:48:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 04:48:48 Context: Bucket flush, response code: 200 dump: 
2021/12/15 04:48:58 Waiting for dst bucket item count to get to: 0 curr count: 8196
2021/12/15 04:49:03 Waiting for dst bucket item count to get to: 0 curr count: 3196
2021/12/15 04:49:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestPauseAndResumeWithWorkerCountChange (180.66s)
=== RUN   TestPauseResumeWithEventingReb
2021/12/15 04:49:13 Waiting for app: TestPauseResumeWithEventingReb to get un-deployed
2021/12/15 04:49:13 App: TestPauseResumeWithEventingReb got un-deployed
2021/12/15 04:49:13 No eventing-con process running
2021/12/15 04:49:13 Function: TestPauseResumeWithEventingReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestPauseResumeWithEventingReb not found"
 }
}
2021/12/15 04:49:15 Context: Bucket flush, response code: 200 dump: 
2021/12/15 04:49:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 04:49:22 Context: Bucket flush, response code: 200 dump: 
2021/12/15 04:49:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 04:49:27 Deploying app: TestPauseResumeWithEventingReb. Item count src bucket: 0 dst bucket: 0
2021/12/15 04:49:28 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestPauseResumeWithEventingReb' in metakv",
  "warnings": null
 }
}
2021/12/15 04:49:33 Waiting for app: TestPauseResumeWithEventingReb to get deployed
2021/12/15 04:49:38 Waiting for app: TestPauseResumeWithEventingReb to get deployed
2021/12/15 04:49:43 Waiting for app: TestPauseResumeWithEventingReb to get deployed
2021/12/15 04:49:48 Waiting for app: TestPauseResumeWithEventingReb to get deployed
2021/12/15 04:49:48 App: TestPauseResumeWithEventingReb got deployed
2021/12/15 04:49:48 Starting bucket ops to source bucket
2021/12/15 04:49:58 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 04:50:03 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 04:50:08 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 04:50:13 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 04:50:18 Waiting for dst bucket item count to get to: 5000 curr count: 1830
2021/12/15 04:50:23 Waiting for dst bucket item count to get to: 5000 curr count: 4330
2021/12/15 04:50:23 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/15 04:50:23 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 04:50:23 Pausing app: bucket_op_with_timer
2021/12/15 04:50:23 Function: TestPauseResumeWithEventingReb update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/15 04:50:28 Waiting for function: TestPauseResumeWithEventingReb status to change to paused
2021/12/15 04:50:28 Function: TestPauseResumeWithEventingReb status changed to paused
2021/12/15 04:50:28 Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/15 04:50:28 Removing node(s): [] from the cluster
2021/12/15 04:50:33 Rebalance progress: 29.16666666666666
2021/12/15 04:50:38 Rebalance progress: 29.16666666666666
2021/12/15 04:50:43 Rebalance progress: 100
2021/12/15 04:50:43 Resuming app: bucket_op_with_timer
2021/12/15 04:50:44 Function: TestPauseResumeWithEventingReb update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:6] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/15 04:50:44 Starting bucket ops to source bucket
2021/12/15 04:50:56 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/15 04:51:01 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/15 04:51:06 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/15 04:51:11 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/15 04:51:16 Waiting for dst bucket item count to get to: 10000 curr count: 5956
2021/12/15 04:51:21 Waiting for dst bucket item count to get to: 10000 curr count: 8036
2021/12/15 04:51:26 Waiting for dst bucket item count to get to: 10000 curr count: 9580
2021/12/15 04:51:26 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/15 04:51:26 Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/15 04:51:31 Rebalance progress: 29.16666666666666
2021/12/15 04:51:36 Rebalance progress: 29.16666666666666
2021/12/15 04:51:41 Rebalance progress: 100

VB distribution across eventing nodes and their workers::

2021/12/15 04:51:46 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:79} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/15 04:52:01 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:79} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/15 04:52:16 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:79} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/15 04:52:31 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:79} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/15 04:52:46 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:79} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/15 04:53:01 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:79} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/15 04:53:16 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:79} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/15 04:53:31 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:79} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/15 04:53:46 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:79} {Name:#primary Bucket:hello-world Completion:100}]

DCP Stream statuses:
	stream status: running
	len: 1024
	vb list dump: [0-1023]

vbucket curr owner:
Producer node: 127.0.0.1:9302	Node UUID: 442585dbdcccd496fd97d0ac70022334
	workerID: worker_TestPauseResumeWithEventingReb_0
	len: 57
	v: [0-56]
	workerID: worker_TestPauseResumeWithEventingReb_1
	len: 57
	v: [57-113]
	workerID: worker_TestPauseResumeWithEventingReb_2
	len: 57
	v: [114-170]
	workerID: worker_TestPauseResumeWithEventingReb_3
	len: 57
	v: [171-227]
	workerID: worker_TestPauseResumeWithEventingReb_4
	len: 57
	v: [228-284]
	workerID: worker_TestPauseResumeWithEventingReb_5
	len: 57
	v: [285-341]
Producer node: 172.31.2.134:9300	Node UUID: bbbe093dbac20cc0f8c45b8df77d5e5a
	workerID: worker_TestPauseResumeWithEventingReb_5
	len: 56
	v: [968-1023]
	workerID: worker_TestPauseResumeWithEventingReb_0
	len: 57
	v: [683-739]
	workerID: worker_TestPauseResumeWithEventingReb_1
	len: 57
	v: [740-796]
	workerID: worker_TestPauseResumeWithEventingReb_2
	len: 57
	v: [797-853]
	workerID: worker_TestPauseResumeWithEventingReb_3
	len: 57
	v: [854-910]
	workerID: worker_TestPauseResumeWithEventingReb_4
	len: 57
	v: [911-967]
Producer node: 127.0.0.1:9303	Node UUID: e26e57bc28e5e3bcf77b18125cbe7409
	workerID: worker_TestPauseResumeWithEventingReb_5
	len: 56
	v: [627-682]
	workerID: worker_TestPauseResumeWithEventingReb_0
	len: 57
	v: [342-398]
	workerID: worker_TestPauseResumeWithEventingReb_1
	len: 57
	v: [399-455]
	workerID: worker_TestPauseResumeWithEventingReb_2
	len: 57
	v: [456-512]
	workerID: worker_TestPauseResumeWithEventingReb_3
	len: 57
	v: [513-569]
	workerID: worker_TestPauseResumeWithEventingReb_4
	len: 57
	v: [570-626]


2021/12/15 04:54:02 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 04:54:02 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 04:54:07 clusterSetup.go:397: Waiting for app: TestPauseResumeWithEventingReb to get un-deployed
2021/12/15 04:54:12 clusterSetup.go:397: Waiting for app: TestPauseResumeWithEventingReb to get un-deployed
2021/12/15 04:54:17 clusterSetup.go:397: Waiting for app: TestPauseResumeWithEventingReb to get un-deployed
2021/12/15 04:54:22 clusterSetup.go:397: Waiting for app: TestPauseResumeWithEventingReb to get un-deployed
2021/12/15 04:54:22 clusterSetup.go:405: App: TestPauseResumeWithEventingReb got un-deployed
2021/12/15 04:54:22 eventingSetup.go:46: No eventing-con process running
2021/12/15 04:54:22 eventingSetup.go:418: Function: TestPauseResumeWithEventingReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestPauseResumeWithEventingReb deleting in the background"
}
2021/12/15 04:54:24 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 04:54:34 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/15 04:54:39 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/15 04:54:39 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 04:54:40 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 04:54:50 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 8122
2021/12/15 04:54:55 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 8122
2021/12/15 04:54:55 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestPauseResumeWithEventingReb (352.04s)
=== RUN   TestBucketDeleteAfterPause
2021/12/15 04:55:05 clusterSetup.go:397: Waiting for app: TestBucketDeleteAfterPause to get un-deployed
2021/12/15 04:55:05 clusterSetup.go:405: App: TestBucketDeleteAfterPause got un-deployed
2021/12/15 04:55:05 eventingSetup.go:46: No eventing-con process running
2021/12/15 04:55:05 eventingSetup.go:418: Function: TestBucketDeleteAfterPause delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestBucketDeleteAfterPause not found"
 }
}
2021/12/15 04:55:07 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 04:55:12 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 04:55:14 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 04:55:19 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 04:55:19 eventingSetup.go:125: Deploying app: TestBucketDeleteAfterPause. Item count src bucket: 0 dst bucket: 0
2021/12/15 04:55:20 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestBucketDeleteAfterPause' in metakv",
  "warnings": null
 }
}
2021/12/15 04:55:25 clusterSetup.go:346: Waiting for app: TestBucketDeleteAfterPause to get deployed
2021/12/15 04:55:30 clusterSetup.go:346: Waiting for app: TestBucketDeleteAfterPause to get deployed
2021/12/15 04:55:35 clusterSetup.go:346: Waiting for app: TestBucketDeleteAfterPause to get deployed
2021/12/15 04:55:40 clusterSetup.go:346: Waiting for app: TestBucketDeleteAfterPause to get deployed
2021/12/15 04:55:40 clusterSetup.go:354: App: TestBucketDeleteAfterPause got deployed
2021/12/15 04:55:40 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/15 04:55:50 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 04:55:55 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 04:55:55 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/15 04:55:55 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 04:55:55 handler_test.go:712: Pausing app: bucket_op_on_update
2021/12/15 04:55:55 eventingSetup.go:270: Function: TestBucketDeleteAfterPause update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/15 04:56:00 eventingSetup.go:719: Waiting for function: TestBucketDeleteAfterPause status to change to paused
2021/12/15 04:56:00 eventingSetup.go:727: Function: TestBucketDeleteAfterPause status changed to paused
2021/12/15 04:56:00 handler_test.go:716: Deleting source bucket: default
2021/12/15 04:56:06 eventingSetup.go:719: Waiting for function: TestBucketDeleteAfterPause status to change to undeployed
2021/12/15 04:56:06 eventingSetup.go:727: Function: TestBucketDeleteAfterPause status changed to undeployed
2021/12/15 04:56:11 clusterSetup.go:397: Waiting for app: TestBucketDeleteAfterPause to get un-deployed
2021/12/15 04:56:11 clusterSetup.go:405: App: TestBucketDeleteAfterPause got un-deployed
2021/12/15 04:56:11 eventingSetup.go:46: No eventing-con process running
2021/12/15 04:56:11 eventingSetup.go:418: Function: TestBucketDeleteAfterPause delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestBucketDeleteAfterPause deleting in the background"
}
2021/12/15 04:56:13 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 04:56:18 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 04:56:20 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 04:56:30 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/15 04:56:35 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/15 04:56:35 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestBucketDeleteAfterPause (99.13s)
=== RUN   TestChangeFnCodeBetweenPauseResume
2021/12/15 04:56:45 clusterSetup.go:397: Waiting for app: TestChangeFnCodeBetweenPauseResume to get un-deployed
2021/12/15 04:56:45 clusterSetup.go:405: App: TestChangeFnCodeBetweenPauseResume got un-deployed
2021/12/15 04:56:45 eventingSetup.go:46: No eventing-con process running
2021/12/15 04:56:45 eventingSetup.go:418: Function: TestChangeFnCodeBetweenPauseResume delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestChangeFnCodeBetweenPauseResume not found"
 }
}
2021/12/15 04:56:46 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 04:56:51 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 04:56:53 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 04:56:58 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 04:56:58 eventingSetup.go:125: Deploying app: TestChangeFnCodeBetweenPauseResume. Item count src bucket: 0 dst bucket: 0
2021/12/15 04:56:59 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestChangeFnCodeBetweenPauseResume' in metakv",
  "warnings": null
 }
}
2021/12/15 04:57:04 clusterSetup.go:346: Waiting for app: TestChangeFnCodeBetweenPauseResume to get deployed
2021/12/15 04:57:09 clusterSetup.go:346: Waiting for app: TestChangeFnCodeBetweenPauseResume to get deployed
2021/12/15 04:57:14 clusterSetup.go:346: Waiting for app: TestChangeFnCodeBetweenPauseResume to get deployed
2021/12/15 04:57:19 clusterSetup.go:346: Waiting for app: TestChangeFnCodeBetweenPauseResume to get deployed
2021/12/15 04:57:19 clusterSetup.go:354: App: TestChangeFnCodeBetweenPauseResume got deployed
2021/12/15 04:57:19 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/15 04:57:54 handler_test.go:739: Pausing function: TestChangeFnCodeBetweenPauseResume
2021/12/15 04:57:54 eventingSetup.go:270: Function: TestChangeFnCodeBetweenPauseResume update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/15 04:57:59 eventingSetup.go:719: Waiting for function: TestChangeFnCodeBetweenPauseResume status to change to paused
2021/12/15 04:57:59 eventingSetup.go:727: Function: TestChangeFnCodeBetweenPauseResume status changed to paused
2021/12/15 05:00:59 handler_test.go:748: Resuming function: TestChangeFnCodeBetweenPauseResume
2021/12/15 05:00:59 eventingSetup.go:125: Deploying app: TestChangeFnCodeBetweenPauseResume. Item count src bucket: 5000 dst bucket: 0
2021/12/15 05:01:00 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestChangeFnCodeBetweenPauseResume' in metakv",
  "warnings": null
 }
}
2021/12/15 05:01:05 eventingSetup.go:795: Waiting for function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter to get to 5000
2021/12/15 05:01:05 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 05:01:05 eventingSetup.go:799: Function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter got to 0 expected 5000
2021/12/15 05:01:10 eventingSetup.go:795: Waiting for function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter to get to 5000
2021/12/15 05:01:10 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 05:01:10 eventingSetup.go:799: Function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter got to 2920 expected 5000
2021/12/15 05:01:15 eventingSetup.go:795: Waiting for function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter to get to 5000
2021/12/15 05:01:15 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 05:01:15 eventingSetup.go:799: Function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter got to 3942 expected 5000
2021/12/15 05:01:20 eventingSetup.go:795: Waiting for function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter to get to 5000
2021/12/15 05:01:20 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 05:01:20 eventingSetup.go:804: Function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter got to 5000
2021/12/15 05:01:20 handler_test.go:753: Undeploying function: TestChangeFnCodeBetweenPauseResume
2021/12/15 05:01:20 eventingSetup.go:270: Function: TestChangeFnCodeBetweenPauseResume update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/15 05:01:30 clusterSetup.go:397: Waiting for app: TestChangeFnCodeBetweenPauseResume to get un-deployed
2021/12/15 05:01:35 clusterSetup.go:397: Waiting for app: TestChangeFnCodeBetweenPauseResume to get un-deployed
2021/12/15 05:01:40 clusterSetup.go:397: Waiting for app: TestChangeFnCodeBetweenPauseResume to get un-deployed
2021/12/15 05:01:40 clusterSetup.go:405: App: TestChangeFnCodeBetweenPauseResume got un-deployed
2021/12/15 05:01:40 eventingSetup.go:46: No eventing-con process running
2021/12/15 05:01:40 eventingSetup.go:418: Function: TestChangeFnCodeBetweenPauseResume delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestChangeFnCodeBetweenPauseResume deleting in the background"
}
2021/12/15 05:01:42 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 05:01:52 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/15 05:01:57 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/15 05:01:57 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 05:01:58 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 05:02:04 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestChangeFnCodeBetweenPauseResume (328.94s)
=== RUN   TestCommentUnCommentOnDelete
2021/12/15 05:02:14 clusterSetup.go:397: Waiting for app: TestCommentUnCommentOnDelete to get un-deployed
2021/12/15 05:02:14 clusterSetup.go:405: App: TestCommentUnCommentOnDelete got un-deployed
2021/12/15 05:02:14 eventingSetup.go:46: No eventing-con process running
2021/12/15 05:02:14 eventingSetup.go:418: Function: TestCommentUnCommentOnDelete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestCommentUnCommentOnDelete not found"
 }
}
2021/12/15 05:02:15 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 05:02:20 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 05:02:22 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 05:02:27 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 05:02:27 eventingSetup.go:125: Deploying app: TestCommentUnCommentOnDelete. Item count src bucket: 0 dst bucket: 0
2021/12/15 05:02:28 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCommentUnCommentOnDelete' in metakv",
  "warnings": null
 }
}
2021/12/15 05:02:33 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/15 05:02:38 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/15 05:02:43 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/15 05:02:48 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/15 05:02:48 clusterSetup.go:354: App: TestCommentUnCommentOnDelete got deployed
2021/12/15 05:02:48 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/15 05:02:57 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 05:03:02 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 877
2021/12/15 05:03:07 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 877
2021/12/15 05:03:07 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/15 05:03:07 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 05:03:07 handler_test.go:780: Undeploying app: TestCommentUnCommentOnDelete
2021/12/15 05:03:07 eventingSetup.go:270: Function: TestCommentUnCommentOnDelete update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/15 05:03:37 eventingSetup.go:125: Deploying app: TestCommentUnCommentOnDelete. Item count src bucket: 5000 dst bucket: 5000
2021/12/15 05:03:38 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCommentUnCommentOnDelete' in metakv",
  "warnings": null
 }
}
2021/12/15 05:03:43 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/15 05:03:48 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/15 05:03:53 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/15 05:03:58 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/15 05:03:58 clusterSetup.go:354: App: TestCommentUnCommentOnDelete got deployed
2021/12/15 05:03:58 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/15 05:04:11 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/15 05:04:16 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/15 05:04:16 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 05:04:16 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 05:04:16 handler_test.go:799: Undeploying app: TestCommentUnCommentOnDelete
2021/12/15 05:04:16 eventingSetup.go:270: Function: TestCommentUnCommentOnDelete update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/15 05:04:26 clusterSetup.go:397: Waiting for app: TestCommentUnCommentOnDelete to get un-deployed
2021/12/15 05:04:31 clusterSetup.go:397: Waiting for app: TestCommentUnCommentOnDelete to get un-deployed
2021/12/15 05:04:36 clusterSetup.go:397: Waiting for app: TestCommentUnCommentOnDelete to get un-deployed
2021/12/15 05:04:36 clusterSetup.go:405: App: TestCommentUnCommentOnDelete got un-deployed
2021/12/15 05:04:36 eventingSetup.go:46: No eventing-con process running
2021/12/15 05:04:36 eventingSetup.go:418: Function: TestCommentUnCommentOnDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCommentUnCommentOnDelete deleting in the background"
}
2021/12/15 05:04:37 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 05:04:42 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 05:04:44 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 05:04:49 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCommentUnCommentOnDelete (165.74s)
=== RUN   TestCPPWorkerCleanup
2021/12/15 05:04:59 clusterSetup.go:397: Waiting for app: TestCPPWorkerCleanup to get un-deployed
2021/12/15 05:04:59 clusterSetup.go:405: App: TestCPPWorkerCleanup got un-deployed
2021/12/15 05:04:59 eventingSetup.go:46: No eventing-con process running
2021/12/15 05:04:59 eventingSetup.go:418: Function: TestCPPWorkerCleanup delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestCPPWorkerCleanup not found"
 }
}
2021/12/15 05:05:01 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 05:05:06 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 05:05:07 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 05:05:12 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 05:05:12 eventingSetup.go:125: Deploying app: TestCPPWorkerCleanup. Item count src bucket: 0 dst bucket: 0
2021/12/15 05:05:14 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCPPWorkerCleanup' in metakv",
  "warnings": null
 }
}
2021/12/15 05:05:14 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/15 05:05:21 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 05:05:26 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 05:05:31 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 05:05:36 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 24
2021/12/15 05:05:41 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 1695
2021/12/15 05:05:46 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 4171
2021/12/15 05:05:46 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/15 05:05:46 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 05:05:46 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 05:05:51 clusterSetup.go:397: Waiting for app: TestCPPWorkerCleanup to get un-deployed
2021/12/15 05:05:56 clusterSetup.go:397: Waiting for app: TestCPPWorkerCleanup to get un-deployed
2021/12/15 05:06:01 clusterSetup.go:397: Waiting for app: TestCPPWorkerCleanup to get un-deployed
2021/12/15 05:06:06 clusterSetup.go:397: Waiting for app: TestCPPWorkerCleanup to get un-deployed
2021/12/15 05:06:06 clusterSetup.go:405: App: TestCPPWorkerCleanup got un-deployed
2021/12/15 05:06:06 eventingSetup.go:46: No eventing-con process running
2021/12/15 05:06:06 eventingSetup.go:418: Function: TestCPPWorkerCleanup delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCPPWorkerCleanup deleting in the background"
}
2021/12/15 05:06:07 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 05:06:17 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/15 05:06:17 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 05:06:19 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 05:06:29 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/15 05:06:34 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/15 05:06:34 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCPPWorkerCleanup (134.92s)
=== RUN   TestWithUserXattrs
2021/12/15 05:07:14 clusterSetup.go:397: Waiting for app: TestWithUserXattrs to get un-deployed
2021/12/15 05:07:14 clusterSetup.go:405: App: TestWithUserXattrs got un-deployed
2021/12/15 05:07:14 eventingSetup.go:46: No eventing-con process running
2021/12/15 05:07:14 eventingSetup.go:418: Function: TestWithUserXattrs delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestWithUserXattrs not found"
 }
}
2021/12/15 05:07:15 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 05:07:20 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 05:07:22 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 05:07:27 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 05:07:27 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/15 05:07:29 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/15 05:07:31 eventingSetup.go:125: Deploying app: TestWithUserXattrs. Item count src bucket: 0 dst bucket: 0
2021/12/15 05:07:33 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestWithUserXattrs' in metakv",
  "warnings": null
 }
}
2021/12/15 05:07:38 clusterSetup.go:346: Waiting for app: TestWithUserXattrs to get deployed
2021/12/15 05:07:43 clusterSetup.go:346: Waiting for app: TestWithUserXattrs to get deployed
2021/12/15 05:07:48 clusterSetup.go:346: Waiting for app: TestWithUserXattrs to get deployed
2021/12/15 05:07:53 clusterSetup.go:346: Waiting for app: TestWithUserXattrs to get deployed
2021/12/15 05:07:53 clusterSetup.go:354: App: TestWithUserXattrs got deployed
2021/12/15 05:07:53 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/15 05:08:02 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 05:08:07 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 2500
2021/12/15 05:08:07 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/15 05:08:07 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 05:08:07 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 05:08:12 clusterSetup.go:397: Waiting for app: TestWithUserXattrs to get un-deployed
2021/12/15 05:08:17 clusterSetup.go:397: Waiting for app: TestWithUserXattrs to get un-deployed
2021/12/15 05:08:22 clusterSetup.go:397: Waiting for app: TestWithUserXattrs to get un-deployed
2021/12/15 05:08:27 clusterSetup.go:397: Waiting for app: TestWithUserXattrs to get un-deployed
2021/12/15 05:08:27 clusterSetup.go:405: App: TestWithUserXattrs got un-deployed
2021/12/15 05:08:27 eventingSetup.go:46: No eventing-con process running
2021/12/15 05:08:27 eventingSetup.go:418: Function: TestWithUserXattrs delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestWithUserXattrs deleting in the background"
}
2021/12/15 05:08:28 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 05:08:38 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/15 05:08:43 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/15 05:08:43 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 05:08:45 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 05:08:55 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/15 05:08:55 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestWithUserXattrs (111.06s)
=== RUN   TestEventProcessingPostBucketFlush
2021/12/15 05:09:05 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/15 05:09:05 clusterSetup.go:405: App: TestEventProcessingPostBucketFlush got un-deployed
2021/12/15 05:09:05 eventingSetup.go:46: No eventing-con process running
2021/12/15 05:09:05 eventingSetup.go:418: Function: TestEventProcessingPostBucketFlush delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventProcessingPostBucketFlush not found"
 }
}
2021/12/15 05:09:07 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 05:09:12 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 05:09:13 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 05:09:18 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 05:09:18 eventingSetup.go:125: Deploying app: TestEventProcessingPostBucketFlush. Item count src bucket: 0 dst bucket: 0
2021/12/15 05:09:19 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventProcessingPostBucketFlush' in metakv",
  "warnings": null
 }
}
2021/12/15 05:09:24 clusterSetup.go:346: Waiting for app: TestEventProcessingPostBucketFlush to get deployed
2021/12/15 05:09:29 clusterSetup.go:346: Waiting for app: TestEventProcessingPostBucketFlush to get deployed
2021/12/15 05:09:34 clusterSetup.go:346: Waiting for app: TestEventProcessingPostBucketFlush to get deployed
2021/12/15 05:09:39 clusterSetup.go:346: Waiting for app: TestEventProcessingPostBucketFlush to get deployed
2021/12/15 05:09:39 clusterSetup.go:354: App: TestEventProcessingPostBucketFlush got deployed
2021/12/15 05:09:39 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/15 05:09:49 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 05:09:54 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 05:09:59 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 2500
2021/12/15 05:09:59 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/15 05:09:59 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 05:10:03 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 05:10:13 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4474
2021/12/15 05:10:18 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1974
2021/12/15 05:10:18 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 05:10:20 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 05:10:30 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2514
2021/12/15 05:10:35 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2514
2021/12/15 05:10:35 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 05:10:35 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/15 05:11:00 eventingSetup.go:441: Waiting for dst bucket item count to get to: 25000 curr count: 16963
2021/12/15 05:11:05 eventingSetup.go:441: Waiting for dst bucket item count to get to: 25000 curr count: 16963
2021/12/15 05:11:05 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 25000 dst bucket count: 25000
2021/12/15 05:11:05 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 05:11:05 eventingSetup.go:270: Function: TestEventProcessingPostBucketFlush update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/15 05:11:10 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/15 05:11:15 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/15 05:11:20 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/15 05:11:25 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/15 05:11:25 clusterSetup.go:405: App: TestEventProcessingPostBucketFlush got un-deployed
2021/12/15 05:11:25 eventingSetup.go:46: No eventing-con process running
2021/12/15 05:11:30 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/15 05:11:30 clusterSetup.go:405: App: TestEventProcessingPostBucketFlush got un-deployed
2021/12/15 05:11:30 eventingSetup.go:46: No eventing-con process running
2021/12/15 05:11:30 eventingSetup.go:418: Function: TestEventProcessingPostBucketFlush delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventProcessingPostBucketFlush deleting in the background"
}
2021/12/15 05:11:31 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 05:11:41 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 25000
2021/12/15 05:11:46 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 25000
2021/12/15 05:11:46 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 05:11:48 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 05:11:58 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 16973
2021/12/15 05:12:03 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4473
2021/12/15 05:12:03 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventProcessingPostBucketFlush (187.72s)
=== RUN   TestMetaBucketDelete
2021/12/15 05:12:13 clusterSetup.go:397: Waiting for app: TestMetaBucketDelete to get un-deployed
2021/12/15 05:12:13 clusterSetup.go:405: App: TestMetaBucketDelete got un-deployed
2021/12/15 05:12:13 eventingSetup.go:46: No eventing-con process running
2021/12/15 05:12:13 eventingSetup.go:418: Function: TestMetaBucketDelete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMetaBucketDelete not found"
 }
}
2021/12/15 05:12:14 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 05:12:19 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 05:12:21 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 05:12:26 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 05:12:26 eventingSetup.go:125: Deploying app: TestMetaBucketDelete. Item count src bucket: 0 dst bucket: 0
2021/12/15 05:12:27 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMetaBucketDelete' in metakv",
  "warnings": null
 }
}
2021/12/15 05:12:32 clusterSetup.go:346: Waiting for app: TestMetaBucketDelete to get deployed
2021/12/15 05:12:37 clusterSetup.go:346: Waiting for app: TestMetaBucketDelete to get deployed
2021/12/15 05:12:42 clusterSetup.go:346: Waiting for app: TestMetaBucketDelete to get deployed
2021/12/15 05:12:47 clusterSetup.go:346: Waiting for app: TestMetaBucketDelete to get deployed
2021/12/15 05:12:47 clusterSetup.go:354: App: TestMetaBucketDelete got deployed
2021/12/15 05:12:47 handler_test.go:902: Deleting metadata bucket: eventing
2021/12/15 05:12:48 handler_test.go:904: Deleted metadata bucket: eventing
2021/12/15 05:12:53 clusterSetup.go:397: Waiting for app: TestMetaBucketDelete to get un-deployed
2021/12/15 05:12:53 clusterSetup.go:405: App: TestMetaBucketDelete got un-deployed
2021/12/15 05:13:08 clusterSetup.go:397: Waiting for app: TestMetaBucketDelete to get un-deployed
2021/12/15 05:13:08 clusterSetup.go:405: App: TestMetaBucketDelete got un-deployed
2021/12/15 05:13:08 eventingSetup.go:46: No eventing-con process running
2021/12/15 05:13:08 eventingSetup.go:418: Function: TestMetaBucketDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMetaBucketDelete deleting in the background"
}
2021/12/15 05:13:10 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 05:13:15 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 05:13:16 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 05:13:22 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestMetaBucketDelete (78.56s)
=== RUN   TestMetaBucketDeleteWithBootstrap
2021/12/15 05:13:32 clusterSetup.go:397: Waiting for app: TestMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/15 05:13:32 clusterSetup.go:405: App: TestMetaBucketDeleteWithBootstrap got un-deployed
2021/12/15 05:13:32 eventingSetup.go:46: No eventing-con process running
2021/12/15 05:13:32 eventingSetup.go:418: Function: TestMetaBucketDeleteWithBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMetaBucketDeleteWithBootstrap not found"
 }
}
2021/12/15 05:13:33 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 05:13:38 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 05:13:40 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 05:13:45 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 05:13:45 eventingSetup.go:125: Deploying app: TestMetaBucketDeleteWithBootstrap. Item count src bucket: 0 dst bucket: 0
2021/12/15 05:13:46 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMetaBucketDeleteWithBootstrap' in metakv",
  "warnings": null
 }
}
2021/12/15 05:14:05 handler_test.go:923: Deleting metadata bucket: eventing
2021/12/15 05:14:05 handler_test.go:925: Deleted metadata bucket: eventing
2021/12/15 05:14:20 clusterSetup.go:397: Waiting for app: TestMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/15 05:14:20 clusterSetup.go:405: App: TestMetaBucketDeleteWithBootstrap got un-deployed
2021/12/15 05:14:35 clusterSetup.go:397: Waiting for app: TestMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/15 05:14:35 clusterSetup.go:405: App: TestMetaBucketDeleteWithBootstrap got un-deployed
2021/12/15 05:14:35 eventingSetup.go:46: No eventing-con process running
2021/12/15 05:14:36 eventingSetup.go:418: Function: TestMetaBucketDeleteWithBootstrap delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMetaBucketDeleteWithBootstrap deleting in the background"
}
2021/12/15 05:14:37 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 05:14:42 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 05:14:44 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 05:14:49 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestMetaBucketDeleteWithBootstrap (87.14s)
=== RUN   TestSourceBucketDelete
2021/12/15 05:14:59 clusterSetup.go:397: Waiting for app: TestSourceBucketDelete to get un-deployed
2021/12/15 05:14:59 clusterSetup.go:405: App: TestSourceBucketDelete got un-deployed
2021/12/15 05:14:59 eventingSetup.go:46: No eventing-con process running
2021/12/15 05:14:59 eventingSetup.go:418: Function: TestSourceBucketDelete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestSourceBucketDelete not found"
 }
}
2021/12/15 05:15:00 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 05:15:05 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 05:15:07 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 05:15:12 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 05:15:12 eventingSetup.go:125: Deploying app: TestSourceBucketDelete. Item count src bucket: 0 dst bucket: 0
2021/12/15 05:15:13 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestSourceBucketDelete' in metakv",
  "warnings": null
 }
}
2021/12/15 05:15:18 clusterSetup.go:346: Waiting for app: TestSourceBucketDelete to get deployed
2021/12/15 05:15:23 clusterSetup.go:346: Waiting for app: TestSourceBucketDelete to get deployed
2021/12/15 05:15:28 clusterSetup.go:346: Waiting for app: TestSourceBucketDelete to get deployed
2021/12/15 05:15:33 clusterSetup.go:346: Waiting for app: TestSourceBucketDelete to get deployed
2021/12/15 05:15:33 clusterSetup.go:354: App: TestSourceBucketDelete got deployed
2021/12/15 05:15:33 handler_test.go:945: Deleting source bucket: default
2021/12/15 05:15:34 handler_test.go:947: Deleted source bucket: default
2021/12/15 05:15:39 clusterSetup.go:397: Waiting for app: TestSourceBucketDelete to get un-deployed
2021/12/15 05:15:39 clusterSetup.go:405: App: TestSourceBucketDelete got un-deployed
2021/12/15 05:15:55 clusterSetup.go:397: Waiting for app: TestSourceBucketDelete to get un-deployed
2021/12/15 05:15:55 clusterSetup.go:405: App: TestSourceBucketDelete got un-deployed
2021/12/15 05:15:55 eventingSetup.go:46: No eventing-con process running
2021/12/15 05:15:55 eventingSetup.go:418: Function: TestSourceBucketDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestSourceBucketDelete deleting in the background"
}
2021/12/15 05:15:56 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 05:16:01 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 05:16:03 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 05:16:08 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestSourceBucketDelete (78.96s)
=== RUN   TestSourceBucketDeleteWithBootstrap
2021/12/15 05:16:18 clusterSetup.go:397: Waiting for app: TestSourceBucketDeleteWithBootstrap to get un-deployed
2021/12/15 05:16:18 clusterSetup.go:405: App: TestSourceBucketDeleteWithBootstrap got un-deployed
2021/12/15 05:16:18 eventingSetup.go:46: No eventing-con process running
2021/12/15 05:16:18 eventingSetup.go:418: Function: TestSourceBucketDeleteWithBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestSourceBucketDeleteWithBootstrap not found"
 }
}
2021/12/15 05:16:19 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 05:16:24 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 05:16:26 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 05:16:31 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 05:16:31 eventingSetup.go:125: Deploying app: TestSourceBucketDeleteWithBootstrap. Item count src bucket: 0 dst bucket: 0
2021/12/15 05:16:32 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestSourceBucketDeleteWithBootstrap' in metakv",
  "warnings": null
 }
}
2021/12/15 05:16:51 handler_test.go:966: Deleting source bucket: default
2021/12/15 05:16:52 handler_test.go:968: Deleted source bucket: default
2021/12/15 05:17:07 clusterSetup.go:397: Waiting for app: TestSourceBucketDeleteWithBootstrap to get un-deployed
2021/12/15 05:17:07 clusterSetup.go:405: App: TestSourceBucketDeleteWithBootstrap got un-deployed
2021/12/15 05:17:22 clusterSetup.go:397: Waiting for app: TestSourceBucketDeleteWithBootstrap to get un-deployed
2021/12/15 05:17:22 clusterSetup.go:405: App: TestSourceBucketDeleteWithBootstrap got un-deployed
2021/12/15 05:17:22 eventingSetup.go:46: No eventing-con process running
2021/12/15 05:17:22 eventingSetup.go:418: Function: TestSourceBucketDeleteWithBootstrap delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestSourceBucketDeleteWithBootstrap deleting in the background"
}
2021/12/15 05:17:24 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 05:17:29 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 05:17:30 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 05:17:35 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestSourceBucketDeleteWithBootstrap (87.79s)
=== RUN   TestSourceAndMetaBucketDeleteWithBootstrap
2021/12/15 05:17:45 clusterSetup.go:397: Waiting for app: TestSourceAndMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/15 05:17:45 clusterSetup.go:405: App: TestSourceAndMetaBucketDeleteWithBootstrap got un-deployed
2021/12/15 05:17:45 eventingSetup.go:46: No eventing-con process running
2021/12/15 05:17:45 eventingSetup.go:418: Function: TestSourceAndMetaBucketDeleteWithBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestSourceAndMetaBucketDeleteWithBootstrap not found"
 }
}
2021/12/15 05:17:47 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 05:17:52 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 05:17:54 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 05:17:59 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 05:17:59 eventingSetup.go:125: Deploying app: TestSourceAndMetaBucketDeleteWithBootstrap. Item count src bucket: 0 dst bucket: 0
2021/12/15 05:18:00 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestSourceAndMetaBucketDeleteWithBootstrap' in metakv",
  "warnings": null
 }
}
2021/12/15 05:18:19 handler_test.go:988: Deleting source bucket: default
2021/12/15 05:18:20 handler_test.go:990: Deleted source bucket: default
2021/12/15 05:18:20 handler_test.go:992: Deleting metadata bucket: eventing
2021/12/15 05:18:21 handler_test.go:994: Deleted metadata bucket: eventing
2021/12/15 05:18:36 clusterSetup.go:397: Waiting for app: TestSourceAndMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/15 05:18:36 clusterSetup.go:405: App: TestSourceAndMetaBucketDeleteWithBootstrap got un-deployed
2021/12/15 05:18:51 clusterSetup.go:397: Waiting for app: TestSourceAndMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/15 05:18:51 clusterSetup.go:405: App: TestSourceAndMetaBucketDeleteWithBootstrap got un-deployed
2021/12/15 05:18:51 eventingSetup.go:46: No eventing-con process running
2021/12/15 05:18:51 eventingSetup.go:418: Function: TestSourceAndMetaBucketDeleteWithBootstrap delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestSourceAndMetaBucketDeleteWithBootstrap deleting in the background"
}
2021/12/15 05:18:52 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 05:18:57 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 05:18:59 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 05:19:04 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestSourceAndMetaBucketDeleteWithBootstrap (88.26s)
=== RUN   TestUndeployDuringBootstrap
2021/12/15 05:19:14 clusterSetup.go:397: Waiting for app: TestUndeployDuringBootstrap to get un-deployed
2021/12/15 05:19:14 clusterSetup.go:405: App: TestUndeployDuringBootstrap got un-deployed
2021/12/15 05:19:14 eventingSetup.go:46: No eventing-con process running
2021/12/15 05:19:14 eventingSetup.go:418: Function: TestUndeployDuringBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestUndeployDuringBootstrap not found"
 }
}
2021/12/15 05:19:15 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 05:19:20 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 05:19:22 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 05:19:27 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 05:19:27 eventingSetup.go:125: Deploying app: TestUndeployDuringBootstrap. Item count src bucket: 0 dst bucket: 0
2021/12/15 05:19:28 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestUndeployDuringBootstrap' in metakv",
  "warnings": null
 }
}
2021/12/15 05:19:28 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/15 05:19:40 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 05:19:40 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 05:19:40 clusterSetup.go:375: Apps undergoing bootstrap: true
2021/12/15 05:19:40 clusterSetup.go:378: App: TestUndeployDuringBootstrap started bootstrap
2021/12/15 05:19:40 clusterSetup.go:375: Apps undergoing bootstrap: true
2021/12/15 05:19:45 clusterSetup.go:375: Apps undergoing bootstrap: true
2021/12/15 05:19:50 clusterSetup.go:367: No apps undergoing bootstrap
2021/12/15 05:19:55 clusterSetup.go:397: Waiting for app: TestUndeployDuringBootstrap to get un-deployed
2021/12/15 05:20:00 clusterSetup.go:397: Waiting for app: TestUndeployDuringBootstrap to get un-deployed
2021/12/15 05:20:05 clusterSetup.go:397: Waiting for app: TestUndeployDuringBootstrap to get un-deployed
2021/12/15 05:20:05 clusterSetup.go:405: App: TestUndeployDuringBootstrap got un-deployed
2021/12/15 05:20:05 eventingSetup.go:46: No eventing-con process running
2021/12/15 05:20:30 clusterSetup.go:397: Waiting for app: TestUndeployDuringBootstrap to get un-deployed
2021/12/15 05:20:30 clusterSetup.go:405: App: TestUndeployDuringBootstrap got un-deployed
2021/12/15 05:20:30 eventingSetup.go:46: No eventing-con process running
2021/12/15 05:20:30 eventingSetup.go:418: Function: TestUndeployDuringBootstrap delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestUndeployDuringBootstrap deleting in the background"
}
2021/12/15 05:20:31 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 05:20:41 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/15 05:20:46 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/15 05:20:46 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 05:20:48 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 05:20:58 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3111
2021/12/15 05:21:03 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 611
2021/12/15 05:21:03 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestUndeployDuringBootstrap (119.53s)
=== RUN   TestDeleteBeforeUndeploy
2021/12/15 05:21:08 eventingSetup.go:125: Deploying app: TestDeleteBeforeUndeploy. Item count src bucket: 0 dst bucket: 0
2021/12/15 05:21:09 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeleteBeforeUndeploy' in metakv",
  "warnings": null
 }
}
2021/12/15 05:21:14 clusterSetup.go:346: Waiting for app: TestDeleteBeforeUndeploy to get deployed
2021/12/15 05:21:19 clusterSetup.go:346: Waiting for app: TestDeleteBeforeUndeploy to get deployed
2021/12/15 05:21:24 clusterSetup.go:346: Waiting for app: TestDeleteBeforeUndeploy to get deployed
2021/12/15 05:21:29 clusterSetup.go:346: Waiting for app: TestDeleteBeforeUndeploy to get deployed
2021/12/15 05:21:29 clusterSetup.go:354: App: TestDeleteBeforeUndeploy got deployed
2021/12/15 05:21:29 eventingSetup.go:418: Function: TestDeleteBeforeUndeploy delete from main store request response code: 422 dump: {
 "name": "ERR_APP_NOT_UNDEPLOYED",
 "code": 22,
 "description": "Function hasn't undeployed",
 "attributes": null,
 "runtime_info": {
  "code": 22,
  "info": "Function: TestDeleteBeforeUndeploy skipping delete request from primary store, as it hasn't been undeployed"
 }
}
2021/12/15 05:21:30 eventingSetup.go:270: Function: TestDeleteBeforeUndeploy update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/15 05:21:30 eventingSetup.go:418: Function: TestDeleteBeforeUndeploy delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDeleteBeforeUndeploy deleting in the background"
}
2021/12/15 05:21:35 clusterSetup.go:397: Waiting for app: TestDeleteBeforeUndeploy to get un-deployed
2021/12/15 05:21:35 clusterSetup.go:405: App: TestDeleteBeforeUndeploy got un-deployed
2021/12/15 05:21:35 eventingSetup.go:46: No eventing-con process running
2021/12/15 05:21:35 eventingSetup.go:418: Function: TestDeleteBeforeUndeploy delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDeleteBeforeUndeploy not found"
 }
}
2021/12/15 05:21:36 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 05:21:41 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 05:21:43 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 05:21:48 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDeleteBeforeUndeploy (44.60s)
=== RUN   TestUndeployWhenTimersAreFired
2021/12/15 05:21:58 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/15 05:21:58 clusterSetup.go:405: App: TestUndeployWhenTimersAreFired got un-deployed
2021/12/15 05:21:58 eventingSetup.go:46: No eventing-con process running
2021/12/15 05:21:58 eventingSetup.go:418: Function: TestUndeployWhenTimersAreFired delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestUndeployWhenTimersAreFired not found"
 }
}
2021/12/15 05:21:59 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 05:22:04 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 05:22:06 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 05:22:11 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 05:22:11 eventingSetup.go:125: Deploying app: TestUndeployWhenTimersAreFired. Item count src bucket: 0 dst bucket: 0
2021/12/15 05:22:12 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestUndeployWhenTimersAreFired' in metakv",
  "warnings": null
 }
}
2021/12/15 05:22:17 clusterSetup.go:346: Waiting for app: TestUndeployWhenTimersAreFired to get deployed
2021/12/15 05:22:22 clusterSetup.go:346: Waiting for app: TestUndeployWhenTimersAreFired to get deployed
2021/12/15 05:22:27 clusterSetup.go:346: Waiting for app: TestUndeployWhenTimersAreFired to get deployed
2021/12/15 05:22:32 clusterSetup.go:346: Waiting for app: TestUndeployWhenTimersAreFired to get deployed
2021/12/15 05:22:32 clusterSetup.go:354: App: TestUndeployWhenTimersAreFired got deployed
2021/12/15 05:22:32 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/15 05:23:02 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 05:23:07 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/15 05:23:13 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/15 05:23:18 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/15 05:23:23 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/15 05:23:23 clusterSetup.go:405: App: TestUndeployWhenTimersAreFired got un-deployed
2021/12/15 05:23:23 eventingSetup.go:46: No eventing-con process running
2021/12/15 05:25:08 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/15 05:25:08 clusterSetup.go:405: App: TestUndeployWhenTimersAreFired got un-deployed
2021/12/15 05:25:08 eventingSetup.go:46: No eventing-con process running
2021/12/15 05:25:08 eventingSetup.go:418: Function: TestUndeployWhenTimersAreFired delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestUndeployWhenTimersAreFired deleting in the background"
}
2021/12/15 05:25:09 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 05:25:19 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/15 05:25:24 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/15 05:25:24 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 05:25:26 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 05:25:36 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/15 05:25:36 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestUndeployWhenTimersAreFired (228.03s)
=== RUN   TestUndeployWithKVFailover
2021/12/15 05:25:46 clusterSetup.go:397: Waiting for app: TestUndeployWithKVFailover to get un-deployed
2021/12/15 05:25:46 clusterSetup.go:405: App: TestUndeployWithKVFailover got un-deployed
2021/12/15 05:25:46 eventingSetup.go:46: No eventing-con process running
2021/12/15 05:25:46 eventingSetup.go:418: Function: TestUndeployWithKVFailover delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestUndeployWithKVFailover not found"
 }
}
2021/12/15 05:25:47 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 05:25:52 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 05:25:54 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 05:25:59 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 05:25:59 eventingSetup.go:125: Deploying app: TestUndeployWithKVFailover. Item count src bucket: 0 dst bucket: 0
2021/12/15 05:26:00 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestUndeployWithKVFailover' in metakv",
  "warnings": null
 }
}
2021/12/15 05:26:00 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/15 05:26:07 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 05:26:12 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 05:26:17 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 05:26:22 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 05:26:27 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 05:26:32 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 05:26:37 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 3307
2021/12/15 05:26:42 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 3307
2021/12/15 05:26:42 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/15 05:26:42 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: kv to the cluster
2021/12/15 05:26:42 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/15 05:26:47 Rebalance progress: 29.16666666666666
2021/12/15 05:26:52 Rebalance progress: 29.16666666666666
2021/12/15 05:26:57 Rebalance progress: 100
2021/12/15 05:26:57 Failing over: [127.0.0.1:19003]
2021/12/15 05:27:08 Function: TestUndeployWithKVFailover update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/15 05:28:08 Removing node(s): [] from the cluster
2021/12/15 05:28:13 Rebalance progress: 29.16666666666666
2021/12/15 05:28:18 Rebalance progress: 29.16666666666666
2021/12/15 05:28:23 Rebalance progress: 100
2021/12/15 05:28:28 Waiting for app: TestUndeployWithKVFailover to get un-deployed
2021/12/15 05:28:28 App: TestUndeployWithKVFailover got un-deployed
2021/12/15 05:28:28 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 05:28:33 Waiting for app: TestUndeployWithKVFailover to get un-deployed
2021/12/15 05:28:33 App: TestUndeployWithKVFailover got un-deployed
2021/12/15 05:28:33 No eventing-con process running
2021/12/15 05:28:33 Function: TestUndeployWithKVFailover delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestUndeployWithKVFailover deleting in the background"
}
2021/12/15 05:28:34 Context: Bucket flush, response code: 200 dump: 
2021/12/15 05:28:44 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/15 05:28:49 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/15 05:28:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 05:28:51 Context: Bucket flush, response code: 200 dump: 
2021/12/15 05:29:01 Waiting for dst bucket item count to get to: 0 curr count: 4452
2021/12/15 05:29:06 Waiting for dst bucket item count to get to: 0 curr count: 4452
2021/12/15 05:29:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestUndeployWithKVFailover (209.99s)
=== RUN   TestBucketFlushWhileFnDeployed
2021/12/15 05:29:16 Waiting for app: TestBucketFlushWhileFnDeployed to get un-deployed
2021/12/15 05:29:16 App: TestBucketFlushWhileFnDeployed got un-deployed
2021/12/15 05:29:16 No eventing-con process running
2021/12/15 05:29:16 Function: TestBucketFlushWhileFnDeployed delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestBucketFlushWhileFnDeployed not found"
 }
}
2021/12/15 05:29:17 Context: Bucket flush, response code: 200 dump: 
2021/12/15 05:29:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 05:29:24 Context: Bucket flush, response code: 200 dump: 
2021/12/15 05:29:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 05:29:29 Deploying app: TestBucketFlushWhileFnDeployed. Item count src bucket: 0 dst bucket: 0
2021/12/15 05:29:30 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestBucketFlushWhileFnDeployed' in metakv",
  "warnings": null
 }
}
2021/12/15 05:29:35 Waiting for app: TestBucketFlushWhileFnDeployed to get deployed
2021/12/15 05:29:40 Waiting for app: TestBucketFlushWhileFnDeployed to get deployed
2021/12/15 05:29:45 Waiting for app: TestBucketFlushWhileFnDeployed to get deployed
2021/12/15 05:29:50 Waiting for app: TestBucketFlushWhileFnDeployed to get deployed
2021/12/15 05:29:50 App: TestBucketFlushWhileFnDeployed got deployed
2021/12/15 05:29:50 Starting bucket ops to source bucket
2021/12/15 05:30:11 Waiting for dst bucket item count to get to: 20000 curr count: 10292
2021/12/15 05:30:11 src & dst bucket item count matched up. src bucket count: 20000 dst bucket count: 20000
2021/12/15 05:30:11 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 05:30:13 Context: Bucket flush, response code: 200 dump: 
2021/12/15 05:30:23 Waiting for dst bucket item count to get to: 0 curr count: 20000
2021/12/15 05:30:28 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/15 05:30:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 05:30:30 Context: Bucket flush, response code: 200 dump: 
2021/12/15 05:30:40 Waiting for dst bucket item count to get to: 0 curr count: 12825
2021/12/15 05:30:45 Waiting for dst bucket item count to get to: 0 curr count: 12825
2021/12/15 05:30:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 05:30:45 Starting bucket ops to source bucket
2021/12/15 05:30:58 Waiting for dst bucket item count to get to: 10000 curr count: 4020
2021/12/15 05:31:03 Waiting for dst bucket item count to get to: 10000 curr count: 4020
2021/12/15 05:31:03 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/15 05:31:03 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 05:31:03 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 05:31:08 Waiting for app: TestBucketFlushWhileFnDeployed to get un-deployed
2021/12/15 05:31:13 Waiting for app: TestBucketFlushWhileFnDeployed to get un-deployed
2021/12/15 05:31:18 Waiting for app: TestBucketFlushWhileFnDeployed to get un-deployed
2021/12/15 05:31:23 Waiting for app: TestBucketFlushWhileFnDeployed to get un-deployed
2021/12/15 05:31:23 App: TestBucketFlushWhileFnDeployed got un-deployed
2021/12/15 05:31:23 No eventing-con process running
2021/12/15 05:31:23 Function: TestBucketFlushWhileFnDeployed delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestBucketFlushWhileFnDeployed deleting in the background"
}
2021/12/15 05:31:24 Context: Bucket flush, response code: 200 dump: 
2021/12/15 05:31:34 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/15 05:31:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 05:31:36 Context: Bucket flush, response code: 200 dump: 
2021/12/15 05:31:46 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/15 05:31:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestBucketFlushWhileFnDeployed (160.42s)
=== RUN   TestTimerOverwrite
2021/12/15 05:31:56 Waiting for app: TestTimerOverwrite to get un-deployed
2021/12/15 05:31:56 App: TestTimerOverwrite got un-deployed
2021/12/15 05:31:56 No eventing-con process running
2021/12/15 05:31:56 Function: TestTimerOverwrite delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestTimerOverwrite not found"
 }
}
2021/12/15 05:31:58 Context: Bucket flush, response code: 200 dump: 
2021/12/15 05:32:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 05:32:04 Context: Bucket flush, response code: 200 dump: 
2021/12/15 05:32:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 05:32:09 Deploying app: TestTimerOverwrite. Item count src bucket: 0 dst bucket: 0
2021/12/15 05:32:10 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestTimerOverwrite' in metakv",
  "warnings": null
 }
}
2021/12/15 05:32:15 Waiting for app: TestTimerOverwrite to get deployed
2021/12/15 05:32:20 Waiting for app: TestTimerOverwrite to get deployed
2021/12/15 05:32:25 Waiting for app: TestTimerOverwrite to get deployed
2021/12/15 05:32:30 Waiting for app: TestTimerOverwrite to get deployed
2021/12/15 05:32:31 App: TestTimerOverwrite got deployed
2021/12/15 05:32:41 Starting bucket ops to source bucket
2021/12/15 05:32:49 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 05:32:54 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 05:32:59 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 05:33:04 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 05:33:09 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 05:33:14 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 05:33:19 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 05:33:24 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 05:33:29 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 05:33:34 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 05:33:40 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 05:33:45 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 05:33:50 Waiting for dst bucket item count to get to: 5000 curr count: 4276
2021/12/15 05:33:55 Waiting for dst bucket item count to get to: 5000 curr count: 4276
2021/12/15 05:33:55 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/15 05:34:05 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 05:34:05 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 05:34:10 Waiting for app: TestTimerOverwrite to get un-deployed
2021/12/15 05:34:15 Waiting for app: TestTimerOverwrite to get un-deployed
2021/12/15 05:34:20 Waiting for app: TestTimerOverwrite to get un-deployed
2021/12/15 05:34:25 Waiting for app: TestTimerOverwrite to get un-deployed
2021/12/15 05:34:25 App: TestTimerOverwrite got un-deployed
2021/12/15 05:34:25 No eventing-con process running
2021/12/15 05:34:25 Function: TestTimerOverwrite delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestTimerOverwrite deleting in the background"
}
2021/12/15 05:34:26 Context: Bucket flush, response code: 200 dump: 
2021/12/15 05:34:36 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/15 05:34:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 05:34:38 Context: Bucket flush, response code: 200 dump: 
2021/12/15 05:34:48 Waiting for dst bucket item count to get to: 0 curr count: 3021
2021/12/15 05:34:53 Waiting for dst bucket item count to get to: 0 curr count: 486
2021/12/15 05:34:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestTimerOverwrite (186.54s)
=== RUN   TestUndeployBackdoorDuringBootstrap
2021/12/15 05:34:58 Adding node: https://127.0.0.1:19003 with role: kv to the cluster
2021/12/15 05:34:58 Removing node(s): [] from the cluster
2021/12/15 05:35:03 Rebalance progress: 100
2021/12/15 05:35:03 Failing over: [127.0.0.1:19003]
2021/12/15 05:35:08 Waiting for app: TestUndeployBackdoorDuringBootstrap to get un-deployed
2021/12/15 05:35:08 App: TestUndeployBackdoorDuringBootstrap got un-deployed
2021/12/15 05:35:08 No eventing-con process running
2021/12/15 05:35:08 Function: TestUndeployBackdoorDuringBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestUndeployBackdoorDuringBootstrap not found"
 }
}
2021/12/15 05:35:10 Context: Bucket flush, response code: 200 dump: 
2021/12/15 05:35:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 05:35:16 Context: Bucket flush, response code: 200 dump: 
2021/12/15 05:35:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 05:35:21 Deploying app: TestUndeployBackdoorDuringBootstrap. Item count src bucket: 0 dst bucket: 0
2021/12/15 05:35:23 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestUndeployBackdoorDuringBootstrap' in metakv",
  "warnings": null
 }
}
2021/12/15 05:35:23 Starting bucket ops to source bucket
2021/12/15 05:35:33 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 05:35:33 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 05:35:33 Context: Post to set retry counter, response code: 200 dump: 
2021/12/15 05:35:33 set retry counter, response &{[] }
2021/12/15 05:36:38 Waiting for app: TestUndeployBackdoorDuringBootstrap to get un-deployed
2021/12/15 05:36:38 App: TestUndeployBackdoorDuringBootstrap got un-deployed
2021/12/15 05:36:38 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 05:36:38 Function: TestUndeployBackdoorDuringBootstrap delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestUndeployBackdoorDuringBootstrap deleting in the background"
}
2021/12/15 05:36:43 Waiting for app: TestUndeployBackdoorDuringBootstrap to get un-deployed
2021/12/15 05:36:43 App: TestUndeployBackdoorDuringBootstrap got un-deployed
2021/12/15 05:36:43 No eventing-con process running
2021/12/15 05:36:43 Function: TestUndeployBackdoorDuringBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestUndeployBackdoorDuringBootstrap not found"
 }
}
2021/12/15 05:36:44 Context: Bucket flush, response code: 200 dump: 
2021/12/15 05:36:54 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/15 05:36:59 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/15 05:36:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 05:37:01 Context: Bucket flush, response code: 200 dump: 
2021/12/15 05:37:11 Waiting for dst bucket item count to get to: 0 curr count: 1666
2021/12/15 05:37:16 Waiting for dst bucket item count to get to: 0 curr count: 1666
2021/12/15 05:37:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 05:37:16 Removing node(s): [] from the cluster
2021/12/15 05:37:21 Rebalance progress: 100
--- PASS: TestUndeployBackdoorDuringBootstrap (148.20s)
=== RUN   TestOnUpdateSrcMutation
2021/12/15 05:37:31 Waiting for app: TestOnUpdateSrcMutation to get un-deployed
2021/12/15 05:37:31 App: TestOnUpdateSrcMutation got un-deployed
2021/12/15 05:37:31 No eventing-con process running
2021/12/15 05:37:31 Function: TestOnUpdateSrcMutation delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateSrcMutation not found"
 }
}
2021/12/15 05:37:32 Context: Bucket flush, response code: 200 dump: 
2021/12/15 05:37:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 05:37:39 Context: Bucket flush, response code: 200 dump: 
2021/12/15 05:37:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 05:37:44 Deploying app: TestOnUpdateSrcMutation. Item count src bucket: 0 dst bucket: 0
2021/12/15 05:37:45 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateSrcMutation' in metakv",
  "warnings": null
 }
}
2021/12/15 05:37:45 Starting bucket ops to source bucket
2021/12/15 05:38:12 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/15 05:38:12 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 05:38:12 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 05:38:17 Waiting for app: TestOnUpdateSrcMutation to get un-deployed
2021/12/15 05:38:22 Waiting for app: TestOnUpdateSrcMutation to get un-deployed
2021/12/15 05:38:27 Waiting for app: TestOnUpdateSrcMutation to get un-deployed
2021/12/15 05:38:32 Waiting for app: TestOnUpdateSrcMutation to get un-deployed
2021/12/15 05:38:32 App: TestOnUpdateSrcMutation got un-deployed
2021/12/15 05:38:32 No eventing-con process running
2021/12/15 05:38:32 Function: TestOnUpdateSrcMutation delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateSrcMutation deleting in the background"
}
2021/12/15 05:38:34 Context: Bucket flush, response code: 200 dump: 
2021/12/15 05:38:44 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/15 05:38:49 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/15 05:38:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 05:38:50 Context: Bucket flush, response code: 200 dump: 
2021/12/15 05:38:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateSrcMutation (94.47s)
=== RUN   TestOnDeleteSrcMutation
2021/12/15 05:39:05 Waiting for app: TestOnDeleteSrcMutation to get un-deployed
2021/12/15 05:39:05 App: TestOnDeleteSrcMutation got un-deployed
2021/12/15 05:39:05 No eventing-con process running
2021/12/15 05:39:05 Function: TestOnDeleteSrcMutation delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnDeleteSrcMutation not found"
 }
}
2021/12/15 05:39:07 Context: Bucket flush, response code: 200 dump: 
2021/12/15 05:39:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 05:39:14 Context: Bucket flush, response code: 200 dump: 
2021/12/15 05:39:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 05:39:19 Deploying app: TestOnDeleteSrcMutation. Item count src bucket: 0 dst bucket: 0
2021/12/15 05:39:20 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnDeleteSrcMutation' in metakv",
  "warnings": null
 }
}
2021/12/15 05:39:20 Starting bucket ops to source bucket
2021/12/15 05:39:48 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/15 05:39:48 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 05:39:49 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 05:39:54 Waiting for app: TestOnDeleteSrcMutation to get un-deployed
2021/12/15 05:39:59 Waiting for app: TestOnDeleteSrcMutation to get un-deployed
2021/12/15 05:40:04 Waiting for app: TestOnDeleteSrcMutation to get un-deployed
2021/12/15 05:40:09 Waiting for app: TestOnDeleteSrcMutation to get un-deployed
2021/12/15 05:40:09 App: TestOnDeleteSrcMutation got un-deployed
2021/12/15 05:40:09 No eventing-con process running
2021/12/15 05:40:09 Function: TestOnDeleteSrcMutation delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnDeleteSrcMutation deleting in the background"
}
2021/12/15 05:40:10 Context: Bucket flush, response code: 200 dump: 
2021/12/15 05:40:20 Waiting for dst bucket item count to get to: 0 curr count: 3022
2021/12/15 05:40:25 Waiting for dst bucket item count to get to: 0 curr count: 3022
2021/12/15 05:40:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 05:40:27 Context: Bucket flush, response code: 200 dump: 
2021/12/15 05:40:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnDeleteSrcMutation (96.31s)
=== RUN   TestOnUpdateSrcMutationWithTimer
2021/12/15 05:40:42 Waiting for app: TestOnUpdateSrcMutationWithTimer to get un-deployed
2021/12/15 05:40:42 App: TestOnUpdateSrcMutationWithTimer got un-deployed
2021/12/15 05:40:42 No eventing-con process running
2021/12/15 05:40:42 Function: TestOnUpdateSrcMutationWithTimer delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateSrcMutationWithTimer not found"
 }
}
2021/12/15 05:40:43 Context: Bucket flush, response code: 200 dump: 
2021/12/15 05:40:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 05:40:50 Context: Bucket flush, response code: 200 dump: 
2021/12/15 05:40:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 05:40:55 Deploying app: TestOnUpdateSrcMutationWithTimer. Item count src bucket: 0 dst bucket: 0
2021/12/15 05:40:56 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateSrcMutationWithTimer' in metakv",
  "warnings": null
 }
}
2021/12/15 05:40:56 Starting bucket ops to source bucket
2021/12/15 05:41:33 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/15 05:41:33 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 05:41:33 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 05:41:38 Waiting for app: TestOnUpdateSrcMutationWithTimer to get un-deployed
2021/12/15 05:41:43 Waiting for app: TestOnUpdateSrcMutationWithTimer to get un-deployed
2021/12/15 05:41:48 Waiting for app: TestOnUpdateSrcMutationWithTimer to get un-deployed
2021/12/15 05:41:53 Waiting for app: TestOnUpdateSrcMutationWithTimer to get un-deployed
2021/12/15 05:41:53 App: TestOnUpdateSrcMutationWithTimer got un-deployed
2021/12/15 05:41:53 No eventing-con process running
2021/12/15 05:41:53 Function: TestOnUpdateSrcMutationWithTimer delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateSrcMutationWithTimer deleting in the background"
}
2021/12/15 05:41:54 Context: Bucket flush, response code: 200 dump: 
2021/12/15 05:42:04 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/15 05:42:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 05:42:06 Context: Bucket flush, response code: 200 dump: 
2021/12/15 05:42:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateSrcMutationWithTimer (99.53s)
=== RUN   TestOnDeleteSrcMutationsWithTimer
2021/12/15 05:42:21 Waiting for app: TestOnDeleteSrcMutationsWithTimer to get un-deployed
2021/12/15 05:42:21 App: TestOnDeleteSrcMutationsWithTimer got un-deployed
2021/12/15 05:42:21 No eventing-con process running
2021/12/15 05:42:21 Function: TestOnDeleteSrcMutationsWithTimer delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnDeleteSrcMutationsWithTimer not found"
 }
}
2021/12/15 05:42:23 Context: Bucket flush, response code: 200 dump: 
2021/12/15 05:42:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 05:42:29 Context: Bucket flush, response code: 200 dump: 
2021/12/15 05:42:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 05:42:34 Deploying app: TestOnDeleteSrcMutationsWithTimer. Item count src bucket: 0 dst bucket: 0
2021/12/15 05:42:36 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnDeleteSrcMutationsWithTimer' in metakv",
  "warnings": null
 }
}
2021/12/15 05:42:36 Starting bucket ops to source bucket
2021/12/15 05:43:25 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/15 05:43:25 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 05:43:25 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 05:43:30 Waiting for app: TestOnDeleteSrcMutationsWithTimer to get un-deployed
2021/12/15 05:43:35 Waiting for app: TestOnDeleteSrcMutationsWithTimer to get un-deployed
2021/12/15 05:43:40 Waiting for app: TestOnDeleteSrcMutationsWithTimer to get un-deployed
2021/12/15 05:43:45 Waiting for app: TestOnDeleteSrcMutationsWithTimer to get un-deployed
2021/12/15 05:43:45 App: TestOnDeleteSrcMutationsWithTimer got un-deployed
2021/12/15 05:43:45 No eventing-con process running
2021/12/15 05:43:45 Function: TestOnDeleteSrcMutationsWithTimer delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnDeleteSrcMutationsWithTimer deleting in the background"
}
2021/12/15 05:43:46 Context: Bucket flush, response code: 200 dump: 
2021/12/15 05:43:56 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/15 05:43:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 05:43:58 Context: Bucket flush, response code: 200 dump: 
2021/12/15 05:44:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnDeleteSrcMutationsWithTimer (111.51s)
=== RUN   TestInterHandlerRecursion
2021/12/15 05:44:13 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/15 05:44:13 App: TestInterHandlerRecursion_source_bucket_update_op got un-deployed
2021/12/15 05:44:13 No eventing-con process running
2021/12/15 05:44:13 Function: TestInterHandlerRecursion_source_bucket_update_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestInterHandlerRecursion_source_bucket_update_op not found"
 }
}
2021/12/15 05:44:14 Context: Bucket flush, response code: 200 dump: 
2021/12/15 05:44:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 05:44:21 Context: Bucket flush, response code: 200 dump: 
2021/12/15 05:44:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 05:44:31 Waiting for app: TestInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/15 05:44:31 App: TestInterHandlerRecursion_src_bucket_op_on_delete got un-deployed
2021/12/15 05:44:31 No eventing-con process running
2021/12/15 05:44:31 Function: TestInterHandlerRecursion_src_bucket_op_on_delete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestInterHandlerRecursion_src_bucket_op_on_delete not found"
 }
}
2021/12/15 05:44:32 Context: Bucket flush, response code: 200 dump: 
2021/12/15 05:44:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 05:44:39 Context: Bucket flush, response code: 200 dump: 
2021/12/15 05:44:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 05:44:44 Deploying app: TestInterHandlerRecursion_source_bucket_update_op. Item count src bucket: 0 dst bucket: 0
2021/12/15 05:44:45 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterHandlerRecursion_source_bucket_update_op' in metakv",
  "warnings": null
 }
}
2021/12/15 05:44:45 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterHandlerRecursion_source_bucket_update_op' in metakv",
  "warnings": null
 }
} err 
2021/12/15 05:44:50 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/15 05:44:55 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/15 05:45:00 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/15 05:45:05 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/15 05:45:05 App: TestInterHandlerRecursion_source_bucket_update_op got deployed
2021/12/15 05:45:05 Deploying app: TestInterHandlerRecursion_src_bucket_op_on_delete. Item count src bucket: 0 dst bucket: 0
2021/12/15 05:45:05 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_INTER_FUNCTION_RECURSION",
 "code": 50,
 "description": "Inter function recursion error, only one function is allowed to do source bucket mutation/delete on a bucket",
 "attributes": null,
 "runtime_info": {
  "code": 50,
  "info": "Inter handler recursion error"
 }
}
2021/12/15 05:45:05 response body {
 "name": "ERR_INTER_FUNCTION_RECURSION",
 "code": 50,
 "description": "Inter function recursion error, only one function is allowed to do source bucket mutation/delete on a bucket",
 "attributes": null,
 "runtime_info": {
  "code": 50,
  "info": "Inter handler recursion error"
 }
} err 
2021/12/15 05:45:05 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 05:45:10 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/15 05:45:15 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/15 05:45:20 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/15 05:45:25 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/15 05:45:25 App: TestInterHandlerRecursion_source_bucket_update_op got un-deployed
2021/12/15 05:45:25 No eventing-con process running
2021/12/15 05:45:25 Function: TestInterHandlerRecursion_source_bucket_update_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestInterHandlerRecursion_source_bucket_update_op deleting in the background"
}
2021/12/15 05:45:27 Context: Bucket flush, response code: 200 dump: 
2021/12/15 05:45:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 05:45:33 Context: Bucket flush, response code: 200 dump: 
2021/12/15 05:45:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 05:45:43 Waiting for app: TestInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/15 05:45:43 App: TestInterHandlerRecursion_src_bucket_op_on_delete got un-deployed
2021/12/15 05:45:43 No eventing-con process running
2021/12/15 05:45:44 Function: TestInterHandlerRecursion_src_bucket_op_on_delete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestInterHandlerRecursion_src_bucket_op_on_delete not found"
 }
}
2021/12/15 05:45:45 Context: Bucket flush, response code: 200 dump: 
2021/12/15 05:45:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 05:45:52 Context: Bucket flush, response code: 200 dump: 
2021/12/15 05:45:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestInterHandlerRecursion (113.80s)
=== RUN   TestInterBucketRecursion
2021/12/15 05:46:02 Deploying app: TestInterBucketRecursion_function1. Item count src bucket: 0 dst bucket: 0
2021/12/15 05:46:03 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterBucketRecursion_function1' in metakv",
  "warnings": null
 }
}
2021/12/15 05:46:03 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterBucketRecursion_function1' in metakv",
  "warnings": null
 }
} err 
2021/12/15 05:46:03 Deploying app: TestInterBucketRecursion_function2. Item count src bucket: 0 dst bucket: 0
2021/12/15 05:46:03 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_INTER_BUCKET_RECURSION",
 "code": 51,
 "description": "Inter bucket recursion error, deployment of current handler will cause inter bucket recursion",
 "attributes": null,
 "runtime_info": {
  "code": 51,
  "info": "Inter bucket recursion error; function: TestInterBucketRecursion_function2 causes a cycle involving functions: [TestInterBucketRecursion_function1], hence deployment is disallowed"
 }
}
2021/12/15 05:46:08 Waiting for app: TestInterBucketRecursion_function1 to get deployed
2021/12/15 05:46:13 Waiting for app: TestInterBucketRecursion_function1 to get deployed
2021/12/15 05:46:18 Waiting for app: TestInterBucketRecursion_function1 to get deployed
2021/12/15 05:46:23 Waiting for app: TestInterBucketRecursion_function1 to get deployed
2021/12/15 05:46:23 App: TestInterBucketRecursion_function1 got deployed
2021/12/15 05:46:23 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 05:46:28 Waiting for app: TestInterBucketRecursion_function1 to get un-deployed
2021/12/15 05:46:33 Waiting for app: TestInterBucketRecursion_function1 to get un-deployed
2021/12/15 05:46:38 Waiting for app: TestInterBucketRecursion_function1 to get un-deployed
2021/12/15 05:46:43 Waiting for app: TestInterBucketRecursion_function1 to get un-deployed
2021/12/15 05:46:43 App: TestInterBucketRecursion_function1 got un-deployed
2021/12/15 05:46:43 No eventing-con process running
2021/12/15 05:46:43 Function: TestInterBucketRecursion_function1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestInterBucketRecursion_function1 deleting in the background"
}
2021/12/15 05:46:44 Context: Bucket flush, response code: 200 dump: 
2021/12/15 05:46:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 05:46:51 Context: Bucket flush, response code: 200 dump: 
2021/12/15 05:46:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 05:47:01 Waiting for app: TestInterBucketRecursion_function2 to get un-deployed
2021/12/15 05:47:01 App: TestInterBucketRecursion_function2 got un-deployed
2021/12/15 05:47:01 No eventing-con process running
2021/12/15 05:47:01 Function: TestInterBucketRecursion_function2 delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestInterBucketRecursion_function2 not found"
 }
}
2021/12/15 05:47:02 Context: Bucket flush, response code: 200 dump: 
2021/12/15 05:47:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 05:47:09 Context: Bucket flush, response code: 200 dump: 
2021/12/15 05:47:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestInterBucketRecursion (77.56s)
=== RUN   TestLargeHandler
2021/12/15 05:47:14 Changed force_compress value to false
2021/12/15 05:47:14 Deploying app: TestLargeHandler_function. Item count src bucket: 0 dst bucket: 0
2021/12/15 05:47:14 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_APPCODE_SIZE",
 "code": 39,
 "description": "Function Code size is more than the configured limit",
 "attributes": null,
 "runtime_info": {
  "code": 39,
  "info": "Function: TestLargeHandler_function handler Code size is more than 131072. Code Size: 131768"
 }
}
2021/12/15 05:47:14 Changed force_compress value to true
2021/12/15 05:47:14 Deploying app: TestLargeHandler_function. Item count src bucket: 0 dst bucket: 0
2021/12/15 05:47:16 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestLargeHandler_function' in metakv",
  "warnings": null
 }
}
2021/12/15 05:47:21 Waiting for app: TestLargeHandler_function to get deployed
2021/12/15 05:47:26 Waiting for app: TestLargeHandler_function to get deployed
2021/12/15 05:47:31 Waiting for app: TestLargeHandler_function to get deployed
2021/12/15 05:47:36 Waiting for app: TestLargeHandler_function to get deployed
2021/12/15 05:47:36 App: TestLargeHandler_function got deployed
2021/12/15 05:47:36 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 05:47:41 Waiting for app: TestLargeHandler_function to get un-deployed
2021/12/15 05:47:46 Waiting for app: TestLargeHandler_function to get un-deployed
2021/12/15 05:47:51 Waiting for app: TestLargeHandler_function to get un-deployed
2021/12/15 05:47:56 Waiting for app: TestLargeHandler_function to get un-deployed
2021/12/15 05:47:56 App: TestLargeHandler_function got un-deployed
2021/12/15 05:47:56 No eventing-con process running
2021/12/15 05:47:57 Function: TestLargeHandler_function delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestLargeHandler_function deleting in the background"
}
2021/12/15 05:47:58 Context: Bucket flush, response code: 200 dump: 
2021/12/15 05:48:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 05:48:04 Context: Bucket flush, response code: 200 dump: 
2021/12/15 05:48:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestLargeHandler (55.32s)
=== RUN   TestAllowInterHandlerRecursion
2021/12/15 05:48:19 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/15 05:48:19 App: TestAllowInterHandlerRecursion_source_bucket_update_op got un-deployed
2021/12/15 05:48:19 No eventing-con process running
2021/12/15 05:48:19 Function: TestAllowInterHandlerRecursion_source_bucket_update_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestAllowInterHandlerRecursion_source_bucket_update_op not found"
 }
}
2021/12/15 05:48:21 Context: Bucket flush, response code: 200 dump: 
2021/12/15 05:48:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 05:48:28 Context: Bucket flush, response code: 200 dump: 
2021/12/15 05:48:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 05:48:38 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/15 05:48:38 App: TestAllowInterHandlerRecursion_src_bucket_op_on_delete got un-deployed
2021/12/15 05:48:38 No eventing-con process running
2021/12/15 05:48:38 Function: TestAllowInterHandlerRecursion_src_bucket_op_on_delete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestAllowInterHandlerRecursion_src_bucket_op_on_delete not found"
 }
}
2021/12/15 05:48:39 Context: Bucket flush, response code: 200 dump: 
2021/12/15 05:48:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 05:48:46 Context: Bucket flush, response code: 200 dump: 
2021/12/15 05:48:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 05:48:51 Deploying app: TestAllowInterHandlerRecursion_source_bucket_update_op. Item count src bucket: 0 dst bucket: 0
2021/12/15 05:48:52 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterHandlerRecursion_source_bucket_update_op' in metakv",
  "warnings": null
 }
}
2021/12/15 05:48:52 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterHandlerRecursion_source_bucket_update_op' in metakv",
  "warnings": null
 }
} err 
2021/12/15 05:48:57 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/15 05:49:02 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/15 05:49:07 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/15 05:49:12 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/15 05:49:12 App: TestAllowInterHandlerRecursion_source_bucket_update_op got deployed
2021/12/15 05:49:12 Deploying app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete. Item count src bucket: 0 dst bucket: 0
2021/12/15 05:49:13 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterHandlerRecursion_src_bucket_op_on_delete' in metakv",
  "warnings": null
 }
}
2021/12/15 05:49:13 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterHandlerRecursion_src_bucket_op_on_delete' in metakv",
  "warnings": null
 }
} err 
2021/12/15 05:49:18 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get deployed
2021/12/15 05:49:23 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get deployed
2021/12/15 05:49:28 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get deployed
2021/12/15 05:49:33 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get deployed
2021/12/15 05:49:33 App: TestAllowInterHandlerRecursion_src_bucket_op_on_delete got deployed
2021/12/15 05:49:33 TestAllowInterHandlerRecursion_src_bucket_op_on_delete is deployed
2021/12/15 05:49:33 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 05:49:38 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/15 05:49:43 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/15 05:49:48 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/15 05:49:53 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/15 05:49:53 App: TestAllowInterHandlerRecursion_source_bucket_update_op got un-deployed
2021/12/15 05:49:53 No eventing-con process running
2021/12/15 05:49:53 Function: TestAllowInterHandlerRecursion_source_bucket_update_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAllowInterHandlerRecursion_source_bucket_update_op deleting in the background"
}
2021/12/15 05:49:56 Context: Bucket flush, response code: 200 dump: 
2021/12/15 05:50:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 05:50:02 Context: Bucket flush, response code: 200 dump: 
2021/12/15 05:50:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 05:50:08 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 05:50:13 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/15 05:50:18 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/15 05:50:23 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/15 05:50:28 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/15 05:50:28 App: TestAllowInterHandlerRecursion_src_bucket_op_on_delete got un-deployed
2021/12/15 05:50:28 No eventing-con process running
2021/12/15 05:50:28 Function: TestAllowInterHandlerRecursion_src_bucket_op_on_delete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAllowInterHandlerRecursion_src_bucket_op_on_delete deleting in the background"
}
2021/12/15 05:50:29 Context: Bucket flush, response code: 200 dump: 
2021/12/15 05:50:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 05:50:36 Context: Bucket flush, response code: 200 dump: 
2021/12/15 05:50:41 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAllowInterHandlerRecursion (151.27s)
=== RUN   TestAllowInterBucketRecursion
2021/12/15 05:50:46 Deploying app: TestAllowInterBucketRecursion_function1. Item count src bucket: 0 dst bucket: 0
2021/12/15 05:50:47 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterBucketRecursion_function1' in metakv",
  "warnings": null
 }
}
2021/12/15 05:50:47 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterBucketRecursion_function1' in metakv",
  "warnings": null
 }
} err 
2021/12/15 05:50:52 Waiting for app: TestAllowInterBucketRecursion_function1 to get deployed
2021/12/15 05:50:57 Waiting for app: TestAllowInterBucketRecursion_function1 to get deployed
2021/12/15 05:51:02 Waiting for app: TestAllowInterBucketRecursion_function1 to get deployed
2021/12/15 05:51:07 Waiting for app: TestAllowInterBucketRecursion_function1 to get deployed
2021/12/15 05:51:07 App: TestAllowInterBucketRecursion_function1 got deployed
2021/12/15 05:51:07 Deploying app: TestAllowInterBucketRecursion_function2. Item count src bucket: 0 dst bucket: 0
2021/12/15 05:51:08 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterBucketRecursion_function2' in metakv",
  "warnings": null
 }
}
2021/12/15 05:51:08 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterBucketRecursion_function2' in metakv",
  "warnings": null
 }
} err 
2021/12/15 05:51:13 Waiting for app: TestAllowInterBucketRecursion_function2 to get deployed
2021/12/15 05:51:18 Waiting for app: TestAllowInterBucketRecursion_function2 to get deployed
2021/12/15 05:51:23 Waiting for app: TestAllowInterBucketRecursion_function2 to get deployed
2021/12/15 05:51:28 Waiting for app: TestAllowInterBucketRecursion_function2 to get deployed
2021/12/15 05:51:28 App: TestAllowInterBucketRecursion_function2 got deployed
2021/12/15 05:51:28 TestAllowInterBucketRecursion_function2 is deployed
2021/12/15 05:51:28 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 05:51:33 Waiting for app: TestAllowInterBucketRecursion_function1 to get un-deployed
2021/12/15 05:51:38 Waiting for app: TestAllowInterBucketRecursion_function1 to get un-deployed
2021/12/15 05:51:43 Waiting for app: TestAllowInterBucketRecursion_function1 to get un-deployed
2021/12/15 05:51:48 Waiting for app: TestAllowInterBucketRecursion_function1 to get un-deployed
2021/12/15 05:51:48 App: TestAllowInterBucketRecursion_function1 got un-deployed
2021/12/15 05:51:48 No eventing-con process running
2021/12/15 05:51:48 Function: TestAllowInterBucketRecursion_function1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAllowInterBucketRecursion_function1 deleting in the background"
}
2021/12/15 05:51:50 Context: Bucket flush, response code: 200 dump: 
2021/12/15 05:51:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 05:51:58 Context: Bucket flush, response code: 200 dump: 
2021/12/15 05:52:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 05:52:03 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 05:52:08 Waiting for app: TestAllowInterBucketRecursion_function2 to get un-deployed
2021/12/15 05:52:13 Waiting for app: TestAllowInterBucketRecursion_function2 to get un-deployed
2021/12/15 05:52:18 Waiting for app: TestAllowInterBucketRecursion_function2 to get un-deployed
2021/12/15 05:52:23 Waiting for app: TestAllowInterBucketRecursion_function2 to get un-deployed
2021/12/15 05:52:23 App: TestAllowInterBucketRecursion_function2 got un-deployed
2021/12/15 05:52:23 No eventing-con process running
2021/12/15 05:52:23 Function: TestAllowInterBucketRecursion_function2 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAllowInterBucketRecursion_function2 deleting in the background"
}
2021/12/15 05:52:24 Context: Bucket flush, response code: 200 dump: 
2021/12/15 05:52:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 05:52:31 Context: Bucket flush, response code: 200 dump: 
2021/12/15 05:52:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAllowInterBucketRecursion (115.44s)
=== RUN   TestInterBucketRecursion2
2021/12/15 05:52:42 Context: Bucket flush, response code: 200 dump: 
2021/12/15 05:52:48 Context: Bucket flush, response code: 200 dump: 
2021/12/15 05:52:54 Context: Bucket flush, response code: 200 dump: 
2021/12/15 05:53:00 Context: Bucket flush, response code: 200 dump: 
2021/12/15 05:53:05 Deploying app: TestInterBucketRecursion2_function1. Item count src bucket: 0 dst bucket: 0
2021/12/15 05:53:06 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterBucketRecursion2_function1' in metakv",
  "warnings": null
 }
}
2021/12/15 05:53:11 Waiting for app: TestInterBucketRecursion2_function1 to get deployed
2021/12/15 05:53:16 Waiting for app: TestInterBucketRecursion2_function1 to get deployed
2021/12/15 05:53:21 Waiting for app: TestInterBucketRecursion2_function1 to get deployed
2021/12/15 05:53:26 Waiting for app: TestInterBucketRecursion2_function1 to get deployed
2021/12/15 05:53:31 Waiting for app: TestInterBucketRecursion2_function1 to get deployed
2021/12/15 05:53:36 Waiting for app: TestInterBucketRecursion2_function1 to get deployed
2021/12/15 05:53:36 App: TestInterBucketRecursion2_function1 got deployed
2021/12/15 05:53:36 Deploying app: TestInterBucketRecursion2_function2. Item count src bucket: 0 dst bucket: 0
2021/12/15 05:53:37 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterBucketRecursion2_function2' in metakv",
  "warnings": null
 }
}
2021/12/15 05:53:42 Waiting for app: TestInterBucketRecursion2_function2 to get deployed
2021/12/15 05:53:47 Waiting for app: TestInterBucketRecursion2_function2 to get deployed
2021/12/15 05:53:52 Waiting for app: TestInterBucketRecursion2_function2 to get deployed
2021/12/15 05:53:57 Waiting for app: TestInterBucketRecursion2_function2 to get deployed
2021/12/15 05:53:57 App: TestInterBucketRecursion2_function2 got deployed
2021/12/15 05:53:57 Deploying app: TestInterBucketRecursion2_function3. Item count src bucket: 0 dst bucket: 0
2021/12/15 05:53:59 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterBucketRecursion2_function3' in metakv",
  "warnings": null
 }
}
2021/12/15 05:54:04 Waiting for app: TestInterBucketRecursion2_function3 to get deployed
2021/12/15 05:54:09 Waiting for app: TestInterBucketRecursion2_function3 to get deployed
2021/12/15 05:54:14 Waiting for app: TestInterBucketRecursion2_function3 to get deployed
2021/12/15 05:54:19 Waiting for app: TestInterBucketRecursion2_function3 to get deployed
2021/12/15 05:54:19 App: TestInterBucketRecursion2_function3 got deployed
2021/12/15 05:54:19 Deploying app: TestInterBucketRecursion2_function4. Item count src bucket: 0 dst bucket: 0
2021/12/15 05:54:19 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_INTER_BUCKET_RECURSION",
 "code": 51,
 "description": "Inter bucket recursion error, deployment of current handler will cause inter bucket recursion",
 "attributes": null,
 "runtime_info": {
  "code": 51,
  "info": "Inter bucket recursion error; function: TestInterBucketRecursion2_function4 causes a cycle involving functions: [TestInterBucketRecursion2_function1 TestInterBucketRecursion2_function2 TestInterBucketRecursion2_function3], hence deployment is disallowed"
 }
}
2021/12/15 05:54:19 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 05:54:24 Waiting for app: TestInterBucketRecursion2_function1 to get un-deployed
2021/12/15 05:54:29 Waiting for app: TestInterBucketRecursion2_function1 to get un-deployed
2021/12/15 05:54:34 Waiting for app: TestInterBucketRecursion2_function1 to get un-deployed
2021/12/15 05:54:39 Waiting for app: TestInterBucketRecursion2_function1 to get un-deployed
2021/12/15 05:54:39 App: TestInterBucketRecursion2_function1 got un-deployed
2021/12/15 05:54:39 No eventing-con process running
2021/12/15 05:54:39 Function: TestInterBucketRecursion2_function1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestInterBucketRecursion2_function1 deleting in the background"
}
2021/12/15 05:54:40 Context: Bucket flush, response code: 200 dump: 
2021/12/15 05:54:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 05:54:47 Context: Bucket flush, response code: 200 dump: 
2021/12/15 05:54:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 05:54:52 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 05:54:57 Waiting for app: TestInterBucketRecursion2_function2 to get un-deployed
2021/12/15 05:55:02 Waiting for app: TestInterBucketRecursion2_function2 to get un-deployed
2021/12/15 05:55:07 Waiting for app: TestInterBucketRecursion2_function2 to get un-deployed
2021/12/15 05:55:12 Waiting for app: TestInterBucketRecursion2_function2 to get un-deployed
2021/12/15 05:55:12 App: TestInterBucketRecursion2_function2 got un-deployed
2021/12/15 05:55:12 No eventing-con process running
2021/12/15 05:55:12 Function: TestInterBucketRecursion2_function2 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestInterBucketRecursion2_function2 deleting in the background"
}
2021/12/15 05:55:14 Context: Bucket flush, response code: 200 dump: 
2021/12/15 05:55:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 05:55:20 Context: Bucket flush, response code: 200 dump: 
2021/12/15 05:55:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 05:55:26 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 05:55:31 Waiting for app: TestInterBucketRecursion2_function3 to get un-deployed
2021/12/15 05:55:36 Waiting for app: TestInterBucketRecursion2_function3 to get un-deployed
2021/12/15 05:55:41 Waiting for app: TestInterBucketRecursion2_function3 to get un-deployed
2021/12/15 05:55:46 Waiting for app: TestInterBucketRecursion2_function3 to get un-deployed
2021/12/15 05:55:46 App: TestInterBucketRecursion2_function3 got un-deployed
2021/12/15 05:55:46 No eventing-con process running
2021/12/15 05:55:46 Function: TestInterBucketRecursion2_function3 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestInterBucketRecursion2_function3 deleting in the background"
}
2021/12/15 05:55:47 Context: Bucket flush, response code: 200 dump: 
2021/12/15 05:55:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 05:55:54 Context: Bucket flush, response code: 200 dump: 
2021/12/15 05:55:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 05:56:04 Waiting for app: TestInterBucketRecursion2_function4 to get un-deployed
2021/12/15 05:56:04 App: TestInterBucketRecursion2_function4 got un-deployed
2021/12/15 05:56:04 No eventing-con process running
2021/12/15 05:56:04 Function: TestInterBucketRecursion2_function4 delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestInterBucketRecursion2_function4 not found"
 }
}
2021/12/15 05:56:05 Context: Bucket flush, response code: 200 dump: 
2021/12/15 05:56:10 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 05:56:12 Context: Bucket flush, response code: 200 dump: 
2021/12/15 05:56:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestInterBucketRecursion2 (222.65s)
=== RUN   TestBucketDeleteWithRebOut
2021/12/15 05:56:24 Removing node(s): [127.0.0.1:19001 127.0.0.1:19002 127.0.0.1:19003] from the cluster
2021/12/15 05:56:29 Rebalance progress: 100

VB distribution across eventing nodes and their workers::

2021/12/15 05:56:32 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:44} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/15 05:56:47 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:44} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/15 05:57:02 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:44} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/15 05:57:17 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:44} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/15 05:57:32 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:44} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/15 05:57:47 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:44} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/15 05:58:02 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:44} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/15 05:58:17 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:44} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/15 05:58:32 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:44} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/15 05:58:47 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:44} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/15 05:59:02 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:44} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/15 05:59:17 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:44} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/15 05:59:32 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:44} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/15 05:59:47 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:44} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/15 06:00:02 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:44} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/15 06:00:17 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:44} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/15 06:00:32 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:44} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/15 06:00:47 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:44} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/15 06:01:02 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:44} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/15 06:01:17 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:44} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/15 06:01:32 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:44} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/15 06:01:47 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:44} {Name:#primary Bucket:hello-world Completion:100}]

DCP Stream statuses:

vbucket curr owner:


2021/12/15 06:02:02 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 06:02:02 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/15 06:02:02 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/15 06:02:07 Rebalance progress: 100

VB distribution across eventing nodes and their workers::


DCP Stream statuses:

vbucket curr owner:


2021/12/15 06:02:07 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 06:02:07 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/15 06:02:08 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/15 06:02:13 Rebalance progress: 100

VB distribution across eventing nodes and their workers::


DCP Stream statuses:

vbucket curr owner:


2021/12/15 06:02:13 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 06:02:18 clusterSetup.go:397: Waiting for app: TestBucketDeleteWithRebOut to get un-deployed
2021/12/15 06:02:18 clusterSetup.go:405: App: TestBucketDeleteWithRebOut got un-deployed
2021/12/15 06:02:18 eventingSetup.go:46: No eventing-con process running
2021/12/15 06:02:18 eventingSetup.go:418: Function: TestBucketDeleteWithRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestBucketDeleteWithRebOut not found"
 }
}
2021/12/15 06:02:19 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 06:02:24 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 06:02:26 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 06:02:31 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 06:02:36 eventingSetup.go:125: Deploying app: TestBucketDeleteWithRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/15 06:02:37 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestBucketDeleteWithRebOut' in metakv",
  "warnings": null
 }
}
2021/12/15 06:02:42 clusterSetup.go:346: Waiting for app: TestBucketDeleteWithRebOut to get deployed
2021/12/15 06:02:47 clusterSetup.go:346: Waiting for app: TestBucketDeleteWithRebOut to get deployed
2021/12/15 06:02:52 clusterSetup.go:346: Waiting for app: TestBucketDeleteWithRebOut to get deployed
2021/12/15 06:02:57 clusterSetup.go:346: Waiting for app: TestBucketDeleteWithRebOut to get deployed
2021/12/15 06:02:57 clusterSetup.go:354: App: TestBucketDeleteWithRebOut got deployed

VB distribution across eventing nodes and their workers::


DCP Stream statuses:
	stream status: running
	len: 1024
	vb list dump: [0-1023]

vbucket curr owner:
Producer node: 127.0.0.1:9302	Node UUID: 442585dbdcccd496fd97d0ac70022334
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [0-113]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [114-227]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 114
	v: [228-341]
Producer node: 172.31.2.134:9300	Node UUID: bbbe093dbac20cc0f8c45b8df77d5e5a
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 113
	v: [911-1023]
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [683-796]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [797-910]
Producer node: 127.0.0.1:9303	Node UUID: e26e57bc28e5e3bcf77b18125cbe7409
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [342-455]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [456-569]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 113
	v: [570-682]


2021/12/15 06:02:57 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 06:02:57 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/15 06:03:02 Rebalance progress: 29.16666666666666
2021/12/15 06:03:07 Rebalance progress: 29.16666666666666
2021/12/15 06:03:12 Rebalance progress: 100

VB distribution across eventing nodes and their workers::


DCP Stream statuses:
	stream status: running
	len: 1024
	vb list dump: [0-1023]

vbucket curr owner:
Producer node: 127.0.0.1:9302	Node UUID: 442585dbdcccd496fd97d0ac70022334
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [114-227]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 114
	v: [228-341]
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [0-113]
Producer node: 172.31.2.134:9300	Node UUID: bbbe093dbac20cc0f8c45b8df77d5e5a
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 113
	v: [911-1023]
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [683-796]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [797-910]
Producer node: 127.0.0.1:9303	Node UUID: e26e57bc28e5e3bcf77b18125cbe7409
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [342-455]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [456-569]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 113
	v: [570-682]


2021/12/15 06:03:12 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 06:03:12 handler_test.go:1674: Deleting source bucket: default
2021/12/15 06:03:14 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/15 06:03:14 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/15 06:03:19 Rebalance progress: 29.16666666666666
2021/12/15 06:03:24 Rebalance progress: 29.16666666666666
2021/12/15 06:03:29 Rebalance progress: 100

VB distribution across eventing nodes and their workers::


DCP Stream statuses:
	stream status: running
	len: 902
	vb list dump: [0, 2-8, 11-12, 14-25, 27, 29-34, 36, 38, 40-45, 47-51, 53-63, 65, 67-69, 71-80, 82-109, 111-114, 116-121, 123-129, 131-139, 141, 143-151, 153-162, 164-172, 174-181, 183-184, 186-196, 198-208, 210-226, 228-234, 236-237, 239, 241-248, 250-255, 257-268, 270-280, 282, 284-312, 314-321, 323-332, 334-335, 337-340, 342-348, 350, 352-354, 356-383, 386-400, 402-415, 417-421, 423-426, 428-429, 431, 433-438, 440-453, 456-464, 467-474, 476-479, 484-485, 487, 489-491, 493-502, 504, 506, 508-570, 572-574, 576-588, 590-612, 614-625, 627-630, 632-634, 636-644, 646-663, 665-671, 673-676, 678, 680-690, 692-697, 699-701, 704-716, 718, 720-721, 723-732, 734-754, 756-757, 759-763, 765-768, 770-794, 796-801, 803-804, 806-807, 809-811, 813, 815-817, 819-822, 824-827, 829-840, 842-850, 853-860, 862-871, 874-882, 884-894, 896-936, 938-946, 948-949, 951, 954-955, 957-960, 962, 965-967, 969-999, 1001-1017, 1019-1023]
	stream status: stopped
	len: 122
	vb list dump: [1, 9-10, 13, 26, 28, 35, 37, 39, 46, 52, 64, 66, 70, 81, 110, 115, 122, 130, 140, 142, 152, 163, 173, 182, 185, 197, 209, 227, 235, 238, 240, 249, 256, 269, 281, 283, 313, 322, 333, 336, 341, 349, 351, 355, 384-385, 401, 416, 422, 427, 430, 432, 439, 454-455, 465-466, 475, 480-483, 486, 488, 492, 503, 505, 507, 571, 575, 589, 613, 626, 631, 635, 645, 664, 672, 677, 679, 691, 698, 702-703, 717, 719, 722, 733, 755, 758, 764, 769, 795, 802, 805, 808, 812, 814, 818, 823, 828, 841, 851-852, 861, 872-873, 883, 895, 937, 947, 950, 952-953, 956, 961, 963-964, 968, 1000, 1018]

vbucket curr owner:
Producer node: 127.0.0.1:9302	Node UUID: 442585dbdcccd496fd97d0ac70022334
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 98
	v: [0, 2-8, 11-12, 14-25, 27, 29-34, 36, 38, 40-45, 47-51, 53-63, 65, 67-69, 71-80, 82-109, 111-113]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 101
	v: [114, 116-121, 123-129, 131-139, 141, 143-151, 153-162, 164-172, 174-181, 183-184, 186-196, 198-208, 210-226]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 101
	v: [228-234, 236-237, 239, 241-248, 250-255, 257-268, 270-280, 282, 284-312, 314-321, 323-332, 334-335, 337-340]
Producer node: 172.31.2.134:9300	Node UUID: bbbe093dbac20cc0f8c45b8df77d5e5a
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 101
	v: [911-936, 938-946, 948-949, 951, 954-955, 957-960, 962, 965-967, 969-999, 1001-1017, 1019-1023]
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 101
	v: [683-690, 692-697, 699-701, 704-716, 718, 720-721, 723-732, 734-754, 756-757, 759-763, 765-768, 770-794, 796]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 98
	v: [797-801, 803-804, 806-807, 809-811, 813, 815-817, 819-822, 824-827, 829-840, 842-850, 853-860, 862-871, 874-882, 884-894, 896-910]
Producer node: 127.0.0.1:9303	Node UUID: e26e57bc28e5e3bcf77b18125cbe7409
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 100
	v: [342-348, 350, 352-354, 356-383, 386-400, 402-415, 417-421, 423-426, 428-429, 431, 433-438, 440-453]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 101
	v: [456-464, 467-474, 476-479, 484-485, 487, 489-491, 493-502, 504, 506, 508-569]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 101
	v: [570, 572-574, 576-588, 590-612, 614-625, 627-630, 632-634, 636-644, 646-663, 665-671, 673-676, 678, 680-682]


2021/12/15 06:03:30 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 06:03:35 eventingSetup.go:719: Waiting for function: TestBucketDeleteWithRebOut status to change to undeployed
2021/12/15 06:03:35 eventingSetup.go:727: Function: TestBucketDeleteWithRebOut status changed to undeployed
2021/12/15 06:03:40 clusterSetup.go:397: Waiting for app: TestBucketDeleteWithRebOut to get un-deployed
2021/12/15 06:03:40 clusterSetup.go:405: App: TestBucketDeleteWithRebOut got un-deployed
2021/12/15 06:03:40 eventingSetup.go:46: No eventing-con process running
2021/12/15 06:03:40 eventingSetup.go:418: Function: TestBucketDeleteWithRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestBucketDeleteWithRebOut deleting in the background"
}
2021/12/15 06:03:41 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 06:03:46 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 06:03:48 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 06:03:53 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 06:03:53 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv,index,n1ql to the cluster
2021/12/15 06:03:53 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/15 06:03:58 Rebalance progress: 100
--- PASS: TestBucketDeleteWithRebOut (461.95s)
=== RUN   TestN1QLRecursion
2021/12/15 06:04:01 Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/15 06:04:01 Error in adding nodes : , response: ["Prepare join failed. Node is already part of cluster."]
2021/12/15 06:04:01 Removing node(s): [] from the cluster
2021/12/15 06:04:06 Rebalance progress: 100
2021/12/15 06:04:12 Context: Bucket flush, response code: 200 dump: 
2021/12/15 06:04:18 Context: Bucket flush, response code: 200 dump: 
2021/12/15 06:04:24 Context: Bucket flush, response code: 200 dump: 
2021/12/15 06:04:29 Deploying app: TestN1QLRecursion_function1. Item count src bucket: 0 dst bucket: 0
2021/12/15 06:04:30 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLRecursion_function1' in metakv",
  "warnings": null
 }
}
2021/12/15 06:04:30 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLRecursion_function1' in metakv",
  "warnings": null
 }
} err 
2021/12/15 06:04:35 Waiting for app: TestN1QLRecursion_function1 to get deployed
2021/12/15 06:04:40 Waiting for app: TestN1QLRecursion_function1 to get deployed
2021/12/15 06:04:45 Waiting for app: TestN1QLRecursion_function1 to get deployed
2021/12/15 06:04:50 Waiting for app: TestN1QLRecursion_function1 to get deployed
2021/12/15 06:04:50 App: TestN1QLRecursion_function1 got deployed
2021/12/15 06:04:50 TestN1QLRecursion_function1 is deployed
2021/12/15 06:04:50 Deploying app: TestN1QLRecursion_function3. Item count src bucket: 0 dst bucket: 0
2021/12/15 06:04:52 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLRecursion_function3' in metakv",
  "warnings": null
 }
}
2021/12/15 06:04:52 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLRecursion_function3' in metakv",
  "warnings": null
 }
} err 
2021/12/15 06:04:57 Waiting for app: TestN1QLRecursion_function3 to get deployed
2021/12/15 06:05:02 Waiting for app: TestN1QLRecursion_function3 to get deployed
2021/12/15 06:05:07 Waiting for app: TestN1QLRecursion_function3 to get deployed
2021/12/15 06:05:12 Waiting for app: TestN1QLRecursion_function3 to get deployed
2021/12/15 06:05:12 App: TestN1QLRecursion_function3 got deployed
2021/12/15 06:05:12 TestN1QLRecursion_function3 is deployed
2021/12/15 06:05:12 Deploying app: TestN1QLRecursion_function2. Item count src bucket: 0 dst bucket: 0
2021/12/15 06:05:12 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_INTER_BUCKET_RECURSION",
 "code": 51,
 "description": "Inter bucket recursion error, deployment of current handler will cause inter bucket recursion",
 "attributes": null,
 "runtime_info": {
  "code": 51,
  "info": "Inter bucket recursion error; function: TestN1QLRecursion_function2 causes a cycle involving functions: [TestN1QLRecursion_function3 TestN1QLRecursion_function1], hence deployment is disallowed"
 }
}
2021/12/15 06:05:12 response body {
 "name": "ERR_INTER_BUCKET_RECURSION",
 "code": 51,
 "description": "Inter bucket recursion error, deployment of current handler will cause inter bucket recursion",
 "attributes": null,
 "runtime_info": {
  "code": 51,
  "info": "Inter bucket recursion error; function: TestN1QLRecursion_function2 causes a cycle involving functions: [TestN1QLRecursion_function3 TestN1QLRecursion_function1], hence deployment is disallowed"
 }
} err 
2021/12/15 06:05:12 Success: TestN1QLRecursion_function2 is not deployed
2021/12/15 06:05:12 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 06:05:17 Waiting for app: TestN1QLRecursion_function1 to get un-deployed
2021/12/15 06:05:22 Waiting for app: TestN1QLRecursion_function1 to get un-deployed
2021/12/15 06:05:27 Waiting for app: TestN1QLRecursion_function1 to get un-deployed
2021/12/15 06:05:32 Waiting for app: TestN1QLRecursion_function1 to get un-deployed
2021/12/15 06:05:32 App: TestN1QLRecursion_function1 got un-deployed
2021/12/15 06:05:32 No eventing-con process running
2021/12/15 06:05:32 Function: TestN1QLRecursion_function1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLRecursion_function1 deleting in the background"
}
2021/12/15 06:05:34 Context: Bucket flush, response code: 200 dump: 
2021/12/15 06:05:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 06:05:41 Context: Bucket flush, response code: 200 dump: 
2021/12/15 06:05:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 06:05:46 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 06:05:51 Waiting for app: TestN1QLRecursion_function3 to get un-deployed
2021/12/15 06:05:56 Waiting for app: TestN1QLRecursion_function3 to get un-deployed
2021/12/15 06:06:01 Waiting for app: TestN1QLRecursion_function3 to get un-deployed
2021/12/15 06:06:06 Waiting for app: TestN1QLRecursion_function3 to get un-deployed
2021/12/15 06:06:06 App: TestN1QLRecursion_function3 got un-deployed
2021/12/15 06:06:06 No eventing-con process running
2021/12/15 06:06:06 Function: TestN1QLRecursion_function3 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLRecursion_function3 deleting in the background"
}
2021/12/15 06:06:08 Context: Bucket flush, response code: 200 dump: 
2021/12/15 06:06:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 06:06:15 Context: Bucket flush, response code: 200 dump: 
2021/12/15 06:06:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 06:06:21 Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/15 06:06:26 Rebalance progress: 100
--- PASS: TestN1QLRecursion (145.37s)
=== RUN   TestN1QLAllowRecursion
2021/12/15 06:06:26 Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/15 06:06:26 Error in adding nodes : , response: ["Prepare join failed. Node is already part of cluster."]
2021/12/15 06:06:26 Removing node(s): [] from the cluster
2021/12/15 06:06:31 Rebalance progress: 100
2021/12/15 06:06:37 Context: Bucket flush, response code: 200 dump: 
2021/12/15 06:06:43 Context: Bucket flush, response code: 200 dump: 
2021/12/15 06:06:49 Context: Bucket flush, response code: 200 dump: 
2021/12/15 06:06:54 Deploying app: TestN1QLAllowRecursion_function1. Item count src bucket: 0 dst bucket: 0
2021/12/15 06:06:55 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function1' in metakv",
  "warnings": null
 }
}
2021/12/15 06:06:55 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function1' in metakv",
  "warnings": null
 }
} err 
2021/12/15 06:07:00 Waiting for app: TestN1QLAllowRecursion_function1 to get deployed
2021/12/15 06:07:05 Waiting for app: TestN1QLAllowRecursion_function1 to get deployed
2021/12/15 06:07:10 Waiting for app: TestN1QLAllowRecursion_function1 to get deployed
2021/12/15 06:07:15 Waiting for app: TestN1QLAllowRecursion_function1 to get deployed
2021/12/15 06:07:15 App: TestN1QLAllowRecursion_function1 got deployed
2021/12/15 06:07:15 TestN1QLAllowRecursion_function1 is deployed
2021/12/15 06:07:15 Deploying app: TestN1QLAllowRecursion_function3. Item count src bucket: 0 dst bucket: 0
2021/12/15 06:07:16 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function3' in metakv",
  "warnings": null
 }
}
2021/12/15 06:07:16 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function3' in metakv",
  "warnings": null
 }
} err 
2021/12/15 06:07:21 Waiting for app: TestN1QLAllowRecursion_function3 to get deployed
2021/12/15 06:07:26 Waiting for app: TestN1QLAllowRecursion_function3 to get deployed
2021/12/15 06:07:31 Waiting for app: TestN1QLAllowRecursion_function3 to get deployed
2021/12/15 06:07:36 Waiting for app: TestN1QLAllowRecursion_function3 to get deployed
2021/12/15 06:07:36 App: TestN1QLAllowRecursion_function3 got deployed
2021/12/15 06:07:36 TestN1QLAllowRecursion_function3 is deployed
2021/12/15 06:07:36 Deploying app: TestN1QLAllowRecursion_function2. Item count src bucket: 0 dst bucket: 0
2021/12/15 06:07:37 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function2' in metakv",
  "warnings": null
 }
}
2021/12/15 06:07:37 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function2' in metakv",
  "warnings": null
 }
} err 
2021/12/15 06:07:42 Waiting for app: TestN1QLAllowRecursion_function2 to get deployed
2021/12/15 06:07:47 Waiting for app: TestN1QLAllowRecursion_function2 to get deployed
2021/12/15 06:07:52 Waiting for app: TestN1QLAllowRecursion_function2 to get deployed
2021/12/15 06:07:57 Waiting for app: TestN1QLAllowRecursion_function2 to get deployed
2021/12/15 06:07:57 App: TestN1QLAllowRecursion_function2 got deployed
2021/12/15 06:07:57 Success: TestN1QLAllowRecursion_function2 is deployed after allow_interbucket_recursion
2021/12/15 06:07:58 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 06:08:03 Waiting for app: TestN1QLAllowRecursion_function1 to get un-deployed
2021/12/15 06:08:08 Waiting for app: TestN1QLAllowRecursion_function1 to get un-deployed
2021/12/15 06:08:13 Waiting for app: TestN1QLAllowRecursion_function1 to get un-deployed
2021/12/15 06:08:18 Waiting for app: TestN1QLAllowRecursion_function1 to get un-deployed
2021/12/15 06:08:18 App: TestN1QLAllowRecursion_function1 got un-deployed
2021/12/15 06:08:18 No eventing-con process running
2021/12/15 06:08:18 Function: TestN1QLAllowRecursion_function1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLAllowRecursion_function1 deleting in the background"
}
2021/12/15 06:08:20 Context: Bucket flush, response code: 200 dump: 
2021/12/15 06:08:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 06:08:26 Context: Bucket flush, response code: 200 dump: 
2021/12/15 06:08:31 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 06:08:32 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 06:08:37 Waiting for app: TestN1QLAllowRecursion_function2 to get un-deployed
2021/12/15 06:08:42 Waiting for app: TestN1QLAllowRecursion_function2 to get un-deployed
2021/12/15 06:08:47 Waiting for app: TestN1QLAllowRecursion_function2 to get un-deployed
2021/12/15 06:08:52 Waiting for app: TestN1QLAllowRecursion_function2 to get un-deployed
2021/12/15 06:08:52 App: TestN1QLAllowRecursion_function2 got un-deployed
2021/12/15 06:08:52 No eventing-con process running
2021/12/15 06:08:52 Function: TestN1QLAllowRecursion_function2 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLAllowRecursion_function2 deleting in the background"
}
2021/12/15 06:08:54 Context: Bucket flush, response code: 200 dump: 
2021/12/15 06:08:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 06:09:00 Context: Bucket flush, response code: 200 dump: 
2021/12/15 06:09:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 06:09:06 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 06:09:11 Waiting for app: TestN1QLAllowRecursion_function3 to get un-deployed
2021/12/15 06:09:16 Waiting for app: TestN1QLAllowRecursion_function3 to get un-deployed
2021/12/15 06:09:21 Waiting for app: TestN1QLAllowRecursion_function3 to get un-deployed
2021/12/15 06:09:26 Waiting for app: TestN1QLAllowRecursion_function3 to get un-deployed
2021/12/15 06:09:26 App: TestN1QLAllowRecursion_function3 got un-deployed
2021/12/15 06:09:26 No eventing-con process running
2021/12/15 06:09:26 Function: TestN1QLAllowRecursion_function3 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLAllowRecursion_function3 deleting in the background"
}
2021/12/15 06:09:28 Context: Bucket flush, response code: 200 dump: 
2021/12/15 06:09:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 06:09:34 Context: Bucket flush, response code: 200 dump: 
2021/12/15 06:09:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 06:09:41 Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/15 06:09:46 Rebalance progress: 100
--- PASS: TestN1QLAllowRecursion (199.64s)
=== RUN   TestOnDeleteExpiryBucketOp
2021/12/15 06:09:46 Sleeping for some time
2021/12/15 06:09:56 Waiting for app: TestOnDeleteExpiryBucketOp to get un-deployed
2021/12/15 06:09:56 App: TestOnDeleteExpiryBucketOp got un-deployed
2021/12/15 06:09:56 No eventing-con process running
2021/12/15 06:09:56 Function: TestOnDeleteExpiryBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnDeleteExpiryBucketOp not found"
 }
}
2021/12/15 06:09:58 Context: Bucket flush, response code: 200 dump: 
2021/12/15 06:10:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 06:10:04 Context: Bucket flush, response code: 200 dump: 
2021/12/15 06:10:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 06:10:09 Deploying app: TestOnDeleteExpiryBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/15 06:10:11 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnDeleteExpiryBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/15 06:10:11 Starting bucket ops to source bucket
2021/12/15 06:11:00 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 06:11:05 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 06:11:05 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/15 06:11:05 Starting bucket ops to source bucket
2021/12/15 06:11:06 Starting bucket ops to source bucket
2021/12/15 06:12:12 Waiting for dst bucket item count to get to: 7000 curr count: 5000
2021/12/15 06:12:12 src & dst bucket item count matched up. src bucket count: 7000 dst bucket count: 7000
2021/12/15 06:12:12 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 06:12:12 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 06:12:17 Waiting for app: TestOnDeleteExpiryBucketOp to get un-deployed
2021/12/15 06:12:22 Waiting for app: TestOnDeleteExpiryBucketOp to get un-deployed
2021/12/15 06:12:27 Waiting for app: TestOnDeleteExpiryBucketOp to get un-deployed
2021/12/15 06:12:32 Waiting for app: TestOnDeleteExpiryBucketOp to get un-deployed
2021/12/15 06:12:32 App: TestOnDeleteExpiryBucketOp got un-deployed
2021/12/15 06:12:32 No eventing-con process running
2021/12/15 06:12:32 Function: TestOnDeleteExpiryBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnDeleteExpiryBucketOp deleting in the background"
}
2021/12/15 06:12:34 Context: Bucket flush, response code: 200 dump: 
2021/12/15 06:12:44 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/15 06:12:49 Waiting for dst bucket item count to get to: 0 curr count: 501
2021/12/15 06:12:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 06:12:51 Context: Bucket flush, response code: 200 dump: 
2021/12/15 06:13:01 Waiting for dst bucket item count to get to: 0 curr count: 6051
2021/12/15 06:13:06 Waiting for dst bucket item count to get to: 0 curr count: 6051
2021/12/15 06:13:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnDeleteExpiryBucketOp (199.90s)
=== RUN   TestTimerOverWriteSameReference
2021/12/15 06:13:11 Waiting for app: TestTimerOverWriteSameReference to get un-deployed
2021/12/15 06:13:11 App: TestTimerOverWriteSameReference got un-deployed
2021/12/15 06:13:11 No eventing-con process running
2021/12/15 06:13:11 Function: TestTimerOverWriteSameReference delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestTimerOverWriteSameReference not found"
 }
}
2021/12/15 06:13:13 Context: Bucket flush, response code: 200 dump: 
2021/12/15 06:13:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 06:13:19 Context: Bucket flush, response code: 200 dump: 
2021/12/15 06:13:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 06:13:24 Deploying app: TestTimerOverWriteSameReference. Item count src bucket: 0 dst bucket: 0
2021/12/15 06:13:26 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestTimerOverWriteSameReference' in metakv",
  "warnings": null
 }
}
2021/12/15 06:13:31 Waiting for app: TestTimerOverWriteSameReference to get deployed
2021/12/15 06:13:36 Waiting for app: TestTimerOverWriteSameReference to get deployed
2021/12/15 06:13:41 Waiting for app: TestTimerOverWriteSameReference to get deployed
2021/12/15 06:13:46 Waiting for app: TestTimerOverWriteSameReference to get deployed
2021/12/15 06:13:46 App: TestTimerOverWriteSameReference got deployed
2021/12/15 06:13:56 Starting bucket ops to source bucket
2021/12/15 06:14:57 src & dst bucket item count matched up. src bucket count: 1 dst bucket count: 1
2021/12/15 06:15:07 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 06:15:07 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 06:15:12 Waiting for app: TestTimerOverWriteSameReference to get un-deployed
2021/12/15 06:15:17 Waiting for app: TestTimerOverWriteSameReference to get un-deployed
2021/12/15 06:15:22 Waiting for app: TestTimerOverWriteSameReference to get un-deployed
2021/12/15 06:15:27 Waiting for app: TestTimerOverWriteSameReference to get un-deployed
2021/12/15 06:15:27 App: TestTimerOverWriteSameReference got un-deployed
2021/12/15 06:15:27 No eventing-con process running
2021/12/15 06:15:27 Function: TestTimerOverWriteSameReference delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestTimerOverWriteSameReference deleting in the background"
}
2021/12/15 06:15:29 Context: Bucket flush, response code: 200 dump: 
2021/12/15 06:15:39 Waiting for dst bucket item count to get to: 0 curr count: 951
2021/12/15 06:15:44 Waiting for dst bucket item count to get to: 0 curr count: 451
2021/12/15 06:15:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 06:15:46 Context: Bucket flush, response code: 200 dump: 
2021/12/15 06:15:56 Waiting for dst bucket item count to get to: 0 curr count: 1
2021/12/15 06:15:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestTimerOverWriteSameReference (169.91s)
=== RUN   TestCancelTimerBucketop
2021/12/15 06:16:01 Waiting for app: TestCancelTimerBucketop to get un-deployed
2021/12/15 06:16:01 App: TestCancelTimerBucketop got un-deployed
2021/12/15 06:16:01 No eventing-con process running
2021/12/15 06:16:01 Function: TestCancelTimerBucketop delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestCancelTimerBucketop not found"
 }
}
2021/12/15 06:16:02 Context: Bucket flush, response code: 200 dump: 
2021/12/15 06:16:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 06:16:09 Context: Bucket flush, response code: 200 dump: 
2021/12/15 06:16:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 06:16:14 Deploying app: TestCancelTimerBucketop. Item count src bucket: 0 dst bucket: 0
2021/12/15 06:16:15 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCancelTimerBucketop' in metakv",
  "warnings": null
 }
}
2021/12/15 06:16:20 Waiting for app: TestCancelTimerBucketop to get deployed
2021/12/15 06:16:25 Waiting for app: TestCancelTimerBucketop to get deployed
2021/12/15 06:16:30 Waiting for app: TestCancelTimerBucketop to get deployed
2021/12/15 06:16:35 Waiting for app: TestCancelTimerBucketop to get deployed
2021/12/15 06:16:35 App: TestCancelTimerBucketop got deployed
2021/12/15 06:16:35 Starting bucket ops to source bucket
2021/12/15 06:16:43 Starting bucket ops to source bucket
2021/12/15 06:17:45 src & dst bucket item count matched up. src bucket count: 1000 dst bucket count: 1000
2021/12/15 06:17:45 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 06:17:45 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 06:17:50 Waiting for app: TestCancelTimerBucketop to get un-deployed
2021/12/15 06:17:55 Waiting for app: TestCancelTimerBucketop to get un-deployed
2021/12/15 06:18:00 Waiting for app: TestCancelTimerBucketop to get un-deployed
2021/12/15 06:18:05 Waiting for app: TestCancelTimerBucketop to get un-deployed
2021/12/15 06:18:05 App: TestCancelTimerBucketop got un-deployed
2021/12/15 06:18:05 No eventing-con process running
2021/12/15 06:18:05 Function: TestCancelTimerBucketop delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCancelTimerBucketop deleting in the background"
}
2021/12/15 06:18:07 Context: Bucket flush, response code: 200 dump: 
2021/12/15 06:18:17 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/15 06:18:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 06:18:19 Context: Bucket flush, response code: 200 dump: 
2021/12/15 06:18:29 Waiting for dst bucket item count to get to: 0 curr count: 945
2021/12/15 06:18:34 Waiting for dst bucket item count to get to: 0 curr count: 445
2021/12/15 06:18:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCancelTimerBucketop (158.19s)
=== RUN   TestJSExpiryDate
2021/12/15 06:18:39 Starting bucket ops to source bucket
2021/12/15 06:18:57 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/15 06:18:57 Deploying app: TestJSExpiryDate. Item count src bucket: 5000 dst bucket: 0
2021/12/15 06:18:58 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestJSExpiryDate' in metakv",
  "warnings": null
 }
}
2021/12/15 06:19:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 06:19:23 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 06:19:23 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 06:19:28 Waiting for app: TestJSExpiryDate to get un-deployed
2021/12/15 06:19:33 Waiting for app: TestJSExpiryDate to get un-deployed
2021/12/15 06:19:38 Waiting for app: TestJSExpiryDate to get un-deployed
2021/12/15 06:19:43 Waiting for app: TestJSExpiryDate to get un-deployed
2021/12/15 06:19:43 App: TestJSExpiryDate got un-deployed
2021/12/15 06:19:43 No eventing-con process running
2021/12/15 06:19:43 Function: TestJSExpiryDate delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestJSExpiryDate deleting in the background"
}
2021/12/15 06:19:45 Context: Bucket flush, response code: 200 dump: 
2021/12/15 06:19:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 06:19:52 Context: Bucket flush, response code: 200 dump: 
2021/12/15 06:19:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestJSExpiryDate (83.70s)
=== RUN   TestBinaryDoc
2021/12/15 06:20:02 Starting bucket ops to source bucket
2021/12/15 06:20:03 Deploying app: TestBinaryDoc. Item count src bucket: 0 dst bucket: 0
2021/12/15 06:20:04 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestBinaryDoc' in metakv",
  "warnings": null
 }
}
2021/12/15 06:20:04 Starting bucket ops to source bucket
2021/12/15 06:20:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 06:20:04 Deleting source bucket docs now
2021/12/15 06:20:04 Starting bucket ops to source bucket
2021/12/15 06:20:41 Waiting for dst bucket item count to get to: 32 curr count: 24
2021/12/15 06:20:46 Waiting for dst bucket item count to get to: 32 curr count: 28
2021/12/15 06:20:46 src & dst bucket item count matched up. src bucket count: 32 dst bucket count: 32
2021/12/15 06:20:46 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 06:20:46 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 06:20:51 Waiting for app: TestBinaryDoc to get un-deployed
2021/12/15 06:20:56 Waiting for app: TestBinaryDoc to get un-deployed
2021/12/15 06:21:01 Waiting for app: TestBinaryDoc to get un-deployed
2021/12/15 06:21:06 Waiting for app: TestBinaryDoc to get un-deployed
2021/12/15 06:21:06 App: TestBinaryDoc got un-deployed
2021/12/15 06:21:06 No eventing-con process running
2021/12/15 06:21:06 Function: TestBinaryDoc delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestBinaryDoc deleting in the background"
}
2021/12/15 06:21:08 Context: Bucket flush, response code: 200 dump: 
2021/12/15 06:21:18 Waiting for dst bucket item count to get to: 0 curr count: 21
2021/12/15 06:21:23 Waiting for dst bucket item count to get to: 0 curr count: 3
2021/12/15 06:21:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 06:21:25 Context: Bucket flush, response code: 200 dump: 
2021/12/15 06:21:35 Waiting for dst bucket item count to get to: 0 curr count: 32
2021/12/15 06:21:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestBinaryDoc (97.27s)
=== RUN   TestConstantBindings
2021/12/15 06:21:35 Deploying app: TestConstantBindings. Item count src bucket: 0 dst bucket: 0
2021/12/15 06:21:36 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestConstantBindings' in metakv",
  "warnings": null
 }
}
2021/12/15 06:21:36 Starting bucket ops to source bucket
2021/12/15 06:21:42 Waiting for dst bucket item count to get to: 6000 curr count: 0
2021/12/15 06:21:47 Waiting for dst bucket item count to get to: 6000 curr count: 0
2021/12/15 06:21:52 Waiting for dst bucket item count to get to: 6000 curr count: 0
2021/12/15 06:21:57 Waiting for dst bucket item count to get to: 6000 curr count: 2364
2021/12/15 06:22:02 Waiting for dst bucket item count to get to: 6000 curr count: 2364
2021/12/15 06:22:02 src & dst bucket item count matched up. src bucket count: 6000 dst bucket count: 6000
2021/12/15 06:22:02 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 06:22:02 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 06:22:07 Waiting for app: TestConstantBindings to get un-deployed
2021/12/15 06:22:12 Waiting for app: TestConstantBindings to get un-deployed
2021/12/15 06:22:17 Waiting for app: TestConstantBindings to get un-deployed
2021/12/15 06:22:22 Waiting for app: TestConstantBindings to get un-deployed
2021/12/15 06:22:22 App: TestConstantBindings got un-deployed
2021/12/15 06:22:22 No eventing-con process running
2021/12/15 06:22:22 Function: TestConstantBindings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestConstantBindings deleting in the background"
}
2021/12/15 06:22:24 Context: Bucket flush, response code: 200 dump: 
2021/12/15 06:22:34 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/15 06:22:39 Waiting for dst bucket item count to get to: 0 curr count: 500
2021/12/15 06:22:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 06:22:40 Context: Bucket flush, response code: 200 dump: 
2021/12/15 06:22:50 Waiting for dst bucket item count to get to: 0 curr count: 4860
2021/12/15 06:22:55 Waiting for dst bucket item count to get to: 0 curr count: 4860
2021/12/15 06:22:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestConstantBindings (80.70s)
=== RUN   TestCodeInsight
2021/12/15 06:23:00 Waiting for app: TestCodeInsight to get un-deployed
2021/12/15 06:23:00 App: TestCodeInsight got un-deployed
2021/12/15 06:23:00 No eventing-con process running
2021/12/15 06:23:00 Function: TestCodeInsight delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestCodeInsight not found"
 }
}
2021/12/15 06:23:02 Context: Bucket flush, response code: 200 dump: 
2021/12/15 06:23:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 06:23:09 Context: Bucket flush, response code: 200 dump: 
2021/12/15 06:23:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 06:23:14 Starting bucket ops to source bucket
2021/12/15 06:23:15 Deploying app: TestCodeInsight. Item count src bucket: 0 dst bucket: 0
2021/12/15 06:23:16 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCodeInsight' in metakv",
  "warnings": null
 }
}
2021/12/15 06:23:21 Waiting for app: TestCodeInsight to get deployed
2021/12/15 06:23:26 Waiting for app: TestCodeInsight to get deployed
2021/12/15 06:23:31 Waiting for app: TestCodeInsight to get deployed
2021/12/15 06:23:36 Waiting for app: TestCodeInsight to get deployed
2021/12/15 06:23:36 App: TestCodeInsight got deployed
2021/12/15 06:23:41 Waiting for dst bucket item count to get to: 1000 curr count: 543
2021/12/15 06:23:41 src & dst bucket item count matched up. src bucket count: 1000 dst bucket count: 1000
2021/12/15 06:23:41 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 06:23:46 Waiting for app: TestCodeInsight to get un-deployed
2021/12/15 06:23:51 Waiting for app: TestCodeInsight to get un-deployed
2021/12/15 06:23:56 Waiting for app: TestCodeInsight to get un-deployed
2021/12/15 06:24:01 Waiting for app: TestCodeInsight to get un-deployed
2021/12/15 06:24:01 App: TestCodeInsight got un-deployed
2021/12/15 06:24:01 No eventing-con process running
2021/12/15 06:24:01 Function: TestCodeInsight delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCodeInsight deleting in the background"
}
2021/12/15 06:24:03 Context: Bucket flush, response code: 200 dump: 
2021/12/15 06:24:13 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/15 06:24:18 Waiting for dst bucket item count to get to: 0 curr count: 500
2021/12/15 06:24:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 06:24:20 Context: Bucket flush, response code: 200 dump: 
2021/12/15 06:24:30 Waiting for dst bucket item count to get to: 0 curr count: 574
2021/12/15 06:24:35 Waiting for dst bucket item count to get to: 0 curr count: 574
2021/12/15 06:24:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCodeInsight (99.46s)
=== RUN   TestParserTransform
--- PASS: TestParserTransform (0.05s)
=== RUN   TestParserDetect
--- PASS: TestParserDetect (0.00s)
PASS
ok  	github.com/couchbase/eventing/tests/functional_tests	13157.686s

Functional tests took 3 hours, 39 minutes and 22 seconds.

N1ql tests

2021/12/15 06:24:45 Started n1ql
2021/12/15 06:24:56 Adding node: https://127.0.0.1:19001 with role: kv,index,n1ql to the cluster
2021/12/15 06:24:56 Error in adding nodes : , response: ["Prepare join failed. Node is already part of cluster."]
2021/12/15 06:24:56 Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/15 06:24:57 Removing node(s): [] from the cluster
2021/12/15 06:25:02 Rebalance progress: 100
=== RUN   TestRecursiveMutationN1QL
2021/12/15 06:25:07 Waiting for app: TestRecursiveMutationN1QL to get un-deployed
2021/12/15 06:25:07 App: TestRecursiveMutationN1QL got un-deployed
2021/12/15 06:25:07 No eventing-con process running
2021/12/15 06:25:07 Function: TestRecursiveMutationN1QL delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestRecursiveMutationN1QL not found"
 }
}
2021/12/15 06:25:09 Context: Bucket flush, response code: 200 dump: 
2021/12/15 06:25:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 06:25:15 Context: Bucket flush, response code: 200 dump: 
2021/12/15 06:25:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 06:25:20 Deploying app: TestRecursiveMutationN1QL. Item count src bucket: 0 dst bucket: 0
2021/12/15 06:25:20 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_INTER_BUCKET_RECURSION",
 "code": 51,
 "description": "Inter bucket recursion error, deployment of current handler will cause inter bucket recursion",
 "attributes": null,
 "runtime_info": {
  "code": 51,
  "info": "Function: TestRecursiveMutationN1QL N1QL DML to source keyspace default:default"
 }
}
--- PASS: TestRecursiveMutationN1QL (18.64s)
=== RUN   TestFlexReset1
2021/12/15 06:25:25 Waiting for app: TestFlexReset1 to get un-deployed
2021/12/15 06:25:25 App: TestFlexReset1 got un-deployed
2021/12/15 06:25:25 No eventing-con process running
2021/12/15 06:25:25 Function: TestFlexReset1 delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestFlexReset1 not found"
 }
}
2021/12/15 06:25:27 Context: Bucket flush, response code: 200 dump: 
2021/12/15 06:25:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 06:25:34 Context: Bucket flush, response code: 200 dump: 
2021/12/15 06:25:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 06:25:39 Starting bucket ops to source bucket
2021/12/15 06:25:40 Deploying app: TestFlexReset1. Item count src bucket: 0 dst bucket: 0
2021/12/15 06:25:41 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestFlexReset1' in metakv",
  "warnings": null
 }
}
2021/12/15 06:25:46 Waiting for app: TestFlexReset1 to get deployed
2021/12/15 06:25:51 Waiting for app: TestFlexReset1 to get deployed
2021/12/15 06:25:56 Waiting for app: TestFlexReset1 to get deployed
2021/12/15 06:26:01 Waiting for app: TestFlexReset1 to get deployed
2021/12/15 06:26:01 App: TestFlexReset1 got deployed
2021/12/15 06:26:01 src & dst bucket item count matched up. src bucket count: 1000 dst bucket count: 1000
2021/12/15 06:26:01 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 06:26:01 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 06:26:06 Waiting for app: TestFlexReset1 to get un-deployed
2021/12/15 06:26:11 Waiting for app: TestFlexReset1 to get un-deployed
2021/12/15 06:26:16 Waiting for app: TestFlexReset1 to get un-deployed
2021/12/15 06:26:21 Waiting for app: TestFlexReset1 to get un-deployed
2021/12/15 06:26:21 App: TestFlexReset1 got un-deployed
2021/12/15 06:26:21 No eventing-con process running
2021/12/15 06:26:21 Function: TestFlexReset1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestFlexReset1 deleting in the background"
}
2021/12/15 06:26:23 Context: Bucket flush, response code: 200 dump: 
2021/12/15 06:26:33 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/15 06:26:38 Waiting for dst bucket item count to get to: 0 curr count: 500
2021/12/15 06:26:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 06:26:40 Context: Bucket flush, response code: 200 dump: 
2021/12/15 06:26:50 Waiting for dst bucket item count to get to: 0 curr count: 503
2021/12/15 06:26:55 Waiting for dst bucket item count to get to: 0 curr count: 503
2021/12/15 06:26:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestFlexReset1 (94.35s)
=== RUN   TestFlexReset2
2021/12/15 06:27:00 Waiting for app: TestFlexReset2 to get un-deployed
2021/12/15 06:27:00 App: TestFlexReset2 got un-deployed
2021/12/15 06:27:00 No eventing-con process running
2021/12/15 06:27:00 Function: TestFlexReset2 delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestFlexReset2 not found"
 }
}
2021/12/15 06:27:02 Context: Bucket flush, response code: 200 dump: 
2021/12/15 06:27:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 06:27:08 Context: Bucket flush, response code: 200 dump: 
2021/12/15 06:27:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 06:27:13 Starting bucket ops to source bucket
2021/12/15 06:27:14 Deploying app: TestFlexReset2. Item count src bucket: 0 dst bucket: 0
2021/12/15 06:27:15 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestFlexReset2' in metakv",
  "warnings": null
 }
}
2021/12/15 06:27:20 Waiting for app: TestFlexReset2 to get deployed
2021/12/15 06:27:25 Waiting for app: TestFlexReset2 to get deployed
2021/12/15 06:27:30 Waiting for app: TestFlexReset2 to get deployed
2021/12/15 06:27:35 Waiting for app: TestFlexReset2 to get deployed
2021/12/15 06:27:35 App: TestFlexReset2 got deployed
2021/12/15 06:27:40 Waiting for dst bucket item count to get to: 1000 curr count: 298
2021/12/15 06:27:45 Waiting for dst bucket item count to get to: 1000 curr count: 789
2021/12/15 06:27:45 src & dst bucket item count matched up. src bucket count: 1000 dst bucket count: 1000
2021/12/15 06:27:45 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 06:27:45 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 06:27:50 Waiting for app: TestFlexReset2 to get un-deployed
2021/12/15 06:27:55 Waiting for app: TestFlexReset2 to get un-deployed
2021/12/15 06:28:00 Waiting for app: TestFlexReset2 to get un-deployed
2021/12/15 06:28:05 Waiting for app: TestFlexReset2 to get un-deployed
2021/12/15 06:28:05 App: TestFlexReset2 got un-deployed
2021/12/15 06:28:05 No eventing-con process running
2021/12/15 06:28:05 Function: TestFlexReset2 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestFlexReset2 deleting in the background"
}
2021/12/15 06:28:07 Context: Bucket flush, response code: 200 dump: 
2021/12/15 06:28:17 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/15 06:28:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 06:28:19 Context: Bucket flush, response code: 200 dump: 
2021/12/15 06:28:29 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/15 06:28:34 Waiting for dst bucket item count to get to: 0 curr count: 500
2021/12/15 06:28:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestFlexReset2 (99.18s)
=== RUN   TestN1QLLabelledBreak
2021/12/15 06:28:39 Waiting for app: TestN1QLLabelledBreak to get un-deployed
2021/12/15 06:28:39 App: TestN1QLLabelledBreak got un-deployed
2021/12/15 06:28:39 No eventing-con process running
2021/12/15 06:28:39 Function: TestN1QLLabelledBreak delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLLabelledBreak not found"
 }
}
2021/12/15 06:28:41 Context: Bucket flush, response code: 200 dump: 
2021/12/15 06:28:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 06:28:48 Context: Bucket flush, response code: 200 dump: 
2021/12/15 06:28:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 06:28:53 Deploying app: TestN1QLLabelledBreak. Item count src bucket: 0 dst bucket: 0
2021/12/15 06:28:54 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLLabelledBreak' in metakv",
  "warnings": null
 }
}
2021/12/15 06:28:59 Waiting for app: TestN1QLLabelledBreak to get deployed
2021/12/15 06:29:04 Waiting for app: TestN1QLLabelledBreak to get deployed
2021/12/15 06:29:09 Waiting for app: TestN1QLLabelledBreak to get deployed
2021/12/15 06:29:14 Waiting for app: TestN1QLLabelledBreak to get deployed
2021/12/15 06:29:14 App: TestN1QLLabelledBreak got deployed
2021/12/15 06:29:14 Starting bucket ops to source bucket
2021/12/15 06:29:23 Waiting for dst bucket item count to get to: 10000 curr count: 0
2021/12/15 06:29:28 Waiting for dst bucket item count to get to: 10000 curr count: 242
2021/12/15 06:29:33 Waiting for dst bucket item count to get to: 10000 curr count: 764
2021/12/15 06:29:38 Waiting for dst bucket item count to get to: 10000 curr count: 2682
2021/12/15 06:29:43 Waiting for dst bucket item count to get to: 10000 curr count: 4755
2021/12/15 06:29:48 Waiting for dst bucket item count to get to: 10000 curr count: 6783
2021/12/15 06:29:53 Waiting for dst bucket item count to get to: 10000 curr count: 8776
2021/12/15 06:29:58 Waiting for dst bucket item count to get to: 10000 curr count: 9588
2021/12/15 06:29:58 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/15 06:29:58 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 06:29:58 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 06:30:03 Waiting for app: TestN1QLLabelledBreak to get un-deployed
2021/12/15 06:30:08 Waiting for app: TestN1QLLabelledBreak to get un-deployed
2021/12/15 06:30:13 Waiting for app: TestN1QLLabelledBreak to get un-deployed
2021/12/15 06:30:18 Waiting for app: TestN1QLLabelledBreak to get un-deployed
2021/12/15 06:30:18 App: TestN1QLLabelledBreak got un-deployed
2021/12/15 06:30:18 No eventing-con process running
2021/12/15 06:30:18 Function: TestN1QLLabelledBreak delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLLabelledBreak deleting in the background"
}
2021/12/15 06:30:20 Context: Bucket flush, response code: 200 dump: 
2021/12/15 06:30:30 Waiting for dst bucket item count to get to: 0 curr count: 3274
2021/12/15 06:30:35 Waiting for dst bucket item count to get to: 0 curr count: 3274
2021/12/15 06:30:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 06:30:37 Context: Bucket flush, response code: 200 dump: 
2021/12/15 06:30:47 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/15 06:30:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLLabelledBreak (132.99s)
=== RUN   TestN1QLUnlabelledBreak
2021/12/15 06:30:52 Waiting for app: TestN1QLUnlabelledBreak to get un-deployed
2021/12/15 06:30:52 App: TestN1QLUnlabelledBreak got un-deployed
2021/12/15 06:30:52 No eventing-con process running
2021/12/15 06:30:52 Function: TestN1QLUnlabelledBreak delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLUnlabelledBreak not found"
 }
}
2021/12/15 06:30:54 Context: Bucket flush, response code: 200 dump: 
2021/12/15 06:30:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 06:31:01 Context: Bucket flush, response code: 200 dump: 
2021/12/15 06:31:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 06:31:06 Deploying app: TestN1QLUnlabelledBreak. Item count src bucket: 0 dst bucket: 0
2021/12/15 06:31:07 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLUnlabelledBreak' in metakv",
  "warnings": null
 }
}
2021/12/15 06:31:12 Waiting for app: TestN1QLUnlabelledBreak to get deployed
2021/12/15 06:31:17 Waiting for app: TestN1QLUnlabelledBreak to get deployed
2021/12/15 06:31:22 Waiting for app: TestN1QLUnlabelledBreak to get deployed
2021/12/15 06:31:27 Waiting for app: TestN1QLUnlabelledBreak to get deployed
2021/12/15 06:31:27 App: TestN1QLUnlabelledBreak got deployed
2021/12/15 06:31:27 Starting bucket ops to source bucket
2021/12/15 06:31:37 Waiting for dst bucket item count to get to: 10000 curr count: 260
2021/12/15 06:31:42 Waiting for dst bucket item count to get to: 10000 curr count: 260
2021/12/15 06:31:47 Waiting for dst bucket item count to get to: 10000 curr count: 4784
2021/12/15 06:31:52 Waiting for dst bucket item count to get to: 10000 curr count: 4784
2021/12/15 06:31:57 Waiting for dst bucket item count to get to: 10000 curr count: 4784
2021/12/15 06:32:02 Waiting for dst bucket item count to get to: 10000 curr count: 9956
2021/12/15 06:32:07 Waiting for dst bucket item count to get to: 10000 curr count: 9956
2021/12/15 06:32:07 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/15 06:32:07 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 06:32:07 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 06:32:12 Waiting for app: TestN1QLUnlabelledBreak to get un-deployed
2021/12/15 06:32:17 Waiting for app: TestN1QLUnlabelledBreak to get un-deployed
2021/12/15 06:32:22 Waiting for app: TestN1QLUnlabelledBreak to get un-deployed
2021/12/15 06:32:27 Waiting for app: TestN1QLUnlabelledBreak to get un-deployed
2021/12/15 06:32:27 App: TestN1QLUnlabelledBreak got un-deployed
2021/12/15 06:32:27 No eventing-con process running
2021/12/15 06:32:27 Function: TestN1QLUnlabelledBreak delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLUnlabelledBreak deleting in the background"
}
2021/12/15 06:32:29 Context: Bucket flush, response code: 200 dump: 
2021/12/15 06:32:39 Waiting for dst bucket item count to get to: 0 curr count: 4666
2021/12/15 06:32:44 Waiting for dst bucket item count to get to: 0 curr count: 2166
2021/12/15 06:32:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 06:32:46 Context: Bucket flush, response code: 200 dump: 
2021/12/15 06:32:56 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/15 06:32:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLUnlabelledBreak (128.57s)
=== RUN   TestN1QLThrowStatement
2021/12/15 06:33:01 Waiting for app: TestN1QLThrowStatement to get un-deployed
2021/12/15 06:33:01 App: TestN1QLThrowStatement got un-deployed
2021/12/15 06:33:01 No eventing-con process running
2021/12/15 06:33:01 Function: TestN1QLThrowStatement delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLThrowStatement not found"
 }
}
2021/12/15 06:33:02 Context: Bucket flush, response code: 200 dump: 
2021/12/15 06:33:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 06:33:09 Context: Bucket flush, response code: 200 dump: 
2021/12/15 06:33:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 06:33:14 Deploying app: TestN1QLThrowStatement. Item count src bucket: 0 dst bucket: 0
2021/12/15 06:33:15 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLThrowStatement' in metakv",
  "warnings": null
 }
}
2021/12/15 06:33:20 Waiting for app: TestN1QLThrowStatement to get deployed
2021/12/15 06:33:25 Waiting for app: TestN1QLThrowStatement to get deployed
2021/12/15 06:33:30 Waiting for app: TestN1QLThrowStatement to get deployed
2021/12/15 06:33:35 Waiting for app: TestN1QLThrowStatement to get deployed
2021/12/15 06:33:35 App: TestN1QLThrowStatement got deployed
2021/12/15 06:33:35 Starting bucket ops to source bucket
2021/12/15 06:33:45 Waiting for dst bucket item count to get to: 10000 curr count: 0
2021/12/15 06:33:50 Waiting for dst bucket item count to get to: 10000 curr count: 390
2021/12/15 06:33:55 Waiting for dst bucket item count to get to: 10000 curr count: 390
2021/12/15 06:34:00 Waiting for dst bucket item count to get to: 10000 curr count: 3985
2021/12/15 06:34:05 Waiting for dst bucket item count to get to: 10000 curr count: 3985
2021/12/15 06:34:10 Waiting for dst bucket item count to get to: 10000 curr count: 8076
2021/12/15 06:34:15 Waiting for dst bucket item count to get to: 10000 curr count: 8076
2021/12/15 06:34:15 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/15 06:34:15 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 06:34:15 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 06:34:20 Waiting for app: TestN1QLThrowStatement to get un-deployed
2021/12/15 06:34:25 Waiting for app: TestN1QLThrowStatement to get un-deployed
2021/12/15 06:34:30 Waiting for app: TestN1QLThrowStatement to get un-deployed
2021/12/15 06:34:35 Waiting for app: TestN1QLThrowStatement to get un-deployed
2021/12/15 06:34:35 App: TestN1QLThrowStatement got un-deployed
2021/12/15 06:34:35 No eventing-con process running
2021/12/15 06:34:35 Function: TestN1QLThrowStatement delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLThrowStatement deleting in the background"
}
2021/12/15 06:34:37 Context: Bucket flush, response code: 200 dump: 
2021/12/15 06:34:47 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/15 06:34:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 06:34:49 Context: Bucket flush, response code: 200 dump: 
2021/12/15 06:34:59 Waiting for dst bucket item count to get to: 0 curr count: 9322
2021/12/15 06:35:04 Waiting for dst bucket item count to get to: 0 curr count: 4322
2021/12/15 06:35:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLThrowStatement (128.15s)
=== RUN   TestN1QLNestedForLoop
2021/12/15 06:35:09 Waiting for app: TestN1QLNestedForLoop to get un-deployed
2021/12/15 06:35:09 App: TestN1QLNestedForLoop got un-deployed
2021/12/15 06:35:09 No eventing-con process running
2021/12/15 06:35:09 Function: TestN1QLNestedForLoop delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLNestedForLoop not found"
 }
}
2021/12/15 06:35:11 Context: Bucket flush, response code: 200 dump: 
2021/12/15 06:35:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 06:35:17 Context: Bucket flush, response code: 200 dump: 
2021/12/15 06:35:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 06:35:22 Deploying app: TestN1QLNestedForLoop. Item count src bucket: 0 dst bucket: 0
2021/12/15 06:35:23 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLNestedForLoop' in metakv",
  "warnings": null
 }
}
2021/12/15 06:35:28 Waiting for app: TestN1QLNestedForLoop to get deployed
2021/12/15 06:35:33 Waiting for app: TestN1QLNestedForLoop to get deployed
2021/12/15 06:35:38 Waiting for app: TestN1QLNestedForLoop to get deployed
2021/12/15 06:35:43 Waiting for app: TestN1QLNestedForLoop to get deployed
2021/12/15 06:35:43 App: TestN1QLNestedForLoop got deployed
2021/12/15 06:35:43 Starting bucket ops to source bucket
2021/12/15 06:35:53 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 06:35:58 Waiting for dst bucket item count to get to: 5000 curr count: 130
2021/12/15 06:36:03 Waiting for dst bucket item count to get to: 5000 curr count: 430
2021/12/15 06:36:08 Waiting for dst bucket item count to get to: 5000 curr count: 1560
2021/12/15 06:36:13 Waiting for dst bucket item count to get to: 5000 curr count: 2770
2021/12/15 06:36:18 Waiting for dst bucket item count to get to: 5000 curr count: 3967
2021/12/15 06:36:23 Waiting for dst bucket item count to get to: 5000 curr count: 4957
2021/12/15 06:36:23 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/15 06:36:23 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 06:36:23 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 06:36:28 Waiting for app: TestN1QLNestedForLoop to get un-deployed
2021/12/15 06:36:33 Waiting for app: TestN1QLNestedForLoop to get un-deployed
2021/12/15 06:36:38 Waiting for app: TestN1QLNestedForLoop to get un-deployed
2021/12/15 06:36:43 Waiting for app: TestN1QLNestedForLoop to get un-deployed
2021/12/15 06:36:43 App: TestN1QLNestedForLoop got un-deployed
2021/12/15 06:36:43 No eventing-con process running
2021/12/15 06:36:43 Function: TestN1QLNestedForLoop delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLNestedForLoop deleting in the background"
}
2021/12/15 06:36:45 Context: Bucket flush, response code: 200 dump: 
2021/12/15 06:36:55 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/15 06:36:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 06:36:57 Context: Bucket flush, response code: 200 dump: 
2021/12/15 06:37:07 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/15 06:37:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLNestedForLoop (123.19s)
=== RUN   TestN1QLAutoClose
2021/12/15 06:37:12 Waiting for app: TestN1QLAutoClose to get un-deployed
2021/12/15 06:37:12 App: TestN1QLAutoClose got un-deployed
2021/12/15 06:37:12 No eventing-con process running
2021/12/15 06:37:12 Function: TestN1QLAutoClose delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLAutoClose not found"
 }
}
2021/12/15 06:37:14 Context: Bucket flush, response code: 200 dump: 
2021/12/15 06:37:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 06:37:20 Context: Bucket flush, response code: 200 dump: 
2021/12/15 06:37:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 06:37:25 Deploying app: TestN1QLAutoClose. Item count src bucket: 0 dst bucket: 0
2021/12/15 06:37:27 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAutoClose' in metakv",
  "warnings": null
 }
}
2021/12/15 06:37:32 Waiting for app: TestN1QLAutoClose to get deployed
2021/12/15 06:37:37 Waiting for app: TestN1QLAutoClose to get deployed
2021/12/15 06:37:42 Waiting for app: TestN1QLAutoClose to get deployed
2021/12/15 06:37:47 Waiting for app: TestN1QLAutoClose to get deployed
2021/12/15 06:37:47 App: TestN1QLAutoClose got deployed
2021/12/15 06:37:47 Starting bucket ops to source bucket
2021/12/15 06:37:52 Waiting for dst bucket item count to get to: 500 curr count: 0
2021/12/15 06:37:57 Waiting for dst bucket item count to get to: 500 curr count: 1
2021/12/15 06:38:02 Waiting for dst bucket item count to get to: 500 curr count: 1
2021/12/15 06:38:07 Waiting for dst bucket item count to get to: 500 curr count: 24
2021/12/15 06:38:12 Waiting for dst bucket item count to get to: 500 curr count: 24
2021/12/15 06:38:17 Waiting for dst bucket item count to get to: 500 curr count: 45
2021/12/15 06:38:22 Waiting for dst bucket item count to get to: 500 curr count: 45
2021/12/15 06:38:27 Waiting for dst bucket item count to get to: 500 curr count: 68
2021/12/15 06:38:32 Waiting for dst bucket item count to get to: 500 curr count: 68
2021/12/15 06:38:37 Waiting for dst bucket item count to get to: 500 curr count: 90
2021/12/15 06:38:42 Waiting for dst bucket item count to get to: 500 curr count: 90
2021/12/15 06:38:47 Waiting for dst bucket item count to get to: 500 curr count: 110
2021/12/15 06:38:52 Waiting for dst bucket item count to get to: 500 curr count: 110
2021/12/15 06:38:57 Waiting for dst bucket item count to get to: 500 curr count: 110
2021/12/15 06:39:02 Waiting for dst bucket item count to get to: 500 curr count: 132
2021/12/15 06:39:07 Waiting for dst bucket item count to get to: 500 curr count: 132
2021/12/15 06:39:12 Waiting for dst bucket item count to get to: 500 curr count: 153
2021/12/15 06:39:17 Waiting for dst bucket item count to get to: 500 curr count: 164
2021/12/15 06:39:22 Waiting for dst bucket item count to get to: 500 curr count: 173
2021/12/15 06:39:27 Waiting for dst bucket item count to get to: 500 curr count: 183
2021/12/15 06:39:32 Waiting for dst bucket item count to get to: 500 curr count: 196
2021/12/15 06:39:37 Waiting for dst bucket item count to get to: 500 curr count: 207
2021/12/15 06:39:42 Waiting for dst bucket item count to get to: 500 curr count: 218
2021/12/15 06:39:47 Waiting for dst bucket item count to get to: 500 curr count: 228
2021/12/15 06:39:52 Waiting for dst bucket item count to get to: 500 curr count: 240
2021/12/15 06:39:57 Waiting for dst bucket item count to get to: 500 curr count: 249
2021/12/15 06:40:02 Waiting for dst bucket item count to get to: 500 curr count: 257
2021/12/15 06:40:07 Waiting for dst bucket item count to get to: 500 curr count: 266
2021/12/15 06:40:13 Waiting for dst bucket item count to get to: 500 curr count: 280
2021/12/15 06:40:18 Waiting for dst bucket item count to get to: 500 curr count: 293
2021/12/15 06:40:23 Waiting for dst bucket item count to get to: 500 curr count: 302
2021/12/15 06:40:28 Waiting for dst bucket item count to get to: 500 curr count: 314
2021/12/15 06:40:33 Waiting for dst bucket item count to get to: 500 curr count: 324
2021/12/15 06:40:38 Waiting for dst bucket item count to get to: 500 curr count: 336
2021/12/15 06:40:43 Waiting for dst bucket item count to get to: 500 curr count: 345
2021/12/15 06:40:48 Waiting for dst bucket item count to get to: 500 curr count: 363
2021/12/15 06:40:53 Waiting for dst bucket item count to get to: 500 curr count: 363
2021/12/15 06:40:58 Waiting for dst bucket item count to get to: 500 curr count: 384
2021/12/15 06:41:03 Waiting for dst bucket item count to get to: 500 curr count: 384
2021/12/15 06:41:08 Waiting for dst bucket item count to get to: 500 curr count: 406
2021/12/15 06:41:13 Waiting for dst bucket item count to get to: 500 curr count: 406
2021/12/15 06:41:18 Waiting for dst bucket item count to get to: 500 curr count: 427
2021/12/15 06:41:23 Waiting for dst bucket item count to get to: 500 curr count: 427
2021/12/15 06:41:28 Waiting for dst bucket item count to get to: 500 curr count: 451
2021/12/15 06:41:33 Waiting for dst bucket item count to get to: 500 curr count: 451
2021/12/15 06:41:38 Waiting for dst bucket item count to get to: 500 curr count: 470
2021/12/15 06:41:43 Waiting for dst bucket item count to get to: 500 curr count: 470
2021/12/15 06:41:48 Waiting for dst bucket item count to get to: 500 curr count: 491
2021/12/15 06:41:53 Waiting for dst bucket item count to get to: 500 curr count: 491
2021/12/15 06:41:53 src & dst bucket item count matched up. src bucket count: 500 dst bucket count: 500
2021/12/15 06:41:53 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 06:41:53 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 06:41:58 Waiting for app: TestN1QLAutoClose to get un-deployed
2021/12/15 06:42:03 Waiting for app: TestN1QLAutoClose to get un-deployed
2021/12/15 06:42:08 Waiting for app: TestN1QLAutoClose to get un-deployed
2021/12/15 06:42:13 Waiting for app: TestN1QLAutoClose to get un-deployed
2021/12/15 06:42:13 App: TestN1QLAutoClose got un-deployed
2021/12/15 06:42:13 No eventing-con process running
2021/12/15 06:42:13 Function: TestN1QLAutoClose delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLAutoClose deleting in the background"
}
2021/12/15 06:42:15 Context: Bucket flush, response code: 200 dump: 
2021/12/15 06:42:25 Waiting for dst bucket item count to get to: 0 curr count: 500
2021/12/15 06:42:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 06:42:26 Context: Bucket flush, response code: 200 dump: 
2021/12/15 06:42:36 Waiting for dst bucket item count to get to: 0 curr count: 500
2021/12/15 06:42:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLAutoClose (329.57s)
=== RUN   TestN1QLPosParams
2021/12/15 06:42:41 Waiting for app: TestN1QLPosParams to get un-deployed
2021/12/15 06:42:41 App: TestN1QLPosParams got un-deployed
2021/12/15 06:42:41 No eventing-con process running
2021/12/15 06:42:41 Function: TestN1QLPosParams delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLPosParams not found"
 }
}
2021/12/15 06:42:43 Context: Bucket flush, response code: 200 dump: 
2021/12/15 06:42:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 06:42:50 Context: Bucket flush, response code: 200 dump: 
2021/12/15 06:42:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 06:42:55 Deploying app: TestN1QLPosParams. Item count src bucket: 0 dst bucket: 0
2021/12/15 06:42:56 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLPosParams' in metakv",
  "warnings": null
 }
}
2021/12/15 06:43:01 Waiting for app: TestN1QLPosParams to get deployed
2021/12/15 06:43:06 Waiting for app: TestN1QLPosParams to get deployed
2021/12/15 06:43:11 Waiting for app: TestN1QLPosParams to get deployed
2021/12/15 06:43:16 Waiting for app: TestN1QLPosParams to get deployed
2021/12/15 06:43:16 App: TestN1QLPosParams got deployed
2021/12/15 06:43:16 Starting bucket ops to source bucket
2021/12/15 06:43:26 Waiting for dst bucket item count to get to: 5000 curr count: 625
2021/12/15 06:43:31 Waiting for dst bucket item count to get to: 5000 curr count: 625
2021/12/15 06:43:36 Waiting for dst bucket item count to get to: 5000 curr count: 625
2021/12/15 06:43:36 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/15 06:43:36 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 06:43:36 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 06:43:41 Waiting for app: TestN1QLPosParams to get un-deployed
2021/12/15 06:43:46 Waiting for app: TestN1QLPosParams to get un-deployed
2021/12/15 06:43:51 Waiting for app: TestN1QLPosParams to get un-deployed
2021/12/15 06:43:56 Waiting for app: TestN1QLPosParams to get un-deployed
2021/12/15 06:43:56 App: TestN1QLPosParams got un-deployed
2021/12/15 06:43:56 No eventing-con process running
2021/12/15 06:43:56 Function: TestN1QLPosParams delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLPosParams deleting in the background"
}
2021/12/15 06:43:58 Context: Bucket flush, response code: 200 dump: 
2021/12/15 06:44:08 Waiting for dst bucket item count to get to: 0 curr count: 3128
2021/12/15 06:44:13 Waiting for dst bucket item count to get to: 0 curr count: 628
2021/12/15 06:44:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 06:44:15 Context: Bucket flush, response code: 200 dump: 
2021/12/15 06:44:25 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/15 06:44:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLPosParams (108.15s)
=== RUN   TestN1QLExhaustConnPool
2021/12/15 06:44:30 Waiting for app: TestN1QLExhaustConnPool to get un-deployed
2021/12/15 06:44:30 App: TestN1QLExhaustConnPool got un-deployed
2021/12/15 06:44:30 No eventing-con process running
2021/12/15 06:44:30 Function: TestN1QLExhaustConnPool delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLExhaustConnPool not found"
 }
}
2021/12/15 06:44:31 Context: Bucket flush, response code: 200 dump: 
2021/12/15 06:44:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 06:44:38 Context: Bucket flush, response code: 200 dump: 
2021/12/15 06:44:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 06:44:43 Deploying app: TestN1QLExhaustConnPool. Item count src bucket: 0 dst bucket: 0
2021/12/15 06:44:44 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLExhaustConnPool' in metakv",
  "warnings": null
 }
}
2021/12/15 06:44:49 Waiting for app: TestN1QLExhaustConnPool to get deployed
2021/12/15 06:44:54 Waiting for app: TestN1QLExhaustConnPool to get deployed
2021/12/15 06:44:59 Waiting for app: TestN1QLExhaustConnPool to get deployed
2021/12/15 06:45:04 Waiting for app: TestN1QLExhaustConnPool to get deployed
2021/12/15 06:45:04 App: TestN1QLExhaustConnPool got deployed
2021/12/15 06:45:04 Starting bucket ops to source bucket
2021/12/15 06:45:10 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/15 06:45:15 Waiting for dst bucket item count to get to: 100 curr count: 42
2021/12/15 06:45:20 Waiting for dst bucket item count to get to: 100 curr count: 92
2021/12/15 06:45:20 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/15 06:45:20 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 06:45:20 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 06:45:25 Waiting for app: TestN1QLExhaustConnPool to get un-deployed
2021/12/15 06:45:30 Waiting for app: TestN1QLExhaustConnPool to get un-deployed
2021/12/15 06:45:35 Waiting for app: TestN1QLExhaustConnPool to get un-deployed
2021/12/15 06:45:40 Waiting for app: TestN1QLExhaustConnPool to get un-deployed
2021/12/15 06:45:40 App: TestN1QLExhaustConnPool got un-deployed
2021/12/15 06:45:40 No eventing-con process running
2021/12/15 06:45:40 Function: TestN1QLExhaustConnPool delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLExhaustConnPool deleting in the background"
}
2021/12/15 06:45:42 Context: Bucket flush, response code: 200 dump: 
2021/12/15 06:45:52 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/15 06:45:57 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/15 06:45:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 06:45:58 Context: Bucket flush, response code: 200 dump: 
2021/12/15 06:46:08 Waiting for dst bucket item count to get to: 0 curr count: 80
2021/12/15 06:46:13 Waiting for dst bucket item count to get to: 0 curr count: 30
2021/12/15 06:46:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLExhaustConnPool (108.76s)
=== RUN   TestN1QLGraceSufficientTimeOut
2021/12/15 06:46:18 Waiting for app: TestN1QLGraceSufficientTimeOut to get un-deployed
2021/12/15 06:46:18 App: TestN1QLGraceSufficientTimeOut got un-deployed
2021/12/15 06:46:18 No eventing-con process running
2021/12/15 06:46:18 Function: TestN1QLGraceSufficientTimeOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLGraceSufficientTimeOut not found"
 }
}
2021/12/15 06:46:20 Context: Bucket flush, response code: 200 dump: 
2021/12/15 06:46:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 06:46:27 Context: Bucket flush, response code: 200 dump: 
2021/12/15 06:46:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 06:46:32 Starting bucket ops to source bucket
2021/12/15 06:46:50 Starting bucket ops to source bucket
2021/12/15 06:47:07 Deploying app: TestN1QLGraceSufficientTimeOut. Item count src bucket: 30000 dst bucket: 15055
2021/12/15 06:47:08 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLGraceSufficientTimeOut' in metakv",
  "warnings": null
 }
}
2021/12/15 06:47:13 Waiting for app: TestN1QLGraceSufficientTimeOut to get deployed
2021/12/15 06:47:18 Waiting for app: TestN1QLGraceSufficientTimeOut to get deployed
2021/12/15 06:47:23 Waiting for app: TestN1QLGraceSufficientTimeOut to get deployed
2021/12/15 06:47:28 Waiting for app: TestN1QLGraceSufficientTimeOut to get deployed
2021/12/15 06:47:28 App: TestN1QLGraceSufficientTimeOut got deployed
2021/12/15 06:47:28 Starting bucket ops to source bucket
2021/12/15 06:47:33 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/15 06:47:38 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/15 06:47:43 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/15 06:47:48 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/15 06:47:53 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/15 06:47:58 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/15 06:48:03 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/15 06:48:08 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/15 06:48:13 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/15 06:48:18 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/15 06:48:23 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/15 06:48:23 src & dst bucket item count matched up. src bucket count: 30001 dst bucket count: 30001
2021/12/15 06:48:23 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 06:48:24 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 06:48:29 Waiting for app: TestN1QLGraceSufficientTimeOut to get un-deployed
2021/12/15 06:48:34 Waiting for app: TestN1QLGraceSufficientTimeOut to get un-deployed
2021/12/15 06:48:39 Waiting for app: TestN1QLGraceSufficientTimeOut to get un-deployed
2021/12/15 06:48:44 Waiting for app: TestN1QLGraceSufficientTimeOut to get un-deployed
2021/12/15 06:48:44 App: TestN1QLGraceSufficientTimeOut got un-deployed
2021/12/15 06:48:44 No eventing-con process running
2021/12/15 06:48:44 Function: TestN1QLGraceSufficientTimeOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLGraceSufficientTimeOut deleting in the background"
}
2021/12/15 06:48:46 Context: Bucket flush, response code: 200 dump: 
2021/12/15 06:48:56 Waiting for dst bucket item count to get to: 0 curr count: 30000
2021/12/15 06:48:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 06:48:57 Context: Bucket flush, response code: 200 dump: 
2021/12/15 06:49:07 Waiting for dst bucket item count to get to: 0 curr count: 30001
2021/12/15 06:49:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLGraceSufficientTimeOut (173.95s)
=== RUN   TestN1QLQueryConsistency
2021/12/15 06:49:12 Waiting for app: n1qlquery_consistency to get un-deployed
2021/12/15 06:49:12 App: n1qlquery_consistency got un-deployed
2021/12/15 06:49:12 No eventing-con process running
2021/12/15 06:49:12 Function: n1qlquery_consistency delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: n1qlquery_consistency not found"
 }
}
2021/12/15 06:49:14 Context: Bucket flush, response code: 200 dump: 
2021/12/15 06:49:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 06:49:21 Context: Bucket flush, response code: 200 dump: 
2021/12/15 06:49:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 06:49:26 Starting bucket ops to source bucket
2021/12/15 06:49:29 Deploying app: n1qlquery_consistency. Item count src bucket: 0 dst bucket: 0
2021/12/15 06:49:30 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'n1qlquery_consistency' in metakv",
  "warnings": null
 }
}
2021/12/15 06:49:35 Waiting for app: n1qlquery_consistency to get deployed
2021/12/15 06:49:40 Waiting for app: n1qlquery_consistency to get deployed
2021/12/15 06:49:45 Waiting for app: n1qlquery_consistency to get deployed
2021/12/15 06:49:50 Waiting for app: n1qlquery_consistency to get deployed
2021/12/15 06:49:50 App: n1qlquery_consistency got deployed
2021/12/15 06:49:55 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 06:50:00 Waiting for dst bucket item count to get to: 5000 curr count: 3535
2021/12/15 06:50:05 Waiting for dst bucket item count to get to: 5000 curr count: 3535
2021/12/15 06:50:05 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/15 06:50:05 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 06:50:05 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 06:50:10 Waiting for app: n1qlquery_consistency to get un-deployed
2021/12/15 06:50:15 Waiting for app: n1qlquery_consistency to get un-deployed
2021/12/15 06:50:20 Waiting for app: n1qlquery_consistency to get un-deployed
2021/12/15 06:50:25 Waiting for app: n1qlquery_consistency to get un-deployed
2021/12/15 06:50:25 App: n1qlquery_consistency got un-deployed
2021/12/15 06:50:25 No eventing-con process running
2021/12/15 06:50:25 Function: n1qlquery_consistency delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: n1qlquery_consistency deleting in the background"
}
2021/12/15 06:50:27 Context: Bucket flush, response code: 200 dump: 
2021/12/15 06:50:37 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/15 06:50:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 06:50:39 Context: Bucket flush, response code: 200 dump: 
2021/12/15 06:50:49 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/15 06:50:54 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/15 06:50:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLQueryConsistency (106.64s)
PASS
ok  	github.com/couchbase/eventing/tests/functional_tests	1563.431s

N1ql tests took 0 hours, 26 minutes and 9 seconds.

Starting server
Starting server: attempt 1

Rebalance tests

2021/12/15 06:52:29 Started rebalance
Post "http://127.0.0.1:9000/pools/default": read tcp 127.0.0.1:57262->127.0.0.1:9000: read: connection reset by peer
Quota setup Post "http://127.0.0.1:9000/pools/default": read tcp 127.0.0.1:57262->127.0.0.1:9000: read: connection reset by peer
=== RUN   TestDuplicateWithEventingRebIn
2021/12/15 06:52:48 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
running apps fetch error json: cannot unmarshal number into Go value of type string
2021/12/15 06:52:53 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
2021/12/15 06:52:53 App: TestDuplicateWithEventingRebIn got un-deployed
2021/12/15 06:52:53 No eventing-con process running
2021/12/15 06:52:53 Function: TestDuplicateWithEventingRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithEventingRebIn not found"
 }
}
2021/12/15 06:52:54 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 06:52:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 06:52:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 06:53:01 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 06:53:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 06:53:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 06:53:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 06:53:11 Deploying app: TestDuplicateWithEventingRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/15 06:53:12 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithEventingRebIn' in metakv",
  "warnings": null
 }
}
2021/12/15 06:53:17 Waiting for app: TestDuplicateWithEventingRebIn to get deployed
2021/12/15 06:53:22 Waiting for app: TestDuplicateWithEventingRebIn to get deployed
2021/12/15 06:53:27 Waiting for app: TestDuplicateWithEventingRebIn to get deployed
2021/12/15 06:53:32 Waiting for app: TestDuplicateWithEventingRebIn to get deployed
2021/12/15 06:53:32 App: TestDuplicateWithEventingRebIn got deployed
2021/12/15 06:53:32 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/15 06:53:32 Starting bucket ops to source bucket
2021/12/15 06:53:46 Removing node(s): [] from the cluster
2021/12/15 06:53:52 Rebalance progress: 0
2021/12/15 06:53:57 Rebalance progress: 0
2021/12/15 06:54:02 Rebalance progress: 25
2021/12/15 06:54:07 Rebalance progress: 25
2021/12/15 06:54:12 Rebalance progress: 25
2021/12/15 06:54:17 Rebalance progress: 25
2021/12/15 06:54:22 Rebalance progress: 25
2021/12/15 06:54:27 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 06:54:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 06:54:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 06:54:27 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9302/api/v1/stats": dial tcp 127.0.0.1:9302: connect: connection refused
2021/12/15 06:54:27 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/15 06:54:27 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/15 06:54:32 Rebalance progress: 25
2021/12/15 06:54:37 Rebalance progress: 25
2021/12/15 06:54:42 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 06:54:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 06:54:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 06:54:42 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9302/api/v1/stats": dial tcp 127.0.0.1:9302: connect: connection refused
2021/12/15 06:54:42 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/15 06:54:42 Waiting for src bucket item count to get to: 200000 curr count: 91081
2021/12/15 06:54:47 Waiting for src bucket item count to get to: 200000 curr count: 107811
2021/12/15 06:54:52 Waiting for src bucket item count to get to: 200000 curr count: 107811
2021/12/15 06:54:57 Waiting for src bucket item count to get to: 200000 curr count: 124280
2021/12/15 06:55:02 Waiting for src bucket item count to get to: 200000 curr count: 124280
2021/12/15 06:55:07 Waiting for src bucket item count to get to: 200000 curr count: 141060
2021/12/15 06:55:12 Waiting for src bucket item count to get to: 200000 curr count: 141060
2021/12/15 06:55:17 Waiting for src bucket item count to get to: 200000 curr count: 157689
2021/12/15 06:55:22 Waiting for src bucket item count to get to: 200000 curr count: 157689
2021/12/15 06:55:27 Waiting for src bucket item count to get to: 200000 curr count: 174121
2021/12/15 06:55:32 Waiting for src bucket item count to get to: 200000 curr count: 174121
2021/12/15 06:55:37 Waiting for src bucket item count to get to: 200000 curr count: 190294
2021/12/15 06:55:42 Waiting for src bucket item count to get to: 200000 curr count: 190294
2021/12/15 06:55:52 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/15 06:55:57 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/15 06:56:02 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/15 06:56:07 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/15 06:56:12 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/15 06:56:17 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/15 06:56:22 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/15 06:56:27 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/15 06:56:32 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/15 06:56:37 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/15 06:56:37 Post SET ops. Expected item count: 200000 got 200001
2021/12/15 06:56:37 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 06:56:42 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
2021/12/15 06:56:47 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
2021/12/15 06:56:52 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
2021/12/15 06:56:57 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
2021/12/15 06:56:57 App: TestDuplicateWithEventingRebIn got un-deployed
2021/12/15 06:56:57 No eventing-con process running
2021/12/15 06:56:57 Function: TestDuplicateWithEventingRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithEventingRebIn deleting in the background"
}
2021/12/15 06:56:59 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 06:57:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 06:57:09 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/15 06:57:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 06:57:10 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 06:57:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 06:57:20 Waiting for dst bucket item count to get to: 0 curr count: 95370
2021/12/15 06:57:25 Waiting for dst bucket item count to get to: 0 curr count: 95370
2021/12/15 06:57:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithEventingRebIn (287.92s)
=== RUN   TestDuplicateWithEventingRebOut
2021/12/15 06:57:30 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/15 06:57:31 Removing node(s): [] from the cluster
2021/12/15 06:57:36 Rebalance progress: 100
2021/12/15 06:57:41 Waiting for app: TestDuplicateWithEventingRebOut to get un-deployed
2021/12/15 06:57:41 App: TestDuplicateWithEventingRebOut got un-deployed
2021/12/15 06:57:41 No eventing-con process running
2021/12/15 06:57:41 Function: TestDuplicateWithEventingRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithEventingRebOut not found"
 }
}
2021/12/15 06:57:43 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 06:57:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 06:57:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 06:57:50 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 06:57:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 06:57:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 06:58:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 06:58:00 Deploying app: TestDuplicateWithEventingRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/15 06:58:01 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithEventingRebOut' in metakv",
  "warnings": null
 }
}
2021/12/15 06:58:06 Waiting for app: TestDuplicateWithEventingRebOut to get deployed
2021/12/15 06:58:11 Waiting for app: TestDuplicateWithEventingRebOut to get deployed
2021/12/15 06:58:16 Waiting for app: TestDuplicateWithEventingRebOut to get deployed
2021/12/15 06:58:21 Waiting for app: TestDuplicateWithEventingRebOut to get deployed
2021/12/15 06:58:21 App: TestDuplicateWithEventingRebOut got deployed
2021/12/15 06:58:21 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/15 06:58:21 Starting bucket ops to source bucket
2021/12/15 06:58:26 Rebalance progress: 25
2021/12/15 06:58:31 Rebalance progress: 25
2021/12/15 06:58:36 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 06:58:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 06:58:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 06:58:36 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9302/api/v1/stats": dial tcp 127.0.0.1:9302: connect: connection refused
2021/12/15 06:58:36 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/15 06:58:36 Waiting for src bucket item count to get to: 200000 curr count: 13221
2021/12/15 06:58:41 Waiting for src bucket item count to get to: 200000 curr count: 29641
2021/12/15 06:58:46 Waiting for src bucket item count to get to: 200000 curr count: 29641
2021/12/15 06:58:51 Waiting for src bucket item count to get to: 200000 curr count: 46271
2021/12/15 06:58:56 Waiting for src bucket item count to get to: 200000 curr count: 46271
2021/12/15 06:59:01 Waiting for src bucket item count to get to: 200000 curr count: 62814
2021/12/15 06:59:06 Waiting for src bucket item count to get to: 200000 curr count: 62814
2021/12/15 06:59:11 Waiting for src bucket item count to get to: 200000 curr count: 62814
2021/12/15 06:59:16 Waiting for src bucket item count to get to: 200000 curr count: 78630
2021/12/15 06:59:21 Waiting for src bucket item count to get to: 200000 curr count: 78630
2021/12/15 06:59:26 Waiting for src bucket item count to get to: 200000 curr count: 94561
2021/12/15 06:59:31 Waiting for src bucket item count to get to: 200000 curr count: 94561
2021/12/15 06:59:36 Waiting for src bucket item count to get to: 200000 curr count: 110593
2021/12/15 06:59:41 Waiting for src bucket item count to get to: 200000 curr count: 110593
2021/12/15 06:59:46 Waiting for src bucket item count to get to: 200000 curr count: 126951
2021/12/15 06:59:51 Waiting for src bucket item count to get to: 200000 curr count: 126951
2021/12/15 06:59:56 Waiting for src bucket item count to get to: 200000 curr count: 143729
2021/12/15 07:00:01 Waiting for src bucket item count to get to: 200000 curr count: 143729
2021/12/15 07:00:06 Waiting for src bucket item count to get to: 200000 curr count: 160277
2021/12/15 07:00:11 Waiting for src bucket item count to get to: 200000 curr count: 160277
2021/12/15 07:00:16 Waiting for src bucket item count to get to: 200000 curr count: 176271
2021/12/15 07:00:21 Waiting for src bucket item count to get to: 200000 curr count: 176271
2021/12/15 07:00:26 Waiting for src bucket item count to get to: 200000 curr count: 192631
2021/12/15 07:00:31 Waiting for src bucket item count to get to: 200000 curr count: 192631
2021/12/15 07:00:36 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/15 07:00:36 Post SET ops. Expected item count: 200000 got 200000
2021/12/15 07:00:36 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 07:00:41 Waiting for app: TestDuplicateWithEventingRebOut to get un-deployed
2021/12/15 07:00:46 Waiting for app: TestDuplicateWithEventingRebOut to get un-deployed
2021/12/15 07:00:51 Waiting for app: TestDuplicateWithEventingRebOut to get un-deployed
2021/12/15 07:00:56 Waiting for app: TestDuplicateWithEventingRebOut to get un-deployed
2021/12/15 07:00:56 App: TestDuplicateWithEventingRebOut got un-deployed
2021/12/15 07:00:56 No eventing-con process running
2021/12/15 07:00:56 Function: TestDuplicateWithEventingRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithEventingRebOut deleting in the background"
}
2021/12/15 07:00:58 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:01:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 07:01:08 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/15 07:01:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 07:01:10 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:01:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 07:01:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithEventingRebOut (229.28s)
=== RUN   TestDuplicateWithEventingSwapReb
2021/12/15 07:01:25 Waiting for app: TestDuplicateWithEventingSwapReb to get un-deployed
2021/12/15 07:01:25 App: TestDuplicateWithEventingSwapReb got un-deployed
2021/12/15 07:01:25 No eventing-con process running
2021/12/15 07:01:25 Function: TestDuplicateWithEventingSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithEventingSwapReb not found"
 }
}
2021/12/15 07:01:26 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:01:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 07:01:31 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 07:01:33 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:01:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 07:01:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:01:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 07:01:43 Deploying app: TestDuplicateWithEventingSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/15 07:01:44 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithEventingSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/15 07:01:49 Waiting for app: TestDuplicateWithEventingSwapReb to get deployed
2021/12/15 07:01:54 Waiting for app: TestDuplicateWithEventingSwapReb to get deployed
2021/12/15 07:01:59 Waiting for app: TestDuplicateWithEventingSwapReb to get deployed
2021/12/15 07:02:04 Waiting for app: TestDuplicateWithEventingSwapReb to get deployed
2021/12/15 07:02:04 App: TestDuplicateWithEventingSwapReb got deployed
2021/12/15 07:02:04 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/15 07:02:04 Starting bucket ops to source bucket
2021/12/15 07:02:05 Removing node(s): [] from the cluster
2021/12/15 07:02:10 Rebalance progress: 25
2021/12/15 07:02:15 Rebalance progress: 25
2021/12/15 07:02:20 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:02:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:02:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 07:02:20 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9302/api/v1/stats": dial tcp 127.0.0.1:9302: connect: connection refused
2021/12/15 07:02:20 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/15 07:02:20 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/15 07:02:27 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/15 07:02:32 Rebalance progress: 16.66666666666667
2021/12/15 07:02:37 Rebalance progress: 16.66666666666667
2021/12/15 07:02:42 Rebalance progress: 16.66666666666667
2021/12/15 07:02:47 Rebalance progress: 91.05691056910568
2021/12/15 07:02:52 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:02:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:02:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 07:02:52 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/15 07:02:52 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/15 07:02:57 Rebalance progress: 16.66666666666667
2021/12/15 07:03:02 Rebalance progress: 16.66666666666667
2021/12/15 07:03:07 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:03:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:03:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 07:03:07 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/15 07:03:07 Waiting for src bucket item count to get to: 200000 curr count: 83259
2021/12/15 07:03:12 Waiting for src bucket item count to get to: 200000 curr count: 99465
2021/12/15 07:03:17 Waiting for src bucket item count to get to: 200000 curr count: 99465
2021/12/15 07:03:22 Waiting for src bucket item count to get to: 200000 curr count: 115571
2021/12/15 07:03:27 Waiting for src bucket item count to get to: 200000 curr count: 115571
2021/12/15 07:03:32 Waiting for src bucket item count to get to: 200000 curr count: 132012
2021/12/15 07:03:37 Waiting for src bucket item count to get to: 200000 curr count: 132012
2021/12/15 07:03:42 Waiting for src bucket item count to get to: 200000 curr count: 148318
2021/12/15 07:03:47 Waiting for src bucket item count to get to: 200000 curr count: 148318
2021/12/15 07:03:52 Waiting for src bucket item count to get to: 200000 curr count: 164743
2021/12/15 07:03:57 Waiting for src bucket item count to get to: 200000 curr count: 164743
2021/12/15 07:04:02 Waiting for src bucket item count to get to: 200000 curr count: 180660
2021/12/15 07:04:07 Waiting for src bucket item count to get to: 200000 curr count: 180660
2021/12/15 07:04:12 Waiting for src bucket item count to get to: 200000 curr count: 196924
2021/12/15 07:04:17 Waiting for src bucket item count to get to: 200000 curr count: 196924
2021/12/15 07:04:22 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/15 07:04:22 Post SET ops. Expected item count: 200000 got 200000
2021/12/15 07:04:22 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 07:04:27 Waiting for app: TestDuplicateWithEventingSwapReb to get un-deployed
2021/12/15 07:04:32 Waiting for app: TestDuplicateWithEventingSwapReb to get un-deployed
2021/12/15 07:04:37 Waiting for app: TestDuplicateWithEventingSwapReb to get un-deployed
2021/12/15 07:04:42 Waiting for app: TestDuplicateWithEventingSwapReb to get un-deployed
2021/12/15 07:04:42 App: TestDuplicateWithEventingSwapReb got un-deployed
2021/12/15 07:04:42 No eventing-con process running
2021/12/15 07:04:42 Function: TestDuplicateWithEventingSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithEventingSwapReb deleting in the background"
}
2021/12/15 07:04:44 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:04:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 07:04:54 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/15 07:04:59 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/15 07:04:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 07:05:01 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:05:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 07:05:11 Waiting for dst bucket item count to get to: 0 curr count: 170058
2021/12/15 07:05:16 Waiting for dst bucket item count to get to: 0 curr count: 170058
2021/12/15 07:05:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithEventingSwapReb (241.17s)
=== RUN   TestDuplicateWithKVRebIn
2021/12/15 07:05:26 Waiting for app: TestDuplicateWithKVRebIn to get un-deployed
2021/12/15 07:05:26 App: TestDuplicateWithKVRebIn got un-deployed
2021/12/15 07:05:26 No eventing-con process running
2021/12/15 07:05:26 Function: TestDuplicateWithKVRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithKVRebIn not found"
 }
}
2021/12/15 07:05:28 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:05:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 07:05:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 07:05:34 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:05:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 07:05:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:05:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 07:05:44 Deploying app: TestDuplicateWithKVRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/15 07:05:46 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithKVRebIn' in metakv",
  "warnings": null
 }
}
2021/12/15 07:05:51 Waiting for app: TestDuplicateWithKVRebIn to get deployed
2021/12/15 07:05:56 Waiting for app: TestDuplicateWithKVRebIn to get deployed
2021/12/15 07:06:01 Waiting for app: TestDuplicateWithKVRebIn to get deployed
2021/12/15 07:06:06 Waiting for app: TestDuplicateWithKVRebIn to get deployed
2021/12/15 07:06:06 App: TestDuplicateWithKVRebIn got deployed
2021/12/15 07:06:06 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/15 07:06:06 Starting bucket ops to source bucket
2021/12/15 07:06:06 Removing node(s): [] from the cluster
2021/12/15 07:06:11 Rebalance progress: 16.66666666666667
2021/12/15 07:06:16 Rebalance progress: 16.66666666666667
2021/12/15 07:06:21 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:06:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:06:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 07:06:21 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/15 07:06:21 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/15 07:06:26 Rebalance progress: 16.66666666666667
2021/12/15 07:06:31 Rebalance progress: 16.66666666666667
2021/12/15 07:06:36 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:06:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:06:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 07:06:36 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/15 07:06:36 Waiting for src bucket item count to get to: 200000 curr count: 37271
2021/12/15 07:06:41 Waiting for src bucket item count to get to: 200000 curr count: 53505
2021/12/15 07:06:46 Waiting for src bucket item count to get to: 200000 curr count: 53505
2021/12/15 07:06:51 Waiting for src bucket item count to get to: 200000 curr count: 69537
2021/12/15 07:06:56 Waiting for src bucket item count to get to: 200000 curr count: 69537
2021/12/15 07:07:01 Waiting for src bucket item count to get to: 200000 curr count: 84639
2021/12/15 07:07:06 Waiting for src bucket item count to get to: 200000 curr count: 84639
2021/12/15 07:07:11 Waiting for src bucket item count to get to: 200000 curr count: 101197
2021/12/15 07:07:16 Waiting for src bucket item count to get to: 200000 curr count: 101197
2021/12/15 07:07:21 Waiting for src bucket item count to get to: 200000 curr count: 101197
2021/12/15 07:07:26 Waiting for src bucket item count to get to: 200000 curr count: 117673
2021/12/15 07:07:31 Waiting for src bucket item count to get to: 200000 curr count: 117673
2021/12/15 07:07:36 Waiting for src bucket item count to get to: 200000 curr count: 133778
2021/12/15 07:07:41 Waiting for src bucket item count to get to: 200000 curr count: 133778
2021/12/15 07:07:46 Waiting for src bucket item count to get to: 200000 curr count: 150095
2021/12/15 07:07:51 Waiting for src bucket item count to get to: 200000 curr count: 150095
2021/12/15 07:07:56 Waiting for src bucket item count to get to: 200000 curr count: 166353
2021/12/15 07:08:01 Waiting for src bucket item count to get to: 200000 curr count: 166353
2021/12/15 07:08:06 Waiting for src bucket item count to get to: 200000 curr count: 181747
2021/12/15 07:08:11 Waiting for src bucket item count to get to: 200000 curr count: 181747
2021/12/15 07:08:16 Waiting for src bucket item count to get to: 200000 curr count: 198165
2021/12/15 07:08:21 Waiting for src bucket item count to get to: 200000 curr count: 198165
2021/12/15 07:08:26 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/15 07:08:26 Post SET ops. Expected item count: 200000 got 200000
2021/12/15 07:08:26 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 07:08:31 Waiting for app: TestDuplicateWithKVRebIn to get un-deployed
2021/12/15 07:08:36 Waiting for app: TestDuplicateWithKVRebIn to get un-deployed
2021/12/15 07:08:41 Waiting for app: TestDuplicateWithKVRebIn to get un-deployed
2021/12/15 07:08:46 Waiting for app: TestDuplicateWithKVRebIn to get un-deployed
2021/12/15 07:08:46 App: TestDuplicateWithKVRebIn got un-deployed
2021/12/15 07:08:46 No eventing-con process running
2021/12/15 07:08:46 Function: TestDuplicateWithKVRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithKVRebIn deleting in the background"
}
2021/12/15 07:08:48 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:08:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 07:08:58 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/15 07:08:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 07:09:00 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:09:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 07:09:10 Waiting for dst bucket item count to get to: 0 curr count: 29921
2021/12/15 07:09:15 Waiting for dst bucket item count to get to: 0 curr count: 29921
2021/12/15 07:09:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithKVRebIn (239.21s)
=== RUN   TestDuplicateWithKVRebOut
2021/12/15 07:09:20 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/15 07:09:21 Removing node(s): [] from the cluster
2021/12/15 07:09:26 Rebalance progress: 100
2021/12/15 07:09:31 Waiting for app: TestDuplicateWithKVRebOut to get un-deployed
2021/12/15 07:09:31 App: TestDuplicateWithKVRebOut got un-deployed
2021/12/15 07:09:31 No eventing-con process running
2021/12/15 07:09:31 Function: TestDuplicateWithKVRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithKVRebOut not found"
 }
}
2021/12/15 07:09:32 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:09:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 07:09:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 07:09:39 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:09:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 07:09:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:09:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 07:09:49 Deploying app: TestDuplicateWithKVRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/15 07:09:50 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithKVRebOut' in metakv",
  "warnings": null
 }
}
2021/12/15 07:09:55 Waiting for app: TestDuplicateWithKVRebOut to get deployed
2021/12/15 07:10:00 Waiting for app: TestDuplicateWithKVRebOut to get deployed
2021/12/15 07:10:05 Waiting for app: TestDuplicateWithKVRebOut to get deployed
2021/12/15 07:10:10 Waiting for app: TestDuplicateWithKVRebOut to get deployed
2021/12/15 07:10:10 App: TestDuplicateWithKVRebOut got deployed
2021/12/15 07:10:10 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/15 07:10:10 Starting bucket ops to source bucket
2021/12/15 07:10:15 Rebalance progress: 16.66666666666667
2021/12/15 07:10:20 Rebalance progress: 16.66666666666667
2021/12/15 07:10:25 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:10:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:10:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 07:10:25 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/15 07:10:25 Waiting for src bucket item count to get to: 200000 curr count: 13668
2021/12/15 07:10:30 Waiting for src bucket item count to get to: 200000 curr count: 29878
2021/12/15 07:10:35 Waiting for src bucket item count to get to: 200000 curr count: 29878
2021/12/15 07:10:40 Waiting for src bucket item count to get to: 200000 curr count: 45993
2021/12/15 07:10:45 Waiting for src bucket item count to get to: 200000 curr count: 45993
2021/12/15 07:10:50 Waiting for src bucket item count to get to: 200000 curr count: 62330
2021/12/15 07:10:55 Waiting for src bucket item count to get to: 200000 curr count: 62330
2021/12/15 07:11:00 Waiting for src bucket item count to get to: 200000 curr count: 62330
2021/12/15 07:11:05 Waiting for src bucket item count to get to: 200000 curr count: 78169
2021/12/15 07:11:11 Waiting for src bucket item count to get to: 200000 curr count: 78169
2021/12/15 07:11:16 Waiting for src bucket item count to get to: 200000 curr count: 93921
2021/12/15 07:11:21 Waiting for src bucket item count to get to: 200000 curr count: 93921
2021/12/15 07:11:26 Waiting for src bucket item count to get to: 200000 curr count: 110037
2021/12/15 07:11:31 Waiting for src bucket item count to get to: 200000 curr count: 110037
2021/12/15 07:11:36 Waiting for src bucket item count to get to: 200000 curr count: 126289
2021/12/15 07:11:41 Waiting for src bucket item count to get to: 200000 curr count: 126289
2021/12/15 07:11:46 Waiting for src bucket item count to get to: 200000 curr count: 142504
2021/12/15 07:11:51 Waiting for src bucket item count to get to: 200000 curr count: 142504
2021/12/15 07:11:56 Waiting for src bucket item count to get to: 200000 curr count: 158974
2021/12/15 07:12:01 Waiting for src bucket item count to get to: 200000 curr count: 158974
2021/12/15 07:12:06 Waiting for src bucket item count to get to: 200000 curr count: 174705
2021/12/15 07:12:11 Waiting for src bucket item count to get to: 200000 curr count: 174705
2021/12/15 07:12:16 Waiting for src bucket item count to get to: 200000 curr count: 190698
2021/12/15 07:12:21 Waiting for src bucket item count to get to: 200000 curr count: 190698
2021/12/15 07:12:26 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/15 07:12:26 Post SET ops. Expected item count: 200000 got 200000
2021/12/15 07:12:26 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 07:12:31 Waiting for app: TestDuplicateWithKVRebOut to get un-deployed
2021/12/15 07:12:36 Waiting for app: TestDuplicateWithKVRebOut to get un-deployed
2021/12/15 07:12:41 Waiting for app: TestDuplicateWithKVRebOut to get un-deployed
2021/12/15 07:12:46 Waiting for app: TestDuplicateWithKVRebOut to get un-deployed
2021/12/15 07:12:46 App: TestDuplicateWithKVRebOut got un-deployed
2021/12/15 07:12:46 No eventing-con process running
2021/12/15 07:12:46 Function: TestDuplicateWithKVRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithKVRebOut deleting in the background"
}
2021/12/15 07:12:48 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:12:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 07:12:58 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/15 07:12:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 07:12:59 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:13:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 07:13:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithKVRebOut (229.27s)
=== RUN   TestDuplicateWithKVSwapReb
2021/12/15 07:13:14 Waiting for app: TestDuplicateWithKVSwapReb to get un-deployed
2021/12/15 07:13:14 App: TestDuplicateWithKVSwapReb got un-deployed
2021/12/15 07:13:14 No eventing-con process running
2021/12/15 07:13:14 Function: TestDuplicateWithKVSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithKVSwapReb not found"
 }
}
2021/12/15 07:13:16 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:13:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 07:13:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 07:13:23 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:13:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 07:13:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:13:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 07:13:33 Deploying app: TestDuplicateWithKVSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/15 07:13:34 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithKVSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/15 07:13:39 Waiting for app: TestDuplicateWithKVSwapReb to get deployed
2021/12/15 07:13:44 Waiting for app: TestDuplicateWithKVSwapReb to get deployed
2021/12/15 07:13:49 Waiting for app: TestDuplicateWithKVSwapReb to get deployed
2021/12/15 07:13:54 Waiting for app: TestDuplicateWithKVSwapReb to get deployed
2021/12/15 07:13:54 App: TestDuplicateWithKVSwapReb got deployed
2021/12/15 07:13:54 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/15 07:13:54 Starting bucket ops to source bucket
2021/12/15 07:13:54 Removing node(s): [] from the cluster
2021/12/15 07:13:59 Rebalance progress: 16.66666666666667
2021/12/15 07:14:04 Rebalance progress: 16.66666666666667
2021/12/15 07:14:09 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:14:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:14:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 07:14:09 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/15 07:14:09 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/15 07:14:10 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/15 07:14:15 Rebalance progress: 16.66666666666667
2021/12/15 07:14:20 Rebalance progress: 16.66666666666667
2021/12/15 07:14:25 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:14:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:14:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 07:14:25 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/15 07:14:25 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/15 07:14:30 Rebalance progress: 16.66666666666667
2021/12/15 07:14:35 Rebalance progress: 16.66666666666667
2021/12/15 07:14:40 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:14:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:14:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 07:14:40 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/15 07:14:40 Waiting for src bucket item count to get to: 200000 curr count: 54529
2021/12/15 07:14:45 Waiting for src bucket item count to get to: 200000 curr count: 71046
2021/12/15 07:14:50 Waiting for src bucket item count to get to: 200000 curr count: 71046
2021/12/15 07:14:55 Waiting for src bucket item count to get to: 200000 curr count: 86247
2021/12/15 07:15:00 Waiting for src bucket item count to get to: 200000 curr count: 86247
2021/12/15 07:15:05 Waiting for src bucket item count to get to: 200000 curr count: 102487
2021/12/15 07:15:10 Waiting for src bucket item count to get to: 200000 curr count: 118671
2021/12/15 07:15:15 Waiting for src bucket item count to get to: 200000 curr count: 118671
2021/12/15 07:15:20 Waiting for src bucket item count to get to: 200000 curr count: 135009
2021/12/15 07:15:25 Waiting for src bucket item count to get to: 200000 curr count: 135009
2021/12/15 07:15:30 Waiting for src bucket item count to get to: 200000 curr count: 151501
2021/12/15 07:15:35 Waiting for src bucket item count to get to: 200000 curr count: 151501
2021/12/15 07:15:40 Waiting for src bucket item count to get to: 200000 curr count: 151501
2021/12/15 07:15:45 Waiting for src bucket item count to get to: 200000 curr count: 168051
2021/12/15 07:15:50 Waiting for src bucket item count to get to: 200000 curr count: 168051
2021/12/15 07:15:55 Waiting for src bucket item count to get to: 200000 curr count: 183295
2021/12/15 07:16:00 Waiting for src bucket item count to get to: 200000 curr count: 183295
2021/12/15 07:16:05 Waiting for src bucket item count to get to: 200000 curr count: 199495
2021/12/15 07:16:10 Waiting for src bucket item count to get to: 200000 curr count: 199495
2021/12/15 07:16:15 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/15 07:16:15 Post SET ops. Expected item count: 200000 got 200000
2021/12/15 07:16:15 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 07:16:20 Waiting for app: TestDuplicateWithKVSwapReb to get un-deployed
2021/12/15 07:16:25 Waiting for app: TestDuplicateWithKVSwapReb to get un-deployed
2021/12/15 07:16:30 Waiting for app: TestDuplicateWithKVSwapReb to get un-deployed
2021/12/15 07:16:35 Waiting for app: TestDuplicateWithKVSwapReb to get un-deployed
2021/12/15 07:16:35 App: TestDuplicateWithKVSwapReb got un-deployed
2021/12/15 07:16:35 No eventing-con process running
2021/12/15 07:16:35 Function: TestDuplicateWithKVSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithKVSwapReb deleting in the background"
}
2021/12/15 07:16:37 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:16:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 07:16:47 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/15 07:16:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 07:16:49 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:16:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 07:16:59 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/15 07:16:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithKVSwapReb (234.58s)
=== RUN   TestEventingRebKVOpsOnUpdateBucketOpOneByOne
2021/12/15 07:17:09 Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/15 07:17:09 App: TestEventingRebKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/15 07:17:09 No eventing-con process running
2021/12/15 07:17:09 Function: TestEventingRebKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpOneByOne not found"
 }
}
2021/12/15 07:17:11 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:17:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 07:17:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 07:17:17 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:17:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 07:17:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:17:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 07:17:27 Deploying app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne. Item count src bucket: 0 dst bucket: 0
2021/12/15 07:17:29 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebKVOpsOnUpdateBucketOpOneByOne' in metakv",
  "warnings": null
 }
}
2021/12/15 07:17:34 Starting bucket ops to source bucket
2021/12/15 07:17:39 Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/15 07:17:44 Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/15 07:17:49 Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/15 07:17:49 App: TestEventingRebKVOpsOnUpdateBucketOpOneByOne got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:17:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:17:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 07:17:49 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/15 07:17:49 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/15 07:17:49 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/15 07:17:54 Rebalance progress: 16.66666666666667
2021/12/15 07:17:59 Rebalance progress: 16.66666666666667
2021/12/15 07:18:04 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:18:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:18:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 07:18:04 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/15 07:18:04 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/15 07:18:04 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/15 07:18:09 Rebalance progress: 16.66666666666667
2021/12/15 07:18:14 Rebalance progress: 16.66666666666667
2021/12/15 07:18:19 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:18:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:18:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 07:18:19 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/15 07:18:19 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/15 07:18:25 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/15 07:18:30 Rebalance progress: 12.5
2021/12/15 07:18:35 Rebalance progress: 12.5
2021/12/15 07:18:40 Rebalance progress: 12.5
2021/12/15 07:18:45 Rebalance progress: 61.03515625
2021/12/15 07:18:50 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:18:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:18:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 07:18:50 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/15 07:18:55 Rebalance progress: 12.5
2021/12/15 07:19:00 Rebalance progress: 12.5
2021/12/15 07:19:05 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:19:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:19:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 07:19:05 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/15 07:19:10 Rebalance progress: 12.5
2021/12/15 07:19:15 Rebalance progress: 12.5
2021/12/15 07:19:20 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:19:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:19:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 07:19:20 clusterSetup.go:253: Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/15 07:19:25 Rebalance progress: 12.5
2021/12/15 07:19:30 Rebalance progress: 12.5
2021/12/15 07:19:35 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:19:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:19:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 07:19:35 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 07:19:40 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/15 07:19:45 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/15 07:19:50 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/15 07:19:55 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/15 07:19:55 clusterSetup.go:405: App: TestEventingRebKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/15 07:19:55 eventingSetup.go:46: No eventing-con process running
2021/12/15 07:19:55 eventingSetup.go:418: Function: TestEventingRebKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpOneByOne deleting in the background"
}
2021/12/15 07:19:57 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:20:02 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 07:20:07 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12143
2021/12/15 07:20:07 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 07:20:09 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:20:14 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 07:20:19 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12143
2021/12/15 07:20:19 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebKVOpsOnUpdateBucketOpOneByOne (199.94s)
=== RUN   TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce
2021/12/15 07:20:29 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get un-deployed
2021/12/15 07:20:29 clusterSetup.go:405: App: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce got un-deployed
2021/12/15 07:20:29 eventingSetup.go:46: No eventing-con process running
2021/12/15 07:20:29 eventingSetup.go:418: Function: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce not found"
 }
}
2021/12/15 07:20:31 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:20:36 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 07:20:36 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 07:20:37 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:20:42 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 07:20:42 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:20:47 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 07:20:47 eventingSetup.go:125: Deploying app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce. Item count src bucket: 0 dst bucket: 0
2021/12/15 07:20:49 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce' in metakv",
  "warnings": null
 }
}
2021/12/15 07:20:54 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/15 07:20:59 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get deployed
2021/12/15 07:21:04 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get deployed
2021/12/15 07:21:09 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get deployed
2021/12/15 07:21:09 clusterSetup.go:354: App: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:21:09 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:21:09 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 07:21:09 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/15 07:21:09 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/15 07:21:10 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/15 07:21:10 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/15 07:21:15 Rebalance progress: 12.5
2021/12/15 07:21:20 Rebalance progress: 12.5
2021/12/15 07:21:25 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:21:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:21:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 07:21:25 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002 127.0.0.1:19003] from the cluster
2021/12/15 07:21:30 Rebalance progress: 12.5
2021/12/15 07:21:35 Rebalance progress: 12.5
2021/12/15 07:21:40 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:21:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:21:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 07:21:40 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 07:21:45 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get un-deployed
2021/12/15 07:21:50 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get un-deployed
2021/12/15 07:21:55 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get un-deployed
2021/12/15 07:22:01 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get un-deployed
2021/12/15 07:22:01 clusterSetup.go:405: App: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce got un-deployed
2021/12/15 07:22:01 eventingSetup.go:46: No eventing-con process running
2021/12/15 07:22:01 eventingSetup.go:418: Function: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce deleting in the background"
}
2021/12/15 07:22:02 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:22:07 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 07:22:12 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4662
2021/12/15 07:22:17 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4662
2021/12/15 07:22:17 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 07:22:19 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:22:24 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 07:22:29 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4662
2021/12/15 07:22:29 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce (130.28s)
=== RUN   TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings
2021/12/15 07:22:39 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/15 07:22:39 clusterSetup.go:405: App: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings got un-deployed
2021/12/15 07:22:39 eventingSetup.go:46: No eventing-con process running
2021/12/15 07:22:39 eventingSetup.go:418: Function: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings not found"
 }
}
2021/12/15 07:22:41 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:22:46 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 07:22:46 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 07:22:48 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:22:53 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 07:22:53 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:22:58 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 07:22:58 eventingSetup.go:125: Deploying app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/15 07:22:59 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/15 07:23:04 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/15 07:23:09 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get deployed
2021/12/15 07:23:14 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get deployed
2021/12/15 07:23:19 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get deployed
2021/12/15 07:23:19 clusterSetup.go:354: App: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:23:19 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:23:19 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 07:23:19 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/15 07:23:19 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/15 07:23:24 Rebalance progress: 12.5
2021/12/15 07:23:29 Rebalance progress: 12.5
2021/12/15 07:23:34 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:23:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:23:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 07:23:34 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/15 07:23:35 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/15 07:23:40 Rebalance progress: 12.5
2021/12/15 07:23:45 Rebalance progress: 12.5
2021/12/15 07:23:50 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:23:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:23:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 07:23:50 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/15 07:23:51 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/15 07:23:56 Rebalance progress: 12.5
2021/12/15 07:24:01 Rebalance progress: 12.5
2021/12/15 07:24:06 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:24:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:24:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 07:24:06 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/15 07:24:11 Rebalance progress: 12.5
2021/12/15 07:24:16 Rebalance progress: 12.5
2021/12/15 07:24:21 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:24:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:24:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 07:24:21 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/15 07:24:26 Rebalance progress: 12.5
2021/12/15 07:24:31 Rebalance progress: 12.5
2021/12/15 07:24:36 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:24:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:24:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 07:24:36 clusterSetup.go:253: Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/15 07:24:41 Rebalance progress: 12.5
2021/12/15 07:24:46 Rebalance progress: 12.5
2021/12/15 07:24:51 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:24:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:24:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 07:24:51 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 07:24:56 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/15 07:25:01 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/15 07:25:06 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/15 07:25:11 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/15 07:25:11 clusterSetup.go:405: App: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings got un-deployed
2021/12/15 07:25:11 eventingSetup.go:46: No eventing-con process running
2021/12/15 07:25:11 eventingSetup.go:418: Function: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings deleting in the background"
}
2021/12/15 07:25:13 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:25:18 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 07:25:23 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 10702
2021/12/15 07:25:28 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 10702
2021/12/15 07:25:28 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 07:25:30 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:25:35 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 07:25:35 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings (185.72s)
=== RUN   TestEventingSwapRebOnUpdateBucketOp
2021/12/15 07:25:45 clusterSetup.go:397: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get un-deployed
2021/12/15 07:25:45 clusterSetup.go:405: App: TestEventingSwapRebOnUpdateBucketOp got un-deployed
2021/12/15 07:25:45 eventingSetup.go:46: No eventing-con process running
2021/12/15 07:25:45 eventingSetup.go:418: Function: TestEventingSwapRebOnUpdateBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingSwapRebOnUpdateBucketOp not found"
 }
}
2021/12/15 07:25:47 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:25:52 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 07:25:52 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 07:25:53 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:25:58 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 07:25:58 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:25:58 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 07:25:58 eventingSetup.go:125: Deploying app: TestEventingSwapRebOnUpdateBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/15 07:26:00 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingSwapRebOnUpdateBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/15 07:26:05 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/15 07:26:10 clusterSetup.go:346: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get deployed
2021/12/15 07:26:15 clusterSetup.go:346: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get deployed
2021/12/15 07:26:20 clusterSetup.go:346: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get deployed
2021/12/15 07:26:20 clusterSetup.go:354: App: TestEventingSwapRebOnUpdateBucketOp got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:26:20 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:26:20 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 07:26:20 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/15 07:26:20 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/15 07:26:25 Rebalance progress: 12.5
2021/12/15 07:26:30 Rebalance progress: 12.5
2021/12/15 07:26:35 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:26:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:26:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 07:26:35 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/15 07:26:36 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/15 07:26:41 Rebalance progress: 12.5
2021/12/15 07:26:46 Rebalance progress: 12.5
2021/12/15 07:26:51 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:26:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:26:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 07:26:51 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/15 07:26:56 Rebalance progress: 12.5
2021/12/15 07:27:01 Rebalance progress: 12.5
2021/12/15 07:27:06 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:27:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:27:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 07:27:06 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 07:27:11 clusterSetup.go:397: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get un-deployed
2021/12/15 07:27:16 clusterSetup.go:397: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get un-deployed
2021/12/15 07:27:21 clusterSetup.go:397: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get un-deployed
2021/12/15 07:27:26 clusterSetup.go:397: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get un-deployed
2021/12/15 07:27:26 clusterSetup.go:405: App: TestEventingSwapRebOnUpdateBucketOp got un-deployed
2021/12/15 07:27:26 eventingSetup.go:46: No eventing-con process running
2021/12/15 07:27:26 eventingSetup.go:418: Function: TestEventingSwapRebOnUpdateBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingSwapRebOnUpdateBucketOp deleting in the background"
}
2021/12/15 07:27:28 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:27:33 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 07:27:38 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6108
2021/12/15 07:27:38 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 07:27:40 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:27:45 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 07:27:45 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingSwapRebOnUpdateBucketOp (129.64s)
=== RUN   TestMetaRollbackWithEventingReb
2021/12/15 07:27:50 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/15 07:27:50 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/15 07:27:55 Rebalance progress: 100
2021/12/15 07:28:00 Waiting for app: TestMetaRollbackWithEventingReb to get un-deployed
2021/12/15 07:28:00 App: TestMetaRollbackWithEventingReb got un-deployed
2021/12/15 07:28:00 No eventing-con process running
2021/12/15 07:28:00 Function: TestMetaRollbackWithEventingReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMetaRollbackWithEventingReb not found"
 }
}
2021/12/15 07:28:02 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:28:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 07:28:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 07:28:08 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:28:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 07:28:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:28:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 07:28:13 Deploying app: TestMetaRollbackWithEventingReb. Item count src bucket: 0 dst bucket: 0
2021/12/15 07:28:15 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMetaRollbackWithEventingReb' in metakv",
  "warnings": null
 }
}
2021/12/15 07:28:20 Starting bucket ops to source bucket
2021/12/15 07:28:25 Waiting for app: TestMetaRollbackWithEventingReb to get deployed
2021/12/15 07:28:30 Waiting for app: TestMetaRollbackWithEventingReb to get deployed
2021/12/15 07:28:35 Waiting for app: TestMetaRollbackWithEventingReb to get deployed
2021/12/15 07:28:35 App: TestMetaRollbackWithEventingReb got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:28:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:28:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 07:28:35 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/15 07:28:35 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/15 07:28:40 Rebalance progress: 12.5
2021/12/15 07:28:45 Rebalance progress: 12.5
2021/12/15 07:28:50 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:28:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:28:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 07:28:50 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002] from the cluster
2021/12/15 07:28:50 bucketOps.go:134: Metakv lookup failed, err: Get "/_metakv/eventing/tempApps/TestMetaRollbackWithEventingReb/0": Unable to initialize cbauth's revrpc: cbauth environment variable CBAUTH_REVRPC_URL is not set
2021/12/15 07:28:55 Rebalance progress: 12.5
2021/12/15 07:29:00 Rebalance progress: 12.5
2021/12/15 07:29:05 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:29:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:29:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 07:29:05 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 07:29:10 clusterSetup.go:397: Waiting for app: TestMetaRollbackWithEventingReb to get un-deployed
2021/12/15 07:29:15 clusterSetup.go:397: Waiting for app: TestMetaRollbackWithEventingReb to get un-deployed
2021/12/15 07:29:20 clusterSetup.go:397: Waiting for app: TestMetaRollbackWithEventingReb to get un-deployed
2021/12/15 07:29:25 clusterSetup.go:397: Waiting for app: TestMetaRollbackWithEventingReb to get un-deployed
2021/12/15 07:29:25 clusterSetup.go:405: App: TestMetaRollbackWithEventingReb got un-deployed
2021/12/15 07:29:25 eventingSetup.go:46: No eventing-con process running
2021/12/15 07:29:25 eventingSetup.go:418: Function: TestMetaRollbackWithEventingReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMetaRollbackWithEventingReb deleting in the background"
}
2021/12/15 07:29:27 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:29:32 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 07:29:37 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4564
2021/12/15 07:29:37 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 07:29:39 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:29:44 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 07:29:49 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4564
2021/12/15 07:29:49 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestMetaRollbackWithEventingReb (124.48s)
=== RUN   TestMetaPartialRollbackWithEventingReb
2021/12/15 07:29:54 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/15 07:29:54 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/15 07:29:59 Rebalance progress: 100
2021/12/15 07:30:04 Waiting for app: TestMetaPartialRollbackWithEventingReb to get un-deployed
2021/12/15 07:30:04 App: TestMetaPartialRollbackWithEventingReb got un-deployed
2021/12/15 07:30:04 No eventing-con process running
2021/12/15 07:30:04 Function: TestMetaPartialRollbackWithEventingReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMetaPartialRollbackWithEventingReb not found"
 }
}
2021/12/15 07:30:06 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:30:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 07:30:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 07:30:13 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:30:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 07:30:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:30:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 07:30:18 Deploying app: TestMetaPartialRollbackWithEventingReb. Item count src bucket: 0 dst bucket: 0
2021/12/15 07:30:19 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMetaPartialRollbackWithEventingReb' in metakv",
  "warnings": null
 }
}
2021/12/15 07:30:24 Starting bucket ops to source bucket
2021/12/15 07:30:29 Waiting for app: TestMetaPartialRollbackWithEventingReb to get deployed
2021/12/15 07:30:34 Waiting for app: TestMetaPartialRollbackWithEventingReb to get deployed
2021/12/15 07:30:39 Waiting for app: TestMetaPartialRollbackWithEventingReb to get deployed
2021/12/15 07:30:39 App: TestMetaPartialRollbackWithEventingReb got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:30:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:30:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 07:30:39 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/15 07:30:39 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/15 07:30:44 Rebalance progress: 12.5
2021/12/15 07:30:49 Rebalance progress: 12.5
2021/12/15 07:30:54 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:30:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:30:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 07:30:54 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002] from the cluster
2021/12/15 07:30:54 bucketOps.go:65: Metakv lookup failed, err: Get "/_metakv/eventing/tempApps/TestMetaPartialRollbackWithEventingReb/0": Unable to initialize cbauth's revrpc: cbauth environment variable CBAUTH_REVRPC_URL is not set
2021/12/15 07:31:00 Rebalance progress: 12.5
2021/12/15 07:31:05 Rebalance progress: 12.5
2021/12/15 07:31:10 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:31:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:31:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 07:31:10 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 07:31:10 bucketOps.go:65: Metakv lookup failed, err: Get "/_metakv/eventing/tempApps/TestMetaPartialRollbackWithEventingReb/0": Unable to initialize cbauth's revrpc: cbauth environment variable CBAUTH_REVRPC_URL is not set
2021/12/15 07:31:15 clusterSetup.go:397: Waiting for app: TestMetaPartialRollbackWithEventingReb to get un-deployed
2021/12/15 07:31:20 clusterSetup.go:397: Waiting for app: TestMetaPartialRollbackWithEventingReb to get un-deployed
2021/12/15 07:31:25 clusterSetup.go:397: Waiting for app: TestMetaPartialRollbackWithEventingReb to get un-deployed
2021/12/15 07:31:30 clusterSetup.go:397: Waiting for app: TestMetaPartialRollbackWithEventingReb to get un-deployed
2021/12/15 07:31:30 clusterSetup.go:405: App: TestMetaPartialRollbackWithEventingReb got un-deployed
2021/12/15 07:31:30 eventingSetup.go:46: No eventing-con process running
2021/12/15 07:31:30 eventingSetup.go:418: Function: TestMetaPartialRollbackWithEventingReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMetaPartialRollbackWithEventingReb deleting in the background"
}
2021/12/15 07:31:31 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:31:36 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 07:31:41 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 37496
2021/12/15 07:31:47 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 37496
2021/12/15 07:31:47 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 07:31:48 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:31:53 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 07:31:58 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 37496
2021/12/15 07:31:58 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestMetaPartialRollbackWithEventingReb (129.32s)
=== RUN   TestEventingRebMultipleHandlersOneByOne
2021/12/15 07:32:08 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get un-deployed
2021/12/15 07:32:08 clusterSetup.go:405: App: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op got un-deployed
2021/12/15 07:32:08 eventingSetup.go:46: No eventing-con process running
2021/12/15 07:32:08 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op not found"
 }
}
2021/12/15 07:32:10 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:32:15 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 07:32:15 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 07:32:17 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:32:22 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 07:32:22 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:32:27 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 07:32:27 eventingSetup.go:125: Deploying app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op. Item count src bucket: 0 dst bucket: 0
2021/12/15 07:32:28 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op' in metakv",
  "warnings": null
 }
}
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:32:28 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 07:32:28 eventingSetup.go:125: Deploying app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/15 07:32:29 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebMultipleHandlersOneByOnebucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/15 07:32:34 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/15 07:32:39 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get deployed
2021/12/15 07:32:44 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get deployed
2021/12/15 07:32:49 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get deployed
2021/12/15 07:32:49 clusterSetup.go:354: App: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op got deployed
2021/12/15 07:32:54 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get deployed
2021/12/15 07:32:59 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get deployed
2021/12/15 07:33:04 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get deployed
2021/12/15 07:33:04 clusterSetup.go:354: App: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:33:04 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:33:04 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 07:33:04 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/15 07:33:05 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/15 07:33:10 Rebalance progress: 12.5
2021/12/15 07:33:15 Rebalance progress: 12.5
2021/12/15 07:33:20 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:33:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:33:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 07:33:20 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/15 07:33:20 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/15 07:33:25 Rebalance progress: 12.5
2021/12/15 07:33:30 Rebalance progress: 12.5
2021/12/15 07:33:35 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:33:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:33:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 07:33:35 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/15 07:33:36 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/15 07:33:41 Rebalance progress: 12.5
2021/12/15 07:33:46 Rebalance progress: 12.5
2021/12/15 07:33:51 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:33:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:33:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 07:33:51 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/15 07:33:56 Rebalance progress: 12.5
2021/12/15 07:34:01 Rebalance progress: 12.5
2021/12/15 07:34:06 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:34:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:34:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 07:34:06 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/15 07:34:11 Rebalance progress: 12.5
2021/12/15 07:34:16 Rebalance progress: 12.5
2021/12/15 07:34:21 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:34:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:34:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 07:34:21 clusterSetup.go:253: Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/15 07:34:26 Rebalance progress: 12.5
2021/12/15 07:34:31 Rebalance progress: 12.5
2021/12/15 07:34:36 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:34:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:34:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 07:34:36 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 07:34:41 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get un-deployed
2021/12/15 07:34:46 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get un-deployed
2021/12/15 07:34:51 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get un-deployed
2021/12/15 07:34:56 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get un-deployed
2021/12/15 07:34:56 clusterSetup.go:405: App: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op got un-deployed
2021/12/15 07:34:56 eventingSetup.go:46: No eventing-con process running
2021/12/15 07:34:56 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op deleting in the background"
}
2021/12/15 07:35:00 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:35:05 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 07:35:05 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 07:35:06 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:35:11 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 07:35:17 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12184
2021/12/15 07:35:17 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 07:35:17 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 07:35:22 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get un-deployed
2021/12/15 07:35:27 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get un-deployed
2021/12/15 07:35:32 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get un-deployed
2021/12/15 07:35:37 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get un-deployed
2021/12/15 07:35:37 clusterSetup.go:405: App: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update got un-deployed
2021/12/15 07:35:37 eventingSetup.go:46: No eventing-con process running
2021/12/15 07:35:37 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update deleting in the background"
}
2021/12/15 07:35:38 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:35:43 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 07:35:43 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 07:35:45 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:35:50 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 07:35:50 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebMultipleHandlersOneByOne (231.94s)
=== RUN   TestEventingRebMultipleHandlersAllAtOnce
2021/12/15 07:36:00 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get un-deployed
2021/12/15 07:36:00 clusterSetup.go:405: App: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op got un-deployed
2021/12/15 07:36:00 eventingSetup.go:46: No eventing-con process running
2021/12/15 07:36:00 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op not found"
 }
}
2021/12/15 07:36:02 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:36:07 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 07:36:07 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 07:36:09 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:36:14 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 07:36:14 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:36:19 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 07:36:19 eventingSetup.go:125: Deploying app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op. Item count src bucket: 0 dst bucket: 0
2021/12/15 07:36:20 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op' in metakv",
  "warnings": null
 }
}
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:36:20 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 07:36:20 eventingSetup.go:125: Deploying app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/15 07:36:21 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/15 07:36:26 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/15 07:36:31 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get deployed
2021/12/15 07:36:36 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get deployed
2021/12/15 07:36:41 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get deployed
2021/12/15 07:36:41 clusterSetup.go:354: App: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op got deployed
2021/12/15 07:36:46 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get deployed
2021/12/15 07:36:51 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get deployed
2021/12/15 07:36:56 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get deployed
2021/12/15 07:36:56 clusterSetup.go:354: App: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:36:56 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:36:56 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 07:36:56 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/15 07:36:57 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/15 07:36:57 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/15 07:36:57 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/15 07:37:02 Rebalance progress: 12.5
2021/12/15 07:37:07 Rebalance progress: 12.5
2021/12/15 07:37:12 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:37:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:37:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 07:37:12 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002 127.0.0.1:19003] from the cluster
2021/12/15 07:37:18 Rebalance progress: 12.5
2021/12/15 07:37:23 Rebalance progress: 12.5
2021/12/15 07:37:28 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:37:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:37:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 07:37:28 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 07:37:33 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get un-deployed
2021/12/15 07:37:38 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get un-deployed
2021/12/15 07:37:43 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get un-deployed
2021/12/15 07:37:48 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get un-deployed
2021/12/15 07:37:48 clusterSetup.go:405: App: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op got un-deployed
2021/12/15 07:37:48 eventingSetup.go:46: No eventing-con process running
2021/12/15 07:37:48 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op deleting in the background"
}
2021/12/15 07:37:51 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:37:56 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 07:38:01 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2319
2021/12/15 07:38:06 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2319
2021/12/15 07:38:06 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 07:38:08 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:38:13 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 07:38:18 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6136
2021/12/15 07:38:18 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 07:38:18 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 07:38:23 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get un-deployed
2021/12/15 07:38:28 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get un-deployed
2021/12/15 07:38:33 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get un-deployed
2021/12/15 07:38:38 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get un-deployed
2021/12/15 07:38:38 clusterSetup.go:405: App: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update got un-deployed
2021/12/15 07:38:38 eventingSetup.go:46: No eventing-con process running
2021/12/15 07:38:38 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update deleting in the background"
}
2021/12/15 07:38:40 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:38:45 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 07:38:45 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 07:38:46 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:38:51 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 07:38:51 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebMultipleHandlersAllAtOnce (181.16s)
=== RUN   TestEventingSwapRebMultipleHandlers
2021/12/15 07:39:01 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get un-deployed
2021/12/15 07:39:01 clusterSetup.go:405: App: TestEventingSwapRebMultipleHandlersbucket_op_on_update got un-deployed
2021/12/15 07:39:01 eventingSetup.go:46: No eventing-con process running
2021/12/15 07:39:01 eventingSetup.go:418: Function: TestEventingSwapRebMultipleHandlersbucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingSwapRebMultipleHandlersbucket_op_on_update not found"
 }
}
2021/12/15 07:39:03 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:39:08 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 07:39:08 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 07:39:10 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:39:15 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 07:39:15 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 07:39:20 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get un-deployed
2021/12/15 07:39:20 clusterSetup.go:405: App: TestEventingSwapRebMultipleHandlerssys_test_bucket_op got un-deployed
2021/12/15 07:39:20 eventingSetup.go:46: No eventing-con process running
2021/12/15 07:39:20 eventingSetup.go:418: Function: TestEventingSwapRebMultipleHandlerssys_test_bucket_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingSwapRebMultipleHandlerssys_test_bucket_op not found"
 }
}
2021/12/15 07:39:22 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:39:27 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 07:39:27 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 07:39:29 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:39:34 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 07:39:34 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:39:34 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 07:39:34 eventingSetup.go:125: Deploying app: TestEventingSwapRebMultipleHandlersbucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/15 07:39:35 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingSwapRebMultipleHandlersbucket_op_on_update' in metakv",
  "warnings": null
 }
}
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:39:35 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 07:39:35 eventingSetup.go:125: Deploying app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op. Item count src bucket: 0 dst bucket: 0
2021/12/15 07:39:36 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingSwapRebMultipleHandlerssys_test_bucket_op' in metakv",
  "warnings": null
 }
}
2021/12/15 07:39:41 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/15 07:39:46 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get deployed
2021/12/15 07:39:51 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get deployed
2021/12/15 07:39:56 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get deployed
2021/12/15 07:39:56 clusterSetup.go:354: App: TestEventingSwapRebMultipleHandlersbucket_op_on_update got deployed
2021/12/15 07:40:01 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get deployed
2021/12/15 07:40:06 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get deployed
2021/12/15 07:40:11 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get deployed
2021/12/15 07:40:11 clusterSetup.go:354: App: TestEventingSwapRebMultipleHandlerssys_test_bucket_op got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:40:11 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:40:11 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 07:40:11 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/15 07:40:11 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/15 07:40:17 Rebalance progress: 12.5
2021/12/15 07:40:22 Rebalance progress: 12.5
2021/12/15 07:40:27 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:40:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:40:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 07:40:27 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/15 07:40:27 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/15 07:40:32 Rebalance progress: 12.5
2021/12/15 07:40:37 Rebalance progress: 12.5
2021/12/15 07:40:42 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:40:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:40:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 07:40:42 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/15 07:40:47 Rebalance progress: 12.5
2021/12/15 07:40:52 Rebalance progress: 12.5
2021/12/15 07:40:57 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:40:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:40:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 07:40:57 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 07:41:02 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get un-deployed
2021/12/15 07:41:07 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get un-deployed
2021/12/15 07:41:12 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get un-deployed
2021/12/15 07:41:17 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get un-deployed
2021/12/15 07:41:17 clusterSetup.go:405: App: TestEventingSwapRebMultipleHandlersbucket_op_on_update got un-deployed
2021/12/15 07:41:17 eventingSetup.go:46: No eventing-con process running
2021/12/15 07:41:17 eventingSetup.go:418: Function: TestEventingSwapRebMultipleHandlersbucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingSwapRebMultipleHandlersbucket_op_on_update deleting in the background"
}
2021/12/15 07:41:21 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:41:26 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 07:41:31 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1509
2021/12/15 07:41:36 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1509
2021/12/15 07:41:36 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 07:41:38 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:41:43 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 07:41:48 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7602
2021/12/15 07:41:48 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 07:41:48 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 07:41:53 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get un-deployed
2021/12/15 07:41:58 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get un-deployed
2021/12/15 07:42:03 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get un-deployed
2021/12/15 07:42:08 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get un-deployed
2021/12/15 07:42:08 clusterSetup.go:405: App: TestEventingSwapRebMultipleHandlerssys_test_bucket_op got un-deployed
2021/12/15 07:42:08 eventingSetup.go:46: No eventing-con process running
2021/12/15 07:42:08 eventingSetup.go:418: Function: TestEventingSwapRebMultipleHandlerssys_test_bucket_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingSwapRebMultipleHandlerssys_test_bucket_op deleting in the background"
}
2021/12/15 07:42:10 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:42:15 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 07:42:15 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 07:42:17 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:42:22 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 07:42:22 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingSwapRebMultipleHandlers (210.18s)
=== RUN   TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne
2021/12/15 07:42:32 clusterSetup.go:397: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/15 07:42:32 clusterSetup.go:405: App: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/15 07:42:32 eventingSetup.go:46: No eventing-con process running
2021/12/15 07:42:32 eventingSetup.go:418: Function: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne not found"
 }
}
2021/12/15 07:42:33 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:42:38 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 07:42:38 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 07:42:40 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:42:45 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 07:42:45 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:42:50 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 07:42:50 eventingSetup.go:125: Deploying app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne. Item count src bucket: 0 dst bucket: 0
2021/12/15 07:42:51 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne' in metakv",
  "warnings": null
 }
}
2021/12/15 07:42:56 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/15 07:43:01 clusterSetup.go:346: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/15 07:43:06 clusterSetup.go:346: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/15 07:43:11 clusterSetup.go:346: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/15 07:43:11 clusterSetup.go:354: App: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:43:11 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:43:11 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 07:43:11 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/15 07:43:12 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/15 07:43:32 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:43:33 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:43:33 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:44:03 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:44:03 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 07:44:03 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/15 07:44:08 Rebalance progress: 12.5
2021/12/15 07:44:13 Rebalance progress: 12.5
2021/12/15 07:44:18 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:44:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:44:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 07:44:18 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/15 07:44:18 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/15 07:44:23 Rebalance progress: 12.5
2021/12/15 07:44:28 Rebalance progress: 12.5
2021/12/15 07:44:33 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:44:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:44:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 07:44:33 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/15 07:44:38 Rebalance progress: 12.5
2021/12/15 07:44:43 Rebalance progress: 12.5
2021/12/15 07:44:48 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:44:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:44:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 07:44:49 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 07:44:54 clusterSetup.go:397: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/15 07:44:59 clusterSetup.go:397: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/15 07:45:04 clusterSetup.go:397: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/15 07:45:09 clusterSetup.go:397: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/15 07:45:09 clusterSetup.go:405: App: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/15 07:45:09 eventingSetup.go:46: No eventing-con process running
2021/12/15 07:45:09 eventingSetup.go:418: Function: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne deleting in the background"
}
2021/12/15 07:45:10 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:45:15 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 07:45:21 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6422
2021/12/15 07:45:26 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6422
2021/12/15 07:45:26 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 07:45:27 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:45:32 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 07:45:37 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 11199
2021/12/15 07:45:37 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne (195.71s)
=== RUN   TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne
2021/12/15 07:45:47 clusterSetup.go:397: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/15 07:45:47 clusterSetup.go:405: App: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/15 07:45:47 eventingSetup.go:46: No eventing-con process running
2021/12/15 07:45:47 eventingSetup.go:418: Function: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne not found"
 }
}
2021/12/15 07:45:49 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:45:54 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 07:45:54 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 07:45:56 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:46:01 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 07:46:01 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:46:06 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 07:46:06 eventingSetup.go:125: Deploying app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne. Item count src bucket: 0 dst bucket: 0
2021/12/15 07:46:07 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne' in metakv",
  "warnings": null
 }
}
2021/12/15 07:46:12 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/15 07:46:17 clusterSetup.go:346: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/15 07:46:22 clusterSetup.go:346: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/15 07:46:27 clusterSetup.go:346: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/15 07:46:27 clusterSetup.go:354: App: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:46:27 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:46:27 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 07:46:27 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/15 07:46:27 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/15 07:46:47 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:46:48 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:46:48 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:46:58 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:46:58 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 07:46:58 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/15 07:46:58 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/15 07:47:18 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:47:19 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:47:19 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:47:29 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:47:29 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 07:47:29 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/15 07:47:29 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/15 07:47:49 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:47:50 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:47:50 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:48:00 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:48:00 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 07:48:00 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/15 07:48:00 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/15 07:48:20 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:48:21 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:48:21 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:48:31 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:48:31 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 07:48:31 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/15 07:48:31 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/15 07:48:51 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:48:51 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:48:51 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:49:01 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:49:01 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 07:49:01 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/15 07:49:06 Rebalance progress: 12.5
2021/12/15 07:49:11 Rebalance progress: 12.5
2021/12/15 07:49:16 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:49:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:49:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 07:49:16 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/15 07:49:17 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/15 07:49:22 Rebalance progress: 12.5
2021/12/15 07:49:27 Rebalance progress: 12.5
2021/12/15 07:49:32 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:49:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:49:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 07:49:32 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/15 07:49:37 Rebalance progress: 12.5
2021/12/15 07:49:42 Rebalance progress: 12.5
2021/12/15 07:49:47 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:49:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:49:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 07:49:47 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 07:49:52 clusterSetup.go:397: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/15 07:49:57 clusterSetup.go:397: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/15 07:50:02 clusterSetup.go:397: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/15 07:50:07 clusterSetup.go:397: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/15 07:50:07 clusterSetup.go:405: App: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/15 07:50:07 eventingSetup.go:46: No eventing-con process running
2021/12/15 07:50:07 eventingSetup.go:418: Function: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne deleting in the background"
}
2021/12/15 07:50:09 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:50:14 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 07:50:19 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 21515
2021/12/15 07:50:19 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 07:50:21 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:50:26 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 07:50:31 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 20430
2021/12/15 07:50:36 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 20430
2021/12/15 07:50:36 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne (298.67s)
=== RUN   TestEventingFailoverOnUpdateBucketOp
2021/12/15 07:50:46 clusterSetup.go:397: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get un-deployed
2021/12/15 07:50:46 clusterSetup.go:405: App: TestEventingFailoverOnUpdateBucketOp got un-deployed
2021/12/15 07:50:46 eventingSetup.go:46: No eventing-con process running
2021/12/15 07:50:46 eventingSetup.go:418: Function: TestEventingFailoverOnUpdateBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingFailoverOnUpdateBucketOp not found"
 }
}
2021/12/15 07:50:48 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:50:53 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 07:50:53 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 07:50:55 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:51:00 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 07:51:00 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:51:05 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 07:51:05 eventingSetup.go:125: Deploying app: TestEventingFailoverOnUpdateBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/15 07:51:06 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingFailoverOnUpdateBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/15 07:51:11 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/15 07:51:16 clusterSetup.go:346: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get deployed
2021/12/15 07:51:21 clusterSetup.go:346: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get deployed
2021/12/15 07:51:26 clusterSetup.go:346: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get deployed
2021/12/15 07:51:26 clusterSetup.go:354: App: TestEventingFailoverOnUpdateBucketOp got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:51:26 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:51:26 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 07:51:26 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/15 07:51:26 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/15 07:51:31 Rebalance progress: 12.5
2021/12/15 07:51:36 Rebalance progress: 12.5
2021/12/15 07:51:41 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:51:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:51:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 07:51:41 clusterSetup.go:229: Failing over: [127.0.0.1:19001]
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:51:51 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:51:51 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 07:51:51 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/15 07:51:56 Rebalance progress: 12.5
2021/12/15 07:52:01 Rebalance progress: 12.5
2021/12/15 07:52:06 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:52:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:52:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 07:52:06 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/15 07:52:07 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/15 07:52:12 Rebalance progress: 12.5
2021/12/15 07:52:17 Rebalance progress: 12.5
2021/12/15 07:52:22 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:52:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:52:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 07:52:22 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/15 07:52:27 Rebalance progress: 12.5
2021/12/15 07:52:32 Rebalance progress: 12.5
2021/12/15 07:52:37 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:52:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:52:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 07:52:37 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 07:52:42 clusterSetup.go:397: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get un-deployed
2021/12/15 07:52:47 clusterSetup.go:397: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get un-deployed
2021/12/15 07:52:52 clusterSetup.go:397: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get un-deployed
2021/12/15 07:52:57 clusterSetup.go:397: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get un-deployed
2021/12/15 07:52:57 clusterSetup.go:405: App: TestEventingFailoverOnUpdateBucketOp got un-deployed
2021/12/15 07:52:57 eventingSetup.go:46: No eventing-con process running
2021/12/15 07:52:57 eventingSetup.go:418: Function: TestEventingFailoverOnUpdateBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingFailoverOnUpdateBucketOp deleting in the background"
}
2021/12/15 07:52:59 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:53:04 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 07:53:09 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 8624
2021/12/15 07:53:09 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 07:53:11 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:53:16 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 07:53:21 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6634
2021/12/15 07:53:26 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6634
2021/12/15 07:53:26 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingFailoverOnUpdateBucketOp (169.80s)
=== RUN   TestEventingKVRebalanceOnUpdateBucketOp
2021/12/15 07:53:36 clusterSetup.go:397: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get un-deployed
2021/12/15 07:53:36 clusterSetup.go:405: App: TestEventingKVRebalanceOnUpdateBucketOp got un-deployed
2021/12/15 07:53:36 eventingSetup.go:46: No eventing-con process running
2021/12/15 07:53:36 eventingSetup.go:418: Function: TestEventingKVRebalanceOnUpdateBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingKVRebalanceOnUpdateBucketOp not found"
 }
}
2021/12/15 07:53:38 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:53:43 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 07:53:43 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 07:53:44 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:53:49 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 07:53:49 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:53:54 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 07:53:54 eventingSetup.go:125: Deploying app: TestEventingKVRebalanceOnUpdateBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/15 07:53:55 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingKVRebalanceOnUpdateBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/15 07:54:00 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/15 07:54:05 clusterSetup.go:346: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get deployed
2021/12/15 07:54:10 clusterSetup.go:346: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get deployed
2021/12/15 07:54:15 clusterSetup.go:346: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get deployed
2021/12/15 07:54:15 clusterSetup.go:354: App: TestEventingKVRebalanceOnUpdateBucketOp got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:54:15 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:54:15 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 07:54:16 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing,kv to the cluster
2021/12/15 07:54:16 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/15 07:54:21 Rebalance progress: 12.5
2021/12/15 07:54:26 Rebalance progress: 12.5
2021/12/15 07:54:31 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:54:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:54:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 07:54:31 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing,kv to the cluster
2021/12/15 07:54:31 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/15 07:54:36 Rebalance progress: 12.5
2021/12/15 07:54:41 Rebalance progress: 12.5
2021/12/15 07:54:46 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:54:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:54:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 07:54:46 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/15 07:54:51 Rebalance progress: 12.5
2021/12/15 07:54:56 Rebalance progress: 12.5
2021/12/15 07:55:01 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:55:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:55:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 07:55:01 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 07:55:06 clusterSetup.go:397: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get un-deployed
2021/12/15 07:55:12 clusterSetup.go:397: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get un-deployed
2021/12/15 07:55:17 clusterSetup.go:397: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get un-deployed
2021/12/15 07:55:22 clusterSetup.go:397: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get un-deployed
2021/12/15 07:55:22 clusterSetup.go:405: App: TestEventingKVRebalanceOnUpdateBucketOp got un-deployed
2021/12/15 07:55:22 eventingSetup.go:46: No eventing-con process running
2021/12/15 07:55:22 eventingSetup.go:418: Function: TestEventingKVRebalanceOnUpdateBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingKVRebalanceOnUpdateBucketOp deleting in the background"
}
2021/12/15 07:55:23 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:55:28 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 07:55:33 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6075
2021/12/15 07:55:38 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6075
2021/12/15 07:55:38 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 07:55:40 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:55:45 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 07:55:50 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1193
2021/12/15 07:55:55 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1193
2021/12/15 07:55:55 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingKVRebalanceOnUpdateBucketOp (149.33s)
=== RUN   TestEventingRebInWithOps
2021/12/15 07:56:05 clusterSetup.go:397: Waiting for app: TestEventingRebInWithOps to get un-deployed
2021/12/15 07:56:05 clusterSetup.go:405: App: TestEventingRebInWithOps got un-deployed
2021/12/15 07:56:05 eventingSetup.go:46: No eventing-con process running
2021/12/15 07:56:05 eventingSetup.go:418: Function: TestEventingRebInWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebInWithOps not found"
 }
}
2021/12/15 07:56:07 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:56:12 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 07:56:12 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 07:56:14 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:56:19 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 07:56:19 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:56:24 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 07:56:24 eventingSetup.go:125: Deploying app: TestEventingRebInWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/15 07:56:25 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebInWithOps' in metakv",
  "warnings": null
 }
}
2021/12/15 07:56:30 clusterSetup.go:346: Waiting for app: TestEventingRebInWithOps to get deployed
2021/12/15 07:56:35 clusterSetup.go:346: Waiting for app: TestEventingRebInWithOps to get deployed
2021/12/15 07:56:40 clusterSetup.go:346: Waiting for app: TestEventingRebInWithOps to get deployed
2021/12/15 07:56:45 clusterSetup.go:346: Waiting for app: TestEventingRebInWithOps to get deployed
2021/12/15 07:56:45 clusterSetup.go:354: App: TestEventingRebInWithOps got deployed
2021/12/15 07:56:45 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/15 07:56:45 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/15 07:56:45 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/15 07:56:50 Rebalance progress: 12.5
2021/12/15 07:56:55 Rebalance progress: 12.5
2021/12/15 07:57:00 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:57:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:57:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 07:57:00 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/15 07:57:05 Rebalance progress: 12.5
2021/12/15 07:57:10 Rebalance progress: 12.5
2021/12/15 07:57:15 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:57:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 07:57:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 07:57:15 Waiting for src bucket item count to get to: 200000 curr count: 36837
2021/12/15 07:57:20 Waiting for src bucket item count to get to: 200000 curr count: 52849
2021/12/15 07:57:25 Waiting for src bucket item count to get to: 200000 curr count: 52849
2021/12/15 07:57:30 Waiting for src bucket item count to get to: 200000 curr count: 69023
2021/12/15 07:57:35 Waiting for src bucket item count to get to: 200000 curr count: 69023
2021/12/15 07:57:40 Waiting for src bucket item count to get to: 200000 curr count: 84172
2021/12/15 07:57:45 Waiting for src bucket item count to get to: 200000 curr count: 84172
2021/12/15 07:57:50 Waiting for src bucket item count to get to: 200000 curr count: 84172
2021/12/15 07:57:55 Waiting for src bucket item count to get to: 200000 curr count: 99891
2021/12/15 07:58:00 Waiting for src bucket item count to get to: 200000 curr count: 99891
2021/12/15 07:58:05 Waiting for src bucket item count to get to: 200000 curr count: 115845
2021/12/15 07:58:10 Waiting for src bucket item count to get to: 200000 curr count: 115845
2021/12/15 07:58:15 Waiting for src bucket item count to get to: 200000 curr count: 131851
2021/12/15 07:58:20 Waiting for src bucket item count to get to: 200000 curr count: 131851
2021/12/15 07:58:25 Waiting for src bucket item count to get to: 200000 curr count: 147863
2021/12/15 07:58:30 Waiting for src bucket item count to get to: 200000 curr count: 147863
2021/12/15 07:58:35 Waiting for src bucket item count to get to: 200000 curr count: 163857
2021/12/15 07:58:40 Waiting for src bucket item count to get to: 200000 curr count: 163857
2021/12/15 07:58:45 Waiting for src bucket item count to get to: 200000 curr count: 178427
2021/12/15 07:58:50 Waiting for src bucket item count to get to: 200000 curr count: 178427
2021/12/15 07:58:55 Waiting for src bucket item count to get to: 200000 curr count: 194160
2021/12/15 07:59:00 Waiting for src bucket item count to get to: 200000 curr count: 194160
2021/12/15 07:59:05 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/15 07:59:05 Post SET ops. Expected item count: 200000 got 200000
2021/12/15 07:59:05 Starting bucket ops to source bucket
2021/12/15 08:02:56 Waiting for dst bucket item count to get to: 0 curr count: 3324
2021/12/15 08:03:01 Waiting for dst bucket item count to get to: 0 curr count: 3324
2021/12/15 08:03:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 08:03:01 Post DELETE ops. Expected item count: 0 got 0
2021/12/15 08:03:01 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 08:03:06 Waiting for app: TestEventingRebInWithOps to get un-deployed
2021/12/15 08:03:11 Waiting for app: TestEventingRebInWithOps to get un-deployed
2021/12/15 08:03:16 Waiting for app: TestEventingRebInWithOps to get un-deployed
2021/12/15 08:03:21 Waiting for app: TestEventingRebInWithOps to get un-deployed
2021/12/15 08:03:21 App: TestEventingRebInWithOps got un-deployed
2021/12/15 08:03:21 No eventing-con process running
2021/12/15 08:03:21 Function: TestEventingRebInWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebInWithOps deleting in the background"
}
2021/12/15 08:03:23 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 08:03:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 08:03:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 08:03:30 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 08:03:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 08:03:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebInWithOps (459.83s)
=== RUN   TestEventingRebOutWithOps
2021/12/15 08:03:40 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/15 08:03:40 Removing node(s): [] from the cluster
2021/12/15 08:03:45 Rebalance progress: 100
2021/12/15 08:03:50 Waiting for app: TestEventingRebOutWithOps to get un-deployed
2021/12/15 08:03:50 App: TestEventingRebOutWithOps got un-deployed
2021/12/15 08:03:50 No eventing-con process running
2021/12/15 08:03:50 Function: TestEventingRebOutWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebOutWithOps not found"
 }
}
2021/12/15 08:03:52 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 08:03:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 08:03:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 08:03:59 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 08:04:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 08:04:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 08:04:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 08:04:09 Deploying app: TestEventingRebOutWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/15 08:04:10 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebOutWithOps' in metakv",
  "warnings": null
 }
}
2021/12/15 08:04:15 Waiting for app: TestEventingRebOutWithOps to get deployed
2021/12/15 08:04:20 Waiting for app: TestEventingRebOutWithOps to get deployed
2021/12/15 08:04:25 Waiting for app: TestEventingRebOutWithOps to get deployed
2021/12/15 08:04:30 Waiting for app: TestEventingRebOutWithOps to get deployed
2021/12/15 08:04:30 App: TestEventingRebOutWithOps got deployed
2021/12/15 08:04:30 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/15 08:04:30 Starting bucket ops to source bucket
2021/12/15 08:04:35 Rebalance progress: 12.5
2021/12/15 08:04:40 Rebalance progress: 12.5
2021/12/15 08:04:45 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 08:04:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 08:04:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 08:04:45 Waiting for src bucket item count to get to: 200000 curr count: 14020
2021/12/15 08:04:50 Waiting for src bucket item count to get to: 200000 curr count: 29776
2021/12/15 08:04:55 Waiting for src bucket item count to get to: 200000 curr count: 29776
2021/12/15 08:05:00 Waiting for src bucket item count to get to: 200000 curr count: 45868
2021/12/15 08:05:05 Waiting for src bucket item count to get to: 200000 curr count: 45868
2021/12/15 08:05:10 Waiting for src bucket item count to get to: 200000 curr count: 61933
2021/12/15 08:05:15 Waiting for src bucket item count to get to: 200000 curr count: 61933
2021/12/15 08:05:20 Waiting for src bucket item count to get to: 200000 curr count: 61933
2021/12/15 08:05:25 Waiting for src bucket item count to get to: 200000 curr count: 77169
2021/12/15 08:05:30 Waiting for src bucket item count to get to: 200000 curr count: 77169
2021/12/15 08:05:35 Waiting for src bucket item count to get to: 200000 curr count: 93129
2021/12/15 08:05:40 Waiting for src bucket item count to get to: 200000 curr count: 93129
2021/12/15 08:05:45 Waiting for src bucket item count to get to: 200000 curr count: 109257
2021/12/15 08:05:50 Waiting for src bucket item count to get to: 200000 curr count: 109257
2021/12/15 08:05:55 Waiting for src bucket item count to get to: 200000 curr count: 125505
2021/12/15 08:06:00 Waiting for src bucket item count to get to: 200000 curr count: 125505
2021/12/15 08:06:05 Waiting for src bucket item count to get to: 200000 curr count: 141763
2021/12/15 08:06:10 Waiting for src bucket item count to get to: 200000 curr count: 141763
2021/12/15 08:06:15 Waiting for src bucket item count to get to: 200000 curr count: 157887
2021/12/15 08:06:20 Waiting for src bucket item count to get to: 200000 curr count: 157887
2021/12/15 08:06:25 Waiting for src bucket item count to get to: 200000 curr count: 173120
2021/12/15 08:06:30 Waiting for src bucket item count to get to: 200000 curr count: 173120
2021/12/15 08:06:35 Waiting for src bucket item count to get to: 200000 curr count: 188947
2021/12/15 08:06:40 Waiting for src bucket item count to get to: 200000 curr count: 188947
2021/12/15 08:06:45 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/15 08:06:45 Post SET ops. Expected item count: 200000 got 200000
2021/12/15 08:06:45 Starting bucket ops to source bucket
2021/12/15 08:10:35 Waiting for dst bucket item count to get to: 0 curr count: 19449
2021/12/15 08:10:40 Waiting for dst bucket item count to get to: 0 curr count: 1319
2021/12/15 08:10:45 Waiting for dst bucket item count to get to: 0 curr count: 1319
2021/12/15 08:10:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 08:10:45 Post DELETE ops. Expected item count: 0 got 0
2021/12/15 08:10:45 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 08:10:50 Waiting for app: TestEventingRebOutWithOps to get un-deployed
2021/12/15 08:10:55 Waiting for app: TestEventingRebOutWithOps to get un-deployed
2021/12/15 08:11:00 Waiting for app: TestEventingRebOutWithOps to get un-deployed
2021/12/15 08:11:05 Waiting for app: TestEventingRebOutWithOps to get un-deployed
2021/12/15 08:11:05 App: TestEventingRebOutWithOps got un-deployed
2021/12/15 08:11:05 No eventing-con process running
2021/12/15 08:11:05 Function: TestEventingRebOutWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebOutWithOps deleting in the background"
}
2021/12/15 08:11:07 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 08:11:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 08:11:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 08:11:14 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 08:11:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 08:11:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebOutWithOps (463.97s)
=== RUN   TestEventingSwapRebWithOps
2021/12/15 08:11:29 Waiting for app: TestEventingSwapRebWithOps to get un-deployed
2021/12/15 08:11:29 App: TestEventingSwapRebWithOps got un-deployed
2021/12/15 08:11:29 No eventing-con process running
2021/12/15 08:11:29 Function: TestEventingSwapRebWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingSwapRebWithOps not found"
 }
}
2021/12/15 08:11:31 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 08:11:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 08:11:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 08:11:38 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 08:11:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 08:11:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 08:11:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 08:11:48 Deploying app: TestEventingSwapRebWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/15 08:11:49 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingSwapRebWithOps' in metakv",
  "warnings": null
 }
}
2021/12/15 08:11:54 Waiting for app: TestEventingSwapRebWithOps to get deployed
2021/12/15 08:11:59 Waiting for app: TestEventingSwapRebWithOps to get deployed
2021/12/15 08:12:04 Waiting for app: TestEventingSwapRebWithOps to get deployed
2021/12/15 08:12:09 Waiting for app: TestEventingSwapRebWithOps to get deployed
2021/12/15 08:12:09 App: TestEventingSwapRebWithOps got deployed
2021/12/15 08:12:09 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/15 08:12:09 Starting bucket ops to source bucket
2021/12/15 08:12:09 Removing node(s): [] from the cluster
2021/12/15 08:12:14 Rebalance progress: 12.5
2021/12/15 08:12:19 Rebalance progress: 12.5
2021/12/15 08:12:24 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 08:12:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 08:12:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 08:12:24 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/15 08:12:25 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/15 08:12:30 Rebalance progress: 12.5
2021/12/15 08:12:35 Rebalance progress: 12.5
2021/12/15 08:12:40 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 08:12:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 08:12:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 08:12:40 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/15 08:12:45 Rebalance progress: 12.5
2021/12/15 08:12:50 Rebalance progress: 12.5
2021/12/15 08:12:55 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 08:12:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 08:12:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 08:12:55 Waiting for src bucket item count to get to: 200000 curr count: 61998
2021/12/15 08:13:00 Waiting for src bucket item count to get to: 200000 curr count: 61998
2021/12/15 08:13:05 Waiting for src bucket item count to get to: 200000 curr count: 76634
2021/12/15 08:13:10 Waiting for src bucket item count to get to: 200000 curr count: 76634
2021/12/15 08:13:15 Waiting for src bucket item count to get to: 200000 curr count: 92416
2021/12/15 08:13:20 Waiting for src bucket item count to get to: 200000 curr count: 92416
2021/12/15 08:13:25 Waiting for src bucket item count to get to: 200000 curr count: 108693
2021/12/15 08:13:30 Waiting for src bucket item count to get to: 200000 curr count: 108693
2021/12/15 08:13:35 Waiting for src bucket item count to get to: 200000 curr count: 124878
2021/12/15 08:13:40 Waiting for src bucket item count to get to: 200000 curr count: 141093
2021/12/15 08:13:45 Waiting for src bucket item count to get to: 200000 curr count: 141093
2021/12/15 08:13:50 Waiting for src bucket item count to get to: 200000 curr count: 157273
2021/12/15 08:13:55 Waiting for src bucket item count to get to: 200000 curr count: 157273
2021/12/15 08:14:00 Waiting for src bucket item count to get to: 200000 curr count: 172371
2021/12/15 08:14:05 Waiting for src bucket item count to get to: 200000 curr count: 172371
2021/12/15 08:14:10 Waiting for src bucket item count to get to: 200000 curr count: 172371
2021/12/15 08:14:15 Waiting for src bucket item count to get to: 200000 curr count: 188178
2021/12/15 08:14:20 Waiting for src bucket item count to get to: 200000 curr count: 188178
2021/12/15 08:14:25 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/15 08:14:25 Post SET ops. Expected item count: 200000 got 200000
2021/12/15 08:14:25 Starting bucket ops to source bucket
2021/12/15 08:18:15 Waiting for dst bucket item count to get to: 0 curr count: 20035
2021/12/15 08:18:20 Waiting for dst bucket item count to get to: 0 curr count: 1969
2021/12/15 08:18:25 Waiting for dst bucket item count to get to: 0 curr count: 1969
2021/12/15 08:18:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 08:18:25 Post DELETE ops. Expected item count: 0 got 0
2021/12/15 08:18:25 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 08:18:30 Waiting for app: TestEventingSwapRebWithOps to get un-deployed
2021/12/15 08:18:36 Waiting for app: TestEventingSwapRebWithOps to get un-deployed
2021/12/15 08:18:41 Waiting for app: TestEventingSwapRebWithOps to get un-deployed
2021/12/15 08:18:46 Waiting for app: TestEventingSwapRebWithOps to get un-deployed
2021/12/15 08:18:46 App: TestEventingSwapRebWithOps got un-deployed
2021/12/15 08:18:46 No eventing-con process running
2021/12/15 08:18:46 Function: TestEventingSwapRebWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingSwapRebWithOps deleting in the background"
}
2021/12/15 08:18:47 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 08:18:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 08:18:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 08:18:54 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 08:18:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 08:18:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingSwapRebWithOps (460.22s)
=== RUN   TestKVRebInWithOps
2021/12/15 08:19:09 Waiting for app: TestKVRebInWithOps to get un-deployed
2021/12/15 08:19:09 App: TestKVRebInWithOps got un-deployed
2021/12/15 08:19:09 No eventing-con process running
2021/12/15 08:19:09 Function: TestKVRebInWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVRebInWithOps not found"
 }
}
2021/12/15 08:19:11 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 08:19:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 08:19:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 08:19:18 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 08:19:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 08:19:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 08:19:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 08:19:28 Deploying app: TestKVRebInWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/15 08:19:29 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVRebInWithOps' in metakv",
  "warnings": null
 }
}
2021/12/15 08:19:34 Waiting for app: TestKVRebInWithOps to get deployed
2021/12/15 08:19:39 Waiting for app: TestKVRebInWithOps to get deployed
2021/12/15 08:19:44 Waiting for app: TestKVRebInWithOps to get deployed
2021/12/15 08:19:49 Waiting for app: TestKVRebInWithOps to get deployed
2021/12/15 08:19:49 App: TestKVRebInWithOps got deployed
2021/12/15 08:19:49 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/15 08:19:49 Starting bucket ops to source bucket
2021/12/15 08:19:49 Removing node(s): [] from the cluster
2021/12/15 08:19:54 Rebalance progress: 12.5
2021/12/15 08:19:59 Rebalance progress: 12.5
2021/12/15 08:20:04 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 08:20:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 08:20:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 08:20:04 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/15 08:20:09 Rebalance progress: 12.5
2021/12/15 08:20:14 Rebalance progress: 12.5
2021/12/15 08:20:19 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 08:20:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 08:20:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 08:20:19 Waiting for src bucket item count to get to: 200000 curr count: 30483
2021/12/15 08:20:24 Waiting for src bucket item count to get to: 200000 curr count: 46502
2021/12/15 08:20:29 Waiting for src bucket item count to get to: 200000 curr count: 46502
2021/12/15 08:20:34 Waiting for src bucket item count to get to: 200000 curr count: 62589
2021/12/15 08:20:39 Waiting for src bucket item count to get to: 200000 curr count: 62589
2021/12/15 08:20:44 Waiting for src bucket item count to get to: 200000 curr count: 77540
2021/12/15 08:20:49 Waiting for src bucket item count to get to: 200000 curr count: 77540
2021/12/15 08:20:54 Waiting for src bucket item count to get to: 200000 curr count: 93436
2021/12/15 08:20:59 Waiting for src bucket item count to get to: 200000 curr count: 93436
2021/12/15 08:21:04 Waiting for src bucket item count to get to: 200000 curr count: 93436
2021/12/15 08:21:09 Waiting for src bucket item count to get to: 200000 curr count: 109241
2021/12/15 08:21:14 Waiting for src bucket item count to get to: 200000 curr count: 109241
2021/12/15 08:21:19 Waiting for src bucket item count to get to: 200000 curr count: 125388
2021/12/15 08:21:24 Waiting for src bucket item count to get to: 200000 curr count: 141456
2021/12/15 08:21:29 Waiting for src bucket item count to get to: 200000 curr count: 141456
2021/12/15 08:21:34 Waiting for src bucket item count to get to: 200000 curr count: 157663
2021/12/15 08:21:39 Waiting for src bucket item count to get to: 200000 curr count: 157663
2021/12/15 08:21:44 Waiting for src bucket item count to get to: 200000 curr count: 172888
2021/12/15 08:21:49 Waiting for src bucket item count to get to: 200000 curr count: 172888
2021/12/15 08:21:54 Waiting for src bucket item count to get to: 200000 curr count: 188698
2021/12/15 08:21:59 Waiting for src bucket item count to get to: 200000 curr count: 188698
2021/12/15 08:22:05 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/15 08:22:05 Post SET ops. Expected item count: 200000 got 200000
2021/12/15 08:22:05 Starting bucket ops to source bucket
2021/12/15 08:25:55 Waiting for dst bucket item count to get to: 0 curr count: 18884
2021/12/15 08:26:00 Waiting for dst bucket item count to get to: 0 curr count: 801
2021/12/15 08:26:05 Waiting for dst bucket item count to get to: 0 curr count: 801
2021/12/15 08:26:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 08:26:05 Post DELETE ops. Expected item count: 0 got 0
2021/12/15 08:26:05 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 08:26:10 Waiting for app: TestKVRebInWithOps to get un-deployed
2021/12/15 08:26:15 Waiting for app: TestKVRebInWithOps to get un-deployed
2021/12/15 08:26:20 Waiting for app: TestKVRebInWithOps to get un-deployed
2021/12/15 08:26:25 Waiting for app: TestKVRebInWithOps to get un-deployed
2021/12/15 08:26:25 App: TestKVRebInWithOps got un-deployed
2021/12/15 08:26:25 No eventing-con process running
2021/12/15 08:26:25 Function: TestKVRebInWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVRebInWithOps deleting in the background"
}
2021/12/15 08:26:27 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 08:26:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 08:26:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 08:26:33 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 08:26:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 08:26:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebInWithOps (459.38s)
=== RUN   TestKVRebOutWithOps
2021/12/15 08:26:43 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/15 08:26:44 Removing node(s): [] from the cluster
2021/12/15 08:26:49 Rebalance progress: 100
2021/12/15 08:26:54 Waiting for app: TestKVRebOutWithOps to get un-deployed
2021/12/15 08:26:54 App: TestKVRebOutWithOps got un-deployed
2021/12/15 08:26:54 No eventing-con process running
2021/12/15 08:26:54 Function: TestKVRebOutWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVRebOutWithOps not found"
 }
}
2021/12/15 08:26:56 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 08:27:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 08:27:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 08:27:02 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 08:27:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 08:27:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 08:27:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 08:27:12 Deploying app: TestKVRebOutWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/15 08:27:14 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVRebOutWithOps' in metakv",
  "warnings": null
 }
}
2021/12/15 08:27:19 Waiting for app: TestKVRebOutWithOps to get deployed
2021/12/15 08:27:24 Waiting for app: TestKVRebOutWithOps to get deployed
2021/12/15 08:27:29 Waiting for app: TestKVRebOutWithOps to get deployed
2021/12/15 08:27:34 Waiting for app: TestKVRebOutWithOps to get deployed
2021/12/15 08:27:34 App: TestKVRebOutWithOps got deployed
2021/12/15 08:27:34 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/15 08:27:34 Starting bucket ops to source bucket
2021/12/15 08:27:39 Rebalance progress: 12.5
2021/12/15 08:27:44 Rebalance progress: 12.5
2021/12/15 08:27:49 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 08:27:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 08:27:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 08:27:49 Waiting for src bucket item count to get to: 200000 curr count: 7978
2021/12/15 08:27:54 Waiting for src bucket item count to get to: 200000 curr count: 23863
2021/12/15 08:27:59 Waiting for src bucket item count to get to: 200000 curr count: 23863
2021/12/15 08:28:04 Waiting for src bucket item count to get to: 200000 curr count: 39975
2021/12/15 08:28:09 Waiting for src bucket item count to get to: 200000 curr count: 39975
2021/12/15 08:28:14 Waiting for src bucket item count to get to: 200000 curr count: 56365
2021/12/15 08:28:19 Waiting for src bucket item count to get to: 200000 curr count: 56365
2021/12/15 08:28:24 Waiting for src bucket item count to get to: 200000 curr count: 56365
2021/12/15 08:28:29 Waiting for src bucket item count to get to: 200000 curr count: 72340
2021/12/15 08:28:34 Waiting for src bucket item count to get to: 200000 curr count: 72340
2021/12/15 08:28:39 Waiting for src bucket item count to get to: 200000 curr count: 87360
2021/12/15 08:28:44 Waiting for src bucket item count to get to: 200000 curr count: 87360
2021/12/15 08:28:49 Waiting for src bucket item count to get to: 200000 curr count: 103370
2021/12/15 08:28:54 Waiting for src bucket item count to get to: 200000 curr count: 103370
2021/12/15 08:28:59 Waiting for src bucket item count to get to: 200000 curr count: 118985
2021/12/15 08:29:04 Waiting for src bucket item count to get to: 200000 curr count: 118985
2021/12/15 08:29:09 Waiting for src bucket item count to get to: 200000 curr count: 135241
2021/12/15 08:29:14 Waiting for src bucket item count to get to: 200000 curr count: 151447
2021/12/15 08:29:19 Waiting for src bucket item count to get to: 200000 curr count: 151447
2021/12/15 08:29:24 Waiting for src bucket item count to get to: 200000 curr count: 167633
2021/12/15 08:29:29 Waiting for src bucket item count to get to: 200000 curr count: 167633
2021/12/15 08:29:34 Waiting for src bucket item count to get to: 200000 curr count: 182385
2021/12/15 08:29:39 Waiting for src bucket item count to get to: 200000 curr count: 182385
2021/12/15 08:29:44 Waiting for src bucket item count to get to: 200000 curr count: 198333
2021/12/15 08:29:49 Waiting for src bucket item count to get to: 200000 curr count: 198333
2021/12/15 08:29:54 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/15 08:29:54 Post SET ops. Expected item count: 200000 got 200000
2021/12/15 08:29:54 Starting bucket ops to source bucket
2021/12/15 08:33:45 Waiting for dst bucket item count to get to: 0 curr count: 18614
2021/12/15 08:33:50 Waiting for dst bucket item count to get to: 0 curr count: 409
2021/12/15 08:33:55 Waiting for dst bucket item count to get to: 0 curr count: 409
2021/12/15 08:33:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 08:33:55 Post DELETE ops. Expected item count: 0 got 0
2021/12/15 08:33:55 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 08:34:00 Waiting for app: TestKVRebOutWithOps to get un-deployed
2021/12/15 08:34:05 Waiting for app: TestKVRebOutWithOps to get un-deployed
2021/12/15 08:34:10 Waiting for app: TestKVRebOutWithOps to get un-deployed
2021/12/15 08:34:15 Waiting for app: TestKVRebOutWithOps to get un-deployed
2021/12/15 08:34:15 App: TestKVRebOutWithOps got un-deployed
2021/12/15 08:34:15 No eventing-con process running
2021/12/15 08:34:15 Function: TestKVRebOutWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVRebOutWithOps deleting in the background"
}
2021/12/15 08:34:16 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 08:34:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 08:34:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 08:34:23 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 08:34:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 08:34:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebOutWithOps (469.81s)
=== RUN   TestKVSwapRebWithOps
2021/12/15 08:34:38 Waiting for app: TestKVSwapRebWithOps to get un-deployed
2021/12/15 08:34:38 App: TestKVSwapRebWithOps got un-deployed
2021/12/15 08:34:38 No eventing-con process running
2021/12/15 08:34:38 Function: TestKVSwapRebWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVSwapRebWithOps not found"
 }
}
2021/12/15 08:34:40 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 08:34:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 08:34:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 08:34:47 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 08:34:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 08:34:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 08:34:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 08:34:57 Deploying app: TestKVSwapRebWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/15 08:34:58 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVSwapRebWithOps' in metakv",
  "warnings": null
 }
}
2021/12/15 08:35:03 Waiting for app: TestKVSwapRebWithOps to get deployed
2021/12/15 08:35:08 Waiting for app: TestKVSwapRebWithOps to get deployed
2021/12/15 08:35:13 Waiting for app: TestKVSwapRebWithOps to get deployed
2021/12/15 08:35:18 Waiting for app: TestKVSwapRebWithOps to get deployed
2021/12/15 08:35:18 App: TestKVSwapRebWithOps got deployed
2021/12/15 08:35:18 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/15 08:35:18 Starting bucket ops to source bucket
2021/12/15 08:35:18 Removing node(s): [] from the cluster
2021/12/15 08:35:23 Rebalance progress: 12.5
2021/12/15 08:35:28 Rebalance progress: 12.5
2021/12/15 08:35:33 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 08:35:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 08:35:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 08:35:33 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/15 08:35:34 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/15 08:35:39 Rebalance progress: 12.5
2021/12/15 08:35:44 Rebalance progress: 12.5
2021/12/15 08:35:49 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 08:35:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 08:35:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 08:35:49 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/15 08:35:54 Rebalance progress: 12.5
2021/12/15 08:35:59 Rebalance progress: 12.5
2021/12/15 08:36:04 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 08:36:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 08:36:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 08:36:04 Waiting for src bucket item count to get to: 200000 curr count: 63217
2021/12/15 08:36:09 Waiting for src bucket item count to get to: 200000 curr count: 63217
2021/12/15 08:36:14 Waiting for src bucket item count to get to: 200000 curr count: 78029
2021/12/15 08:36:19 Waiting for src bucket item count to get to: 200000 curr count: 78029
2021/12/15 08:36:24 Waiting for src bucket item count to get to: 200000 curr count: 93791
2021/12/15 08:36:29 Waiting for src bucket item count to get to: 200000 curr count: 93791
2021/12/15 08:36:34 Waiting for src bucket item count to get to: 200000 curr count: 109721
2021/12/15 08:36:39 Waiting for src bucket item count to get to: 200000 curr count: 109721
2021/12/15 08:36:44 Waiting for src bucket item count to get to: 200000 curr count: 125688
2021/12/15 08:36:49 Waiting for src bucket item count to get to: 200000 curr count: 125688
2021/12/15 08:36:54 Waiting for src bucket item count to get to: 200000 curr count: 125688
2021/12/15 08:36:59 Waiting for src bucket item count to get to: 200000 curr count: 141661
2021/12/15 08:37:04 Waiting for src bucket item count to get to: 200000 curr count: 141661
2021/12/15 08:37:09 Waiting for src bucket item count to get to: 200000 curr count: 157943
2021/12/15 08:37:14 Waiting for src bucket item count to get to: 200000 curr count: 157943
2021/12/15 08:37:19 Waiting for src bucket item count to get to: 200000 curr count: 173030
2021/12/15 08:37:24 Waiting for src bucket item count to get to: 200000 curr count: 173030
2021/12/15 08:37:29 Waiting for src bucket item count to get to: 200000 curr count: 189207
2021/12/15 08:37:34 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/15 08:37:34 Post SET ops. Expected item count: 200000 got 200000
2021/12/15 08:37:34 Starting bucket ops to source bucket
2021/12/15 08:41:24 Waiting for dst bucket item count to get to: 0 curr count: 17592
2021/12/15 08:41:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 08:41:24 Post DELETE ops. Expected item count: 0 got 0
2021/12/15 08:41:24 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 08:41:29 Waiting for app: TestKVSwapRebWithOps to get un-deployed
2021/12/15 08:41:34 Waiting for app: TestKVSwapRebWithOps to get un-deployed
2021/12/15 08:41:39 Waiting for app: TestKVSwapRebWithOps to get un-deployed
2021/12/15 08:41:44 Waiting for app: TestKVSwapRebWithOps to get un-deployed
2021/12/15 08:41:44 App: TestKVSwapRebWithOps got un-deployed
2021/12/15 08:41:44 No eventing-con process running
2021/12/15 08:41:44 Function: TestKVSwapRebWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVSwapRebWithOps deleting in the background"
}
2021/12/15 08:41:46 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 08:41:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 08:41:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 08:41:53 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 08:41:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 08:41:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVSwapRebWithOps (449.44s)
=== RUN   TestEventingRebInWithTimers
2021/12/15 08:42:08 Waiting for app: TestEventingRebInWithTimers to get un-deployed
2021/12/15 08:42:08 App: TestEventingRebInWithTimers got un-deployed
2021/12/15 08:42:08 No eventing-con process running
2021/12/15 08:42:08 Function: TestEventingRebInWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebInWithTimers not found"
 }
}
2021/12/15 08:42:10 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 08:42:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 08:42:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 08:42:16 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 08:42:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 08:42:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 08:42:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 08:42:26 Deploying app: TestEventingRebInWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/15 08:42:27 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebInWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/15 08:42:32 Waiting for app: TestEventingRebInWithTimers to get deployed
2021/12/15 08:42:37 Waiting for app: TestEventingRebInWithTimers to get deployed
2021/12/15 08:42:42 Waiting for app: TestEventingRebInWithTimers to get deployed
2021/12/15 08:42:47 Waiting for app: TestEventingRebInWithTimers to get deployed
2021/12/15 08:42:47 App: TestEventingRebInWithTimers got deployed
2021/12/15 08:42:47 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/15 08:42:47 Starting bucket ops to source bucket
2021/12/15 08:42:48 Removing node(s): [] from the cluster
2021/12/15 08:42:53 Rebalance progress: 12.5
2021/12/15 08:42:58 Rebalance progress: 12.5
2021/12/15 08:43:03 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 08:43:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 08:43:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 08:43:03 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/15 08:43:08 Rebalance progress: 12.5
2021/12/15 08:43:13 Rebalance progress: 12.5
2021/12/15 08:43:18 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 08:43:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 08:43:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 08:43:18 Waiting for src bucket item count to get to: 50000 curr count: 27705
2021/12/15 08:43:23 Waiting for src bucket item count to get to: 50000 curr count: 37779
2021/12/15 08:43:28 Waiting for src bucket item count to get to: 50000 curr count: 37779
2021/12/15 08:43:38 Waiting for dst bucket item count to get to: 50000 curr count: 30674
2021/12/15 08:43:43 Waiting for dst bucket item count to get to: 50000 curr count: 30674
2021/12/15 08:43:48 Waiting for dst bucket item count to get to: 50000 curr count: 45024
2021/12/15 08:43:53 Waiting for dst bucket item count to get to: 50000 curr count: 45024
2021/12/15 08:43:53 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/15 08:43:53 Post SET ops. Expected item count: 50000 got 50000
2021/12/15 08:43:53 Starting bucket ops to source bucket
2021/12/15 08:45:04 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/15 08:45:09 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/15 08:45:14 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/15 08:45:19 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/15 08:45:24 Waiting for dst bucket item count to get to: 0 curr count: 41486
2021/12/15 08:45:29 Waiting for dst bucket item count to get to: 0 curr count: 41486
2021/12/15 08:45:34 Waiting for dst bucket item count to get to: 0 curr count: 32222
2021/12/15 08:45:39 Waiting for dst bucket item count to get to: 0 curr count: 32222
2021/12/15 08:45:44 Waiting for dst bucket item count to get to: 0 curr count: 13319
2021/12/15 08:45:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 08:45:44 Post DELETE ops. Expected item count: 0 got 0
2021/12/15 08:45:44 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 08:45:49 Waiting for app: TestEventingRebInWithTimers to get un-deployed
2021/12/15 08:45:54 Waiting for app: TestEventingRebInWithTimers to get un-deployed
2021/12/15 08:45:59 Waiting for app: TestEventingRebInWithTimers to get un-deployed
2021/12/15 08:46:04 Waiting for app: TestEventingRebInWithTimers to get un-deployed
2021/12/15 08:46:04 App: TestEventingRebInWithTimers got un-deployed
2021/12/15 08:46:04 No eventing-con process running
2021/12/15 08:46:04 Function: TestEventingRebInWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebInWithTimers deleting in the background"
}
2021/12/15 08:46:06 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 08:46:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 08:46:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 08:46:13 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 08:46:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 08:46:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebInWithTimers (260.36s)
=== RUN   TestEventingRebOutWithTimers
2021/12/15 08:46:23 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/15 08:46:23 Removing node(s): [] from the cluster
2021/12/15 08:46:29 Rebalance progress: 100
2021/12/15 08:46:34 Waiting for app: TestEventingRebOutWithTimers to get un-deployed
2021/12/15 08:46:34 App: TestEventingRebOutWithTimers got un-deployed
2021/12/15 08:46:34 No eventing-con process running
2021/12/15 08:46:34 Function: TestEventingRebOutWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebOutWithTimers not found"
 }
}
2021/12/15 08:46:35 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 08:46:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 08:46:40 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 08:46:42 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 08:46:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 08:46:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 08:46:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 08:46:52 Deploying app: TestEventingRebOutWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/15 08:46:53 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebOutWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/15 08:46:58 Waiting for app: TestEventingRebOutWithTimers to get deployed
2021/12/15 08:47:03 Waiting for app: TestEventingRebOutWithTimers to get deployed
2021/12/15 08:47:08 Waiting for app: TestEventingRebOutWithTimers to get deployed
2021/12/15 08:47:13 Waiting for app: TestEventingRebOutWithTimers to get deployed
2021/12/15 08:47:13 App: TestEventingRebOutWithTimers got deployed
2021/12/15 08:47:13 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/15 08:47:13 Starting bucket ops to source bucket
2021/12/15 08:47:18 Rebalance progress: 12.5
2021/12/15 08:47:23 Rebalance progress: 12.5
2021/12/15 08:47:28 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 08:47:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 08:47:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 08:47:28 Waiting for src bucket item count to get to: 50000 curr count: 7941
2021/12/15 08:47:33 Waiting for src bucket item count to get to: 50000 curr count: 21270
2021/12/15 08:47:38 Waiting for src bucket item count to get to: 50000 curr count: 21270
2021/12/15 08:47:43 Waiting for src bucket item count to get to: 50000 curr count: 32766
2021/12/15 08:47:48 Waiting for src bucket item count to get to: 50000 curr count: 32766
2021/12/15 08:47:53 Waiting for src bucket item count to get to: 50000 curr count: 43349
2021/12/15 08:47:58 Waiting for src bucket item count to get to: 50000 curr count: 43349
2021/12/15 08:48:03 Waiting for src bucket item count to get to: 50000 curr count: 43349
2021/12/15 08:48:13 Waiting for dst bucket item count to get to: 50000 curr count: 37710
2021/12/15 08:48:18 Waiting for dst bucket item count to get to: 50000 curr count: 37710
2021/12/15 08:48:23 Waiting for dst bucket item count to get to: 50000 curr count: 48761
2021/12/15 08:48:28 Waiting for dst bucket item count to get to: 50000 curr count: 48761
2021/12/15 08:48:28 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/15 08:48:28 Post SET ops. Expected item count: 50000 got 50000
2021/12/15 08:48:28 Starting bucket ops to source bucket
2021/12/15 08:49:39 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/15 08:49:44 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/15 08:49:49 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/15 08:49:54 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/15 08:49:59 Waiting for dst bucket item count to get to: 0 curr count: 35495
2021/12/15 08:50:04 Waiting for dst bucket item count to get to: 0 curr count: 35495
2021/12/15 08:50:09 Waiting for dst bucket item count to get to: 0 curr count: 24907
2021/12/15 08:50:14 Waiting for dst bucket item count to get to: 0 curr count: 24907
2021/12/15 08:50:19 Waiting for dst bucket item count to get to: 0 curr count: 3460
2021/12/15 08:50:24 Waiting for dst bucket item count to get to: 0 curr count: 3460
2021/12/15 08:50:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 08:50:24 Post DELETE ops. Expected item count: 0 got 0
2021/12/15 08:50:24 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 08:50:29 Waiting for app: TestEventingRebOutWithTimers to get un-deployed
2021/12/15 08:50:34 Waiting for app: TestEventingRebOutWithTimers to get un-deployed
2021/12/15 08:50:39 Waiting for app: TestEventingRebOutWithTimers to get un-deployed
2021/12/15 08:50:44 Waiting for app: TestEventingRebOutWithTimers to get un-deployed
2021/12/15 08:50:44 App: TestEventingRebOutWithTimers got un-deployed
2021/12/15 08:50:44 No eventing-con process running
2021/12/15 08:50:44 Function: TestEventingRebOutWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebOutWithTimers deleting in the background"
}
2021/12/15 08:50:46 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 08:50:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 08:50:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 08:50:53 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 08:50:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 08:50:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebOutWithTimers (279.45s)
=== RUN   TestEventingSwapRebWithTimers
2021/12/15 08:51:08 Waiting for app: TestEventingSwapRebWithTimers to get un-deployed
2021/12/15 08:51:08 App: TestEventingSwapRebWithTimers got un-deployed
2021/12/15 08:51:08 No eventing-con process running
2021/12/15 08:51:08 Function: TestEventingSwapRebWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingSwapRebWithTimers not found"
 }
}
2021/12/15 08:51:09 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 08:51:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 08:51:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 08:51:16 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 08:51:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 08:51:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 08:51:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 08:51:26 Deploying app: TestEventingSwapRebWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/15 08:51:27 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingSwapRebWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/15 08:51:32 Waiting for app: TestEventingSwapRebWithTimers to get deployed
2021/12/15 08:51:37 Waiting for app: TestEventingSwapRebWithTimers to get deployed
2021/12/15 08:51:42 Waiting for app: TestEventingSwapRebWithTimers to get deployed
2021/12/15 08:51:47 Waiting for app: TestEventingSwapRebWithTimers to get deployed
2021/12/15 08:51:47 App: TestEventingSwapRebWithTimers got deployed
2021/12/15 08:51:47 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/15 08:51:47 Starting bucket ops to source bucket
2021/12/15 08:51:48 Removing node(s): [] from the cluster
2021/12/15 08:51:53 Rebalance progress: 12.5
2021/12/15 08:51:58 Rebalance progress: 12.5
2021/12/15 08:52:03 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 08:52:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 08:52:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 08:52:03 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/15 08:52:04 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/15 08:52:09 Rebalance progress: 12.5
2021/12/15 08:52:14 Rebalance progress: 12.5
2021/12/15 08:52:19 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 08:52:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 08:52:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 08:52:19 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/15 08:52:24 Rebalance progress: 12.5
2021/12/15 08:52:29 Rebalance progress: 12.5
2021/12/15 08:52:34 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 08:52:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 08:52:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 08:52:34 Waiting for src bucket item count to get to: 50000 curr count: 49507
2021/12/15 08:52:39 Waiting for src bucket item count to get to: 50000 curr count: 49507
2021/12/15 08:52:49 Waiting for dst bucket item count to get to: 50000 curr count: 43078
2021/12/15 08:52:54 Waiting for dst bucket item count to get to: 50000 curr count: 43078
2021/12/15 08:52:54 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/15 08:52:54 Post SET ops. Expected item count: 50000 got 50000
2021/12/15 08:52:54 Starting bucket ops to source bucket
2021/12/15 08:54:04 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/15 08:54:09 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/15 08:54:14 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/15 08:54:19 Waiting for dst bucket item count to get to: 0 curr count: 42473
2021/12/15 08:54:24 Waiting for dst bucket item count to get to: 0 curr count: 42473
2021/12/15 08:54:29 Waiting for dst bucket item count to get to: 0 curr count: 32986
2021/12/15 08:54:34 Waiting for dst bucket item count to get to: 0 curr count: 32986
2021/12/15 08:54:39 Waiting for dst bucket item count to get to: 0 curr count: 14682
2021/12/15 08:54:44 Waiting for dst bucket item count to get to: 0 curr count: 14682
2021/12/15 08:54:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 08:54:44 Post DELETE ops. Expected item count: 0 got 0
2021/12/15 08:54:45 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 08:54:50 Waiting for app: TestEventingSwapRebWithTimers to get un-deployed
2021/12/15 08:54:55 Waiting for app: TestEventingSwapRebWithTimers to get un-deployed
2021/12/15 08:55:00 Waiting for app: TestEventingSwapRebWithTimers to get un-deployed
2021/12/15 08:55:05 Waiting for app: TestEventingSwapRebWithTimers to get un-deployed
2021/12/15 08:55:05 App: TestEventingSwapRebWithTimers got un-deployed
2021/12/15 08:55:05 No eventing-con process running
2021/12/15 08:55:05 Function: TestEventingSwapRebWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingSwapRebWithTimers deleting in the background"
}
2021/12/15 08:55:07 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 08:55:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 08:55:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 08:55:13 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 08:55:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 08:55:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingSwapRebWithTimers (260.73s)
=== RUN   TestKVRebInWithTimers
2021/12/15 08:55:28 Waiting for app: TestKVRebInWithTimers to get un-deployed
2021/12/15 08:55:28 App: TestKVRebInWithTimers got un-deployed
2021/12/15 08:55:28 No eventing-con process running
2021/12/15 08:55:28 Function: TestKVRebInWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVRebInWithTimers not found"
 }
}
2021/12/15 08:55:30 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 08:55:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 08:55:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 08:55:37 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 08:55:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 08:55:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 08:55:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 08:55:47 Deploying app: TestKVRebInWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/15 08:55:48 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVRebInWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/15 08:55:53 Waiting for app: TestKVRebInWithTimers to get deployed
2021/12/15 08:55:58 Waiting for app: TestKVRebInWithTimers to get deployed
2021/12/15 08:56:03 Waiting for app: TestKVRebInWithTimers to get deployed
2021/12/15 08:56:08 Waiting for app: TestKVRebInWithTimers to get deployed
2021/12/15 08:56:08 App: TestKVRebInWithTimers got deployed
2021/12/15 08:56:08 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/15 08:56:08 Starting bucket ops to source bucket
2021/12/15 08:56:08 Removing node(s): [] from the cluster
2021/12/15 08:56:13 Rebalance progress: 12.5
2021/12/15 08:56:18 Rebalance progress: 12.5
2021/12/15 08:56:23 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 08:56:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 08:56:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 08:56:23 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/15 08:56:29 Rebalance progress: 12.5
2021/12/15 08:56:34 Rebalance progress: 12.5
2021/12/15 08:56:39 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 08:56:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 08:56:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 08:56:39 Waiting for src bucket item count to get to: 50000 curr count: 27981
2021/12/15 08:56:44 Waiting for src bucket item count to get to: 50000 curr count: 37448
2021/12/15 08:56:49 Waiting for src bucket item count to get to: 50000 curr count: 37448
2021/12/15 08:56:54 Waiting for src bucket item count to get to: 50000 curr count: 49989
2021/12/15 08:56:59 Waiting for src bucket item count to get to: 50000 curr count: 49989
2021/12/15 08:57:09 Waiting for dst bucket item count to get to: 50000 curr count: 43263
2021/12/15 08:57:14 Waiting for dst bucket item count to get to: 50000 curr count: 43263
2021/12/15 08:57:19 Waiting for dst bucket item count to get to: 50000 curr count: 49985
2021/12/15 08:57:24 Waiting for dst bucket item count to get to: 50000 curr count: 49985
2021/12/15 08:57:24 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/15 08:57:24 Post SET ops. Expected item count: 50000 got 50000
2021/12/15 08:57:24 Starting bucket ops to source bucket
2021/12/15 08:58:34 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/15 08:58:39 Waiting for dst bucket item count to get to: 0 curr count: 49625
2021/12/15 08:58:44 Waiting for dst bucket item count to get to: 0 curr count: 49625
2021/12/15 08:58:49 Waiting for dst bucket item count to get to: 0 curr count: 46143
2021/12/15 08:58:54 Waiting for dst bucket item count to get to: 0 curr count: 46143
2021/12/15 08:58:59 Waiting for dst bucket item count to get to: 0 curr count: 32310
2021/12/15 08:59:04 Waiting for dst bucket item count to get to: 0 curr count: 32310
2021/12/15 08:59:09 Waiting for dst bucket item count to get to: 0 curr count: 14336
2021/12/15 08:59:14 Waiting for dst bucket item count to get to: 0 curr count: 14336
2021/12/15 08:59:19 Waiting for dst bucket item count to get to: 0 curr count: 3460
2021/12/15 08:59:24 Waiting for dst bucket item count to get to: 0 curr count: 3460
2021/12/15 08:59:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 08:59:24 Post DELETE ops. Expected item count: 0 got 0
2021/12/15 08:59:24 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 08:59:29 Waiting for app: TestKVRebInWithTimers to get un-deployed
2021/12/15 08:59:34 Waiting for app: TestKVRebInWithTimers to get un-deployed
2021/12/15 08:59:39 Waiting for app: TestKVRebInWithTimers to get un-deployed
2021/12/15 08:59:44 Waiting for app: TestKVRebInWithTimers to get un-deployed
2021/12/15 08:59:44 App: TestKVRebInWithTimers got un-deployed
2021/12/15 08:59:44 No eventing-con process running
2021/12/15 08:59:44 Function: TestKVRebInWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVRebInWithTimers deleting in the background"
}
2021/12/15 08:59:46 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 08:59:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 08:59:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 08:59:53 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 08:59:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 08:59:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebInWithTimers (279.66s)
=== RUN   TestKVRebOutWithTimers
2021/12/15 09:00:03 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/15 09:00:03 Removing node(s): [] from the cluster
2021/12/15 09:00:08 Rebalance progress: 100
2021/12/15 09:00:13 Waiting for app: TestKVRebOutWithTimers to get un-deployed
2021/12/15 09:00:13 App: TestKVRebOutWithTimers got un-deployed
2021/12/15 09:00:13 No eventing-con process running
2021/12/15 09:00:13 Function: TestKVRebOutWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVRebOutWithTimers not found"
 }
}
2021/12/15 09:00:15 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:00:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 09:00:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 09:00:22 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:00:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 09:00:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:00:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 09:00:32 Deploying app: TestKVRebOutWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/15 09:00:33 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVRebOutWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/15 09:00:38 Waiting for app: TestKVRebOutWithTimers to get deployed
2021/12/15 09:00:43 Waiting for app: TestKVRebOutWithTimers to get deployed
2021/12/15 09:00:48 Waiting for app: TestKVRebOutWithTimers to get deployed
2021/12/15 09:00:53 Waiting for app: TestKVRebOutWithTimers to get deployed
2021/12/15 09:00:53 App: TestKVRebOutWithTimers got deployed
2021/12/15 09:00:53 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/15 09:00:53 Starting bucket ops to source bucket
2021/12/15 09:00:58 Rebalance progress: 12.5
2021/12/15 09:01:03 Rebalance progress: 12.5
2021/12/15 09:01:08 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:01:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:01:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 09:01:08 Waiting for src bucket item count to get to: 50000 curr count: 7897
2021/12/15 09:01:13 Waiting for src bucket item count to get to: 50000 curr count: 21508
2021/12/15 09:01:18 Waiting for src bucket item count to get to: 50000 curr count: 21508
2021/12/15 09:01:23 Waiting for src bucket item count to get to: 50000 curr count: 32998
2021/12/15 09:01:28 Waiting for src bucket item count to get to: 50000 curr count: 32998
2021/12/15 09:01:33 Waiting for src bucket item count to get to: 50000 curr count: 43189
2021/12/15 09:01:38 Waiting for src bucket item count to get to: 50000 curr count: 43189
2021/12/15 09:01:43 Waiting for src bucket item count to get to: 50000 curr count: 43189
2021/12/15 09:01:53 Waiting for dst bucket item count to get to: 50000 curr count: 35514
2021/12/15 09:01:58 Waiting for dst bucket item count to get to: 50000 curr count: 35514
2021/12/15 09:02:03 Waiting for dst bucket item count to get to: 50000 curr count: 48852
2021/12/15 09:02:08 Waiting for dst bucket item count to get to: 50000 curr count: 48852
2021/12/15 09:02:08 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/15 09:02:08 Post SET ops. Expected item count: 50000 got 50000
2021/12/15 09:02:08 Starting bucket ops to source bucket
2021/12/15 09:03:18 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/15 09:03:23 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/15 09:03:28 Waiting for dst bucket item count to get to: 0 curr count: 49703
2021/12/15 09:03:33 Waiting for dst bucket item count to get to: 0 curr count: 49703
2021/12/15 09:03:38 Waiting for dst bucket item count to get to: 0 curr count: 35442
2021/12/15 09:03:43 Waiting for dst bucket item count to get to: 0 curr count: 35442
2021/12/15 09:03:48 Waiting for dst bucket item count to get to: 0 curr count: 25133
2021/12/15 09:03:53 Waiting for dst bucket item count to get to: 0 curr count: 25133
2021/12/15 09:03:59 Waiting for dst bucket item count to get to: 0 curr count: 3661
2021/12/15 09:04:04 Waiting for dst bucket item count to get to: 0 curr count: 3661
2021/12/15 09:04:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 09:04:04 Post DELETE ops. Expected item count: 0 got 0
2021/12/15 09:04:04 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 09:04:09 Waiting for app: TestKVRebOutWithTimers to get un-deployed
2021/12/15 09:04:14 Waiting for app: TestKVRebOutWithTimers to get un-deployed
2021/12/15 09:04:19 Waiting for app: TestKVRebOutWithTimers to get un-deployed
2021/12/15 09:04:24 Waiting for app: TestKVRebOutWithTimers to get un-deployed
2021/12/15 09:04:24 App: TestKVRebOutWithTimers got un-deployed
2021/12/15 09:04:24 No eventing-con process running
2021/12/15 09:04:24 Function: TestKVRebOutWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVRebOutWithTimers deleting in the background"
}
2021/12/15 09:04:26 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:04:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 09:04:31 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 09:04:32 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:04:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 09:04:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebOutWithTimers (279.35s)
=== RUN   TestKVSwapRebWithTimers
2021/12/15 09:04:47 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/15 09:04:47 App: TestKVSwapRebWithTimers got un-deployed
2021/12/15 09:04:47 No eventing-con process running
2021/12/15 09:04:47 Function: TestKVSwapRebWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVSwapRebWithTimers not found"
 }
}
2021/12/15 09:04:49 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:04:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 09:04:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 09:04:56 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:05:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 09:05:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:05:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 09:05:06 Deploying app: TestKVSwapRebWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/15 09:05:07 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVSwapRebWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/15 09:05:12 Waiting for app: TestKVSwapRebWithTimers to get deployed
2021/12/15 09:05:17 Waiting for app: TestKVSwapRebWithTimers to get deployed
2021/12/15 09:05:22 Waiting for app: TestKVSwapRebWithTimers to get deployed
2021/12/15 09:05:27 Waiting for app: TestKVSwapRebWithTimers to get deployed
2021/12/15 09:05:27 App: TestKVSwapRebWithTimers got deployed
2021/12/15 09:05:27 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/15 09:05:27 Starting bucket ops to source bucket
2021/12/15 09:05:28 Removing node(s): [] from the cluster
2021/12/15 09:05:33 Rebalance progress: 12.5
2021/12/15 09:05:38 Rebalance progress: 12.5
2021/12/15 09:05:43 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:05:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:05:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 09:05:43 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/15 09:05:43 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/15 09:05:49 Rebalance progress: 12.5
2021/12/15 09:05:54 Rebalance progress: 12.5
2021/12/15 09:05:59 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:05:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:05:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 09:05:59 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/15 09:06:04 Rebalance progress: 12.5
2021/12/15 09:06:09 Rebalance progress: 12.5
2021/12/15 09:06:14 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:06:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:06:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 09:06:14 Waiting for src bucket item count to get to: 50000 curr count: 49141
2021/12/15 09:06:19 Waiting for src bucket item count to get to: 50000 curr count: 49141
2021/12/15 09:06:29 Waiting for dst bucket item count to get to: 50000 curr count: 41213
2021/12/15 09:06:34 Waiting for dst bucket item count to get to: 50000 curr count: 41213
2021/12/15 09:06:39 Waiting for dst bucket item count to get to: 50000 curr count: 48638
2021/12/15 09:06:44 Waiting for dst bucket item count to get to: 50000 curr count: 48638
2021/12/15 09:06:44 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/15 09:06:44 Post SET ops. Expected item count: 50000 got 50000
2021/12/15 09:06:44 Starting bucket ops to source bucket
2021/12/15 09:07:55 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/15 09:08:00 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/15 09:08:05 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/15 09:08:10 Waiting for dst bucket item count to get to: 0 curr count: 48543
2021/12/15 09:08:15 Waiting for dst bucket item count to get to: 0 curr count: 48543
2021/12/15 09:08:20 Waiting for dst bucket item count to get to: 0 curr count: 30356
2021/12/15 09:08:25 Waiting for dst bucket item count to get to: 0 curr count: 30356
2021/12/15 09:08:30 Waiting for dst bucket item count to get to: 0 curr count: 17089
2021/12/15 09:08:35 Waiting for dst bucket item count to get to: 0 curr count: 17089
2021/12/15 09:08:40 Waiting for dst bucket item count to get to: 0 curr count: 1432
2021/12/15 09:08:45 Waiting for dst bucket item count to get to: 0 curr count: 1432
2021/12/15 09:08:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 09:08:45 Post DELETE ops. Expected item count: 0 got 0
2021/12/15 09:08:45 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 09:08:50 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/15 09:08:55 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/15 09:09:00 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/15 09:09:05 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/15 09:09:10 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/15 09:09:10 App: TestKVSwapRebWithTimers got un-deployed
2021/12/15 09:09:10 No eventing-con process running
2021/12/15 09:09:10 Function: TestKVSwapRebWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVSwapRebWithTimers deleting in the background"
}
2021/12/15 09:09:12 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:09:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 09:09:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 09:09:19 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:09:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 09:09:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVSwapRebWithTimers (286.27s)
=== RUN   TestDuplicateTimerCreationEventingRebIn
2021/12/15 09:09:34 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/15 09:09:34 App: TestDuplicateTimerCreationEventingRebIn got un-deployed
2021/12/15 09:09:34 No eventing-con process running
2021/12/15 09:09:34 Function: TestDuplicateTimerCreationEventingRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationEventingRebIn not found"
 }
}
2021/12/15 09:09:35 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:09:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 09:09:40 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 09:09:42 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:09:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 09:09:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:09:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 09:09:52 Deploying app: TestDuplicateTimerCreationEventingRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/15 09:09:53 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationEventingRebIn' in metakv",
  "warnings": null
 }
}
2021/12/15 09:09:58 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get deployed
2021/12/15 09:10:03 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get deployed
2021/12/15 09:10:08 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get deployed
2021/12/15 09:10:13 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get deployed
2021/12/15 09:10:13 App: TestDuplicateTimerCreationEventingRebIn got deployed
2021/12/15 09:10:13 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/15 09:10:13 Starting bucket ops to source bucket
2021/12/15 09:10:14 Removing node(s): [] from the cluster
2021/12/15 09:10:19 Rebalance progress: 12.5
2021/12/15 09:10:24 Rebalance progress: 12.5
2021/12/15 09:10:29 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:10:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:10:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 09:10:29 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/15 09:10:34 Rebalance progress: 12.5
2021/12/15 09:10:39 Rebalance progress: 12.5
2021/12/15 09:10:44 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:10:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:10:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 09:10:44 Waiting for src bucket item count to get to: 50000 curr count: 34319
2021/12/15 09:10:49 Waiting for src bucket item count to get to: 50000 curr count: 34319
2021/12/15 09:10:54 Waiting for src bucket item count to get to: 50000 curr count: 44324
2021/12/15 09:10:59 Waiting for src bucket item count to get to: 50000 curr count: 44324
2021/12/15 09:11:09 Waiting for dst bucket item count to get to: 50000 curr count: 28825
2021/12/15 09:11:14 Waiting for dst bucket item count to get to: 50000 curr count: 28825
2021/12/15 09:11:19 Waiting for dst bucket item count to get to: 50000 curr count: 38969
2021/12/15 09:11:24 Waiting for dst bucket item count to get to: 50000 curr count: 38969
2021/12/15 09:11:29 Waiting for dst bucket item count to get to: 50000 curr count: 49987
2021/12/15 09:11:34 Waiting for dst bucket item count to get to: 50000 curr count: 49987
2021/12/15 09:11:34 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/15 09:11:34 Post SET ops. Expected item count: 50000 got 50000
2021/12/15 09:11:34 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 09:11:39 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/15 09:11:44 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/15 09:11:49 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/15 09:11:54 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/15 09:11:59 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/15 09:12:04 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/15 09:12:04 App: TestDuplicateTimerCreationEventingRebIn got un-deployed
2021/12/15 09:12:04 No eventing-con process running
2021/12/15 09:12:04 Function: TestDuplicateTimerCreationEventingRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationEventingRebIn deleting in the background"
}
2021/12/15 09:12:06 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:12:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 09:12:16 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/15 09:12:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 09:12:18 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:12:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 09:12:28 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/15 09:12:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationEventingRebIn (184.44s)
=== RUN   TestDuplicateTimerCreationEventingRebOut
2021/12/15 09:12:33 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/15 09:12:33 Removing node(s): [] from the cluster
2021/12/15 09:12:38 Rebalance progress: 100
2021/12/15 09:12:43 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/15 09:12:43 App: TestDuplicateTimerCreationEventingRebOut got un-deployed
2021/12/15 09:12:43 No eventing-con process running
2021/12/15 09:12:43 Function: TestDuplicateTimerCreationEventingRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationEventingRebOut not found"
 }
}
2021/12/15 09:12:45 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:12:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 09:12:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 09:12:52 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:12:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 09:12:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:13:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 09:13:02 Deploying app: TestDuplicateTimerCreationEventingRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/15 09:13:03 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationEventingRebOut' in metakv",
  "warnings": null
 }
}
2021/12/15 09:13:08 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get deployed
2021/12/15 09:13:13 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get deployed
2021/12/15 09:13:18 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get deployed
2021/12/15 09:13:23 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get deployed
2021/12/15 09:13:23 App: TestDuplicateTimerCreationEventingRebOut got deployed
2021/12/15 09:13:23 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/15 09:13:23 Starting bucket ops to source bucket
2021/12/15 09:13:28 Rebalance progress: 12.5
2021/12/15 09:13:33 Rebalance progress: 12.5
2021/12/15 09:13:38 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:13:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:13:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 09:13:38 Waiting for src bucket item count to get to: 50000 curr count: 7816
2021/12/15 09:13:43 Waiting for src bucket item count to get to: 50000 curr count: 21514
2021/12/15 09:13:48 Waiting for src bucket item count to get to: 50000 curr count: 21514
2021/12/15 09:13:53 Waiting for src bucket item count to get to: 50000 curr count: 35487
2021/12/15 09:13:58 Waiting for src bucket item count to get to: 50000 curr count: 35487
2021/12/15 09:14:03 Waiting for src bucket item count to get to: 50000 curr count: 35487
2021/12/15 09:14:08 Waiting for src bucket item count to get to: 50000 curr count: 45431
2021/12/15 09:14:13 Waiting for src bucket item count to get to: 50000 curr count: 45431
2021/12/15 09:14:23 Waiting for dst bucket item count to get to: 50000 curr count: 28671
2021/12/15 09:14:28 Waiting for dst bucket item count to get to: 50000 curr count: 28671
2021/12/15 09:14:33 Waiting for dst bucket item count to get to: 50000 curr count: 40910
2021/12/15 09:14:38 Waiting for dst bucket item count to get to: 50000 curr count: 40910
2021/12/15 09:14:38 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/15 09:14:38 Post SET ops. Expected item count: 50000 got 50000
2021/12/15 09:14:38 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 09:14:43 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/15 09:14:48 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/15 09:14:54 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/15 09:14:59 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/15 09:15:04 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/15 09:15:09 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/15 09:15:09 App: TestDuplicateTimerCreationEventingRebOut got un-deployed
2021/12/15 09:15:09 No eventing-con process running
2021/12/15 09:15:09 Function: TestDuplicateTimerCreationEventingRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationEventingRebOut deleting in the background"
}
2021/12/15 09:15:10 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:15:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 09:15:20 Waiting for dst bucket item count to get to: 0 curr count: 24341
2021/12/15 09:15:25 Waiting for dst bucket item count to get to: 0 curr count: 24341
2021/12/15 09:15:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 09:15:27 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:15:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 09:15:37 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/15 09:15:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationEventingRebOut (189.32s)
=== RUN   TestDuplicateTimerCreationEventingSwapReb
2021/12/15 09:15:47 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/15 09:15:47 App: TestDuplicateTimerCreationEventingSwapReb got un-deployed
2021/12/15 09:15:47 No eventing-con process running
2021/12/15 09:15:47 Function: TestDuplicateTimerCreationEventingSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationEventingSwapReb not found"
 }
}
2021/12/15 09:15:49 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:15:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 09:15:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 09:15:56 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:16:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 09:16:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:16:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 09:16:06 Deploying app: TestDuplicateTimerCreationEventingSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/15 09:16:07 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationEventingSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/15 09:16:12 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get deployed
2021/12/15 09:16:17 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get deployed
2021/12/15 09:16:22 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get deployed
2021/12/15 09:16:27 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get deployed
2021/12/15 09:16:27 App: TestDuplicateTimerCreationEventingSwapReb got deployed
2021/12/15 09:16:27 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/15 09:16:27 Starting bucket ops to source bucket
2021/12/15 09:16:28 Removing node(s): [] from the cluster
2021/12/15 09:16:33 Rebalance progress: 12.5
2021/12/15 09:16:38 Rebalance progress: 12.5
2021/12/15 09:16:43 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:16:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:16:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 09:16:43 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/15 09:16:43 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/15 09:16:48 Rebalance progress: 12.5
2021/12/15 09:16:53 Rebalance progress: 12.5
2021/12/15 09:16:58 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:16:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:16:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 09:16:58 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/15 09:17:03 Rebalance progress: 12.5
2021/12/15 09:17:08 Rebalance progress: 12.5
2021/12/15 09:17:13 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:17:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:17:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 09:17:13 Waiting for src bucket item count to get to: 50000 curr count: 47677
2021/12/15 09:17:18 Waiting for src bucket item count to get to: 50000 curr count: 47677
2021/12/15 09:17:28 Waiting for dst bucket item count to get to: 50000 curr count: 38620
2021/12/15 09:17:33 Waiting for dst bucket item count to get to: 50000 curr count: 38620
2021/12/15 09:17:38 Waiting for dst bucket item count to get to: 50000 curr count: 45753
2021/12/15 09:17:43 Waiting for dst bucket item count to get to: 50000 curr count: 45753
2021/12/15 09:17:43 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/15 09:17:43 Post SET ops. Expected item count: 50000 got 50000
2021/12/15 09:17:43 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 09:17:48 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/15 09:17:53 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/15 09:17:58 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/15 09:18:03 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/15 09:18:08 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/15 09:18:14 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/15 09:18:14 App: TestDuplicateTimerCreationEventingSwapReb got un-deployed
2021/12/15 09:18:14 No eventing-con process running
2021/12/15 09:18:14 Function: TestDuplicateTimerCreationEventingSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationEventingSwapReb deleting in the background"
}
2021/12/15 09:18:15 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:18:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 09:18:25 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/15 09:18:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 09:18:27 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:18:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 09:18:37 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/15 09:18:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationEventingSwapReb (179.81s)
=== RUN   TestDuplicateTimerCreationKVRebIn
2021/12/15 09:18:47 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/15 09:18:47 App: TestDuplicateTimerCreationKVRebIn got un-deployed
2021/12/15 09:18:47 No eventing-con process running
2021/12/15 09:18:47 Function: TestDuplicateTimerCreationKVRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationKVRebIn not found"
 }
}
2021/12/15 09:18:49 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:18:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 09:18:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 09:18:56 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:19:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 09:19:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:19:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 09:19:06 Deploying app: TestDuplicateTimerCreationKVRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/15 09:19:07 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationKVRebIn' in metakv",
  "warnings": null
 }
}
2021/12/15 09:19:12 Waiting for app: TestDuplicateTimerCreationKVRebIn to get deployed
2021/12/15 09:19:17 Waiting for app: TestDuplicateTimerCreationKVRebIn to get deployed
2021/12/15 09:19:22 Waiting for app: TestDuplicateTimerCreationKVRebIn to get deployed
2021/12/15 09:19:27 Waiting for app: TestDuplicateTimerCreationKVRebIn to get deployed
2021/12/15 09:19:27 App: TestDuplicateTimerCreationKVRebIn got deployed
2021/12/15 09:19:27 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/15 09:19:27 Starting bucket ops to source bucket
2021/12/15 09:19:28 Removing node(s): [] from the cluster
2021/12/15 09:19:33 Rebalance progress: 12.5
2021/12/15 09:19:38 Rebalance progress: 12.5
2021/12/15 09:19:43 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:19:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:19:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 09:19:43 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/15 09:19:48 Rebalance progress: 12.5
2021/12/15 09:19:53 Rebalance progress: 12.5
2021/12/15 09:19:58 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:19:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:19:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 09:19:58 Waiting for src bucket item count to get to: 50000 curr count: 28611
2021/12/15 09:20:03 Waiting for src bucket item count to get to: 50000 curr count: 39786
2021/12/15 09:20:08 Waiting for src bucket item count to get to: 50000 curr count: 39786
2021/12/15 09:20:18 Waiting for dst bucket item count to get to: 50000 curr count: 23001
2021/12/15 09:20:23 Waiting for dst bucket item count to get to: 50000 curr count: 23001
2021/12/15 09:20:28 Waiting for dst bucket item count to get to: 50000 curr count: 35465
2021/12/15 09:20:33 Waiting for dst bucket item count to get to: 50000 curr count: 35465
2021/12/15 09:20:38 Waiting for dst bucket item count to get to: 50000 curr count: 42936
2021/12/15 09:20:43 Waiting for dst bucket item count to get to: 50000 curr count: 42936
2021/12/15 09:20:43 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/15 09:20:43 Post SET ops. Expected item count: 50000 got 50000
2021/12/15 09:20:43 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 09:20:48 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/15 09:20:53 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/15 09:20:58 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/15 09:21:03 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/15 09:21:08 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/15 09:21:13 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/15 09:21:13 App: TestDuplicateTimerCreationKVRebIn got un-deployed
2021/12/15 09:21:13 No eventing-con process running
2021/12/15 09:21:13 Function: TestDuplicateTimerCreationKVRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationKVRebIn deleting in the background"
}
2021/12/15 09:21:15 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:21:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 09:21:25 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/15 09:21:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 09:21:27 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:21:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 09:21:37 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/15 09:21:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationKVRebIn (179.97s)
=== RUN   TestDuplicateTimerCreationKVRebOut
2021/12/15 09:21:42 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/15 09:21:42 Removing node(s): [] from the cluster
2021/12/15 09:21:47 Rebalance progress: 100
2021/12/15 09:21:52 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/15 09:21:52 App: TestDuplicateTimerCreationKVRebOut got un-deployed
2021/12/15 09:21:52 No eventing-con process running
2021/12/15 09:21:52 Function: TestDuplicateTimerCreationKVRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationKVRebOut not found"
 }
}
2021/12/15 09:21:54 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:21:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 09:21:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 09:22:01 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:22:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 09:22:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:22:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 09:22:11 Deploying app: TestDuplicateTimerCreationKVRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/15 09:22:12 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationKVRebOut' in metakv",
  "warnings": null
 }
}
2021/12/15 09:22:17 Waiting for app: TestDuplicateTimerCreationKVRebOut to get deployed
2021/12/15 09:22:22 Waiting for app: TestDuplicateTimerCreationKVRebOut to get deployed
2021/12/15 09:22:27 Waiting for app: TestDuplicateTimerCreationKVRebOut to get deployed
2021/12/15 09:22:32 Waiting for app: TestDuplicateTimerCreationKVRebOut to get deployed
2021/12/15 09:22:32 App: TestDuplicateTimerCreationKVRebOut got deployed
2021/12/15 09:22:32 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/15 09:22:32 Starting bucket ops to source bucket
2021/12/15 09:22:37 Rebalance progress: 12.5
2021/12/15 09:22:42 Rebalance progress: 12.5
2021/12/15 09:22:47 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:22:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:22:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 09:22:47 Waiting for src bucket item count to get to: 50000 curr count: 8457
2021/12/15 09:22:52 Waiting for src bucket item count to get to: 50000 curr count: 21919
2021/12/15 09:22:57 Waiting for src bucket item count to get to: 50000 curr count: 21919
2021/12/15 09:23:02 Waiting for src bucket item count to get to: 50000 curr count: 34420
2021/12/15 09:23:07 Waiting for src bucket item count to get to: 50000 curr count: 34420
2021/12/15 09:23:12 Waiting for src bucket item count to get to: 50000 curr count: 34420
2021/12/15 09:23:17 Waiting for src bucket item count to get to: 50000 curr count: 43750
2021/12/15 09:23:22 Waiting for src bucket item count to get to: 50000 curr count: 43750
2021/12/15 09:23:32 Waiting for dst bucket item count to get to: 50000 curr count: 31933
2021/12/15 09:23:37 Waiting for dst bucket item count to get to: 50000 curr count: 31933
2021/12/15 09:23:42 Waiting for dst bucket item count to get to: 50000 curr count: 46141
2021/12/15 09:23:47 Waiting for dst bucket item count to get to: 50000 curr count: 46141
2021/12/15 09:23:47 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/15 09:23:47 Post SET ops. Expected item count: 50000 got 50000
2021/12/15 09:23:47 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 09:23:52 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/15 09:23:57 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/15 09:24:03 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/15 09:24:08 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/15 09:24:13 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/15 09:24:18 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/15 09:24:18 App: TestDuplicateTimerCreationKVRebOut got un-deployed
2021/12/15 09:24:18 No eventing-con process running
2021/12/15 09:24:18 Function: TestDuplicateTimerCreationKVRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationKVRebOut deleting in the background"
}
2021/12/15 09:24:19 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:24:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 09:24:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 09:24:26 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:24:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 09:24:36 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/15 09:24:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationKVRebOut (179.17s)
=== RUN   TestDuplicateTimerCreationKVSwapReb
2021/12/15 09:24:46 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/15 09:24:46 App: TestDuplicateTimerCreationKVSwapReb got un-deployed
2021/12/15 09:24:46 No eventing-con process running
2021/12/15 09:24:46 Function: TestDuplicateTimerCreationKVSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationKVSwapReb not found"
 }
}
2021/12/15 09:24:48 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:24:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 09:24:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 09:24:55 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:25:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 09:25:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:25:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 09:25:05 Deploying app: TestDuplicateTimerCreationKVSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/15 09:25:06 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationKVSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/15 09:25:11 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get deployed
2021/12/15 09:25:16 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get deployed
2021/12/15 09:25:21 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get deployed
2021/12/15 09:25:26 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get deployed
2021/12/15 09:25:26 App: TestDuplicateTimerCreationKVSwapReb got deployed
2021/12/15 09:25:26 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/15 09:25:26 Starting bucket ops to source bucket
2021/12/15 09:25:27 Removing node(s): [] from the cluster
2021/12/15 09:25:32 Rebalance progress: 12.5
2021/12/15 09:25:37 Rebalance progress: 12.5
2021/12/15 09:25:42 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:25:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:25:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 09:25:42 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/15 09:25:42 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/15 09:25:47 Rebalance progress: 12.5
2021/12/15 09:25:52 Rebalance progress: 12.5
2021/12/15 09:25:57 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:25:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:25:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 09:25:57 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/15 09:26:02 Rebalance progress: 12.5
2021/12/15 09:26:07 Rebalance progress: 12.5
2021/12/15 09:26:12 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:26:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:26:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 09:26:12 Waiting for src bucket item count to get to: 50000 curr count: 47676
2021/12/15 09:26:17 Waiting for src bucket item count to get to: 50000 curr count: 47676
2021/12/15 09:26:27 Waiting for dst bucket item count to get to: 50000 curr count: 38103
2021/12/15 09:26:32 Waiting for dst bucket item count to get to: 50000 curr count: 38103
2021/12/15 09:26:37 Waiting for dst bucket item count to get to: 50000 curr count: 48879
2021/12/15 09:26:42 Waiting for dst bucket item count to get to: 50000 curr count: 48879
2021/12/15 09:26:42 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/15 09:26:42 Post SET ops. Expected item count: 50000 got 50000
2021/12/15 09:26:42 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 09:26:47 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/15 09:26:52 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/15 09:26:57 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/15 09:27:02 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/15 09:27:08 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/15 09:27:13 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/15 09:27:13 App: TestDuplicateTimerCreationKVSwapReb got un-deployed
2021/12/15 09:27:13 No eventing-con process running
2021/12/15 09:27:13 Function: TestDuplicateTimerCreationKVSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationKVSwapReb deleting in the background"
}
2021/12/15 09:27:14 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:27:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 09:27:24 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/15 09:27:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 09:27:26 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:27:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 09:27:36 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/15 09:27:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationKVSwapReb (179.86s)
=== RUN   TestDuplicateTimerFiringEventingRebIn
2021/12/15 09:27:46 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/15 09:27:46 App: TestDuplicateTimerFiringEventingRebIn got un-deployed
2021/12/15 09:27:46 No eventing-con process running
2021/12/15 09:27:46 Function: TestDuplicateTimerFiringEventingRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringEventingRebIn not found"
 }
}
2021/12/15 09:27:48 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:27:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 09:27:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 09:27:55 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:28:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 09:28:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:28:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 09:28:05 Deploying app: TestDuplicateTimerFiringEventingRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/15 09:28:06 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringEventingRebIn' in metakv",
  "warnings": null
 }
}
2021/12/15 09:28:11 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get deployed
2021/12/15 09:28:16 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get deployed
2021/12/15 09:28:21 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get deployed
2021/12/15 09:28:26 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get deployed
2021/12/15 09:28:26 App: TestDuplicateTimerFiringEventingRebIn got deployed
2021/12/15 09:28:26 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/15 09:28:26 Starting bucket ops to source bucket
2021/12/15 09:28:26 Removing node(s): [] from the cluster
2021/12/15 09:28:31 Rebalance progress: 12.5
2021/12/15 09:28:36 Rebalance progress: 12.5
2021/12/15 09:28:41 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:28:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:28:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 09:28:42 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/15 09:28:47 Rebalance progress: 12.5
2021/12/15 09:28:52 Rebalance progress: 12.5
2021/12/15 09:28:57 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:28:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:28:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 09:28:57 Waiting for src bucket item count to get to: 50000 curr count: 28328
2021/12/15 09:29:02 Waiting for src bucket item count to get to: 50000 curr count: 39481
2021/12/15 09:29:07 Waiting for src bucket item count to get to: 50000 curr count: 39481
2021/12/15 09:29:17 Waiting for dst bucket item count to get to: 50000 curr count: 33326
2021/12/15 09:29:22 Waiting for dst bucket item count to get to: 50000 curr count: 33326
2021/12/15 09:29:27 Waiting for dst bucket item count to get to: 50000 curr count: 41825
2021/12/15 09:29:32 Waiting for dst bucket item count to get to: 50000 curr count: 41825
2021/12/15 09:29:32 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/15 09:29:32 Post SET ops. Expected item count: 50000 got 50000
2021/12/15 09:29:32 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 09:29:37 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/15 09:29:42 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/15 09:29:47 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/15 09:29:52 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/15 09:29:57 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/15 09:30:02 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/15 09:30:02 App: TestDuplicateTimerFiringEventingRebIn got un-deployed
2021/12/15 09:30:02 No eventing-con process running
2021/12/15 09:30:02 Function: TestDuplicateTimerFiringEventingRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringEventingRebIn deleting in the background"
}
2021/12/15 09:30:04 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:30:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 09:30:14 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/15 09:30:19 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/15 09:30:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 09:30:21 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:30:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 09:30:31 Waiting for dst bucket item count to get to: 0 curr count: 32890
2021/12/15 09:30:36 Waiting for dst bucket item count to get to: 0 curr count: 32890
2021/12/15 09:30:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringEventingRebIn (179.49s)
=== RUN   TestDuplicateTimerFiringEventingRebOut
2021/12/15 09:30:41 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/15 09:30:41 Removing node(s): [] from the cluster
2021/12/15 09:30:46 Rebalance progress: 100
2021/12/15 09:30:51 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/15 09:30:51 App: TestDuplicateTimerFiringEventingRebOut got un-deployed
2021/12/15 09:30:51 No eventing-con process running
2021/12/15 09:30:51 Function: TestDuplicateTimerFiringEventingRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringEventingRebOut not found"
 }
}
2021/12/15 09:30:53 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:30:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 09:30:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 09:31:00 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:31:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 09:31:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:31:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 09:31:10 Deploying app: TestDuplicateTimerFiringEventingRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/15 09:31:11 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringEventingRebOut' in metakv",
  "warnings": null
 }
}
2021/12/15 09:31:16 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get deployed
2021/12/15 09:31:21 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get deployed
2021/12/15 09:31:26 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get deployed
2021/12/15 09:31:31 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get deployed
2021/12/15 09:31:31 App: TestDuplicateTimerFiringEventingRebOut got deployed
2021/12/15 09:31:31 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/15 09:31:31 Starting bucket ops to source bucket
2021/12/15 09:31:36 Rebalance progress: 12.5
2021/12/15 09:31:41 Rebalance progress: 12.5
2021/12/15 09:31:46 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:31:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:31:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 09:31:46 Waiting for src bucket item count to get to: 50000 curr count: 11180
2021/12/15 09:31:51 Waiting for src bucket item count to get to: 50000 curr count: 23411
2021/12/15 09:31:56 Waiting for src bucket item count to get to: 50000 curr count: 23411
2021/12/15 09:32:01 Waiting for src bucket item count to get to: 50000 curr count: 33854
2021/12/15 09:32:06 Waiting for src bucket item count to get to: 50000 curr count: 33854
2021/12/15 09:32:11 Waiting for src bucket item count to get to: 50000 curr count: 33854
2021/12/15 09:32:16 Waiting for src bucket item count to get to: 50000 curr count: 44768
2021/12/15 09:32:21 Waiting for src bucket item count to get to: 50000 curr count: 44768
2021/12/15 09:32:31 Waiting for dst bucket item count to get to: 50000 curr count: 40791
2021/12/15 09:32:36 Waiting for dst bucket item count to get to: 50000 curr count: 40791
2021/12/15 09:32:36 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/15 09:32:36 Post SET ops. Expected item count: 50000 got 50000
2021/12/15 09:32:36 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 09:32:41 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/15 09:32:46 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/15 09:32:51 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/15 09:32:56 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/15 09:33:01 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/15 09:33:06 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/15 09:33:06 App: TestDuplicateTimerFiringEventingRebOut got un-deployed
2021/12/15 09:33:06 No eventing-con process running
2021/12/15 09:33:06 Function: TestDuplicateTimerFiringEventingRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringEventingRebOut deleting in the background"
}
2021/12/15 09:33:08 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:33:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 09:33:18 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/15 09:33:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 09:33:20 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:33:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 09:33:30 Waiting for dst bucket item count to get to: 0 curr count: 4496
2021/12/15 09:33:35 Waiting for dst bucket item count to get to: 0 curr count: 4496
2021/12/15 09:33:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringEventingRebOut (179.38s)
=== RUN   TestDuplicateTimerFiringEventingSwapReb
2021/12/15 09:33:45 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/15 09:33:45 App: TestDuplicateTimerFiringEventingSwapReb got un-deployed
2021/12/15 09:33:45 No eventing-con process running
2021/12/15 09:33:45 Function: TestDuplicateTimerFiringEventingSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringEventingSwapReb not found"
 }
}
2021/12/15 09:33:47 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:33:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 09:33:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 09:33:54 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:33:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 09:33:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:34:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 09:34:04 Deploying app: TestDuplicateTimerFiringEventingSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/15 09:34:05 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringEventingSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/15 09:34:10 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get deployed
2021/12/15 09:34:15 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get deployed
2021/12/15 09:34:20 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get deployed
2021/12/15 09:34:25 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get deployed
2021/12/15 09:34:25 App: TestDuplicateTimerFiringEventingSwapReb got deployed
2021/12/15 09:34:25 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/15 09:34:25 Starting bucket ops to source bucket
2021/12/15 09:34:25 Removing node(s): [] from the cluster
2021/12/15 09:34:30 Rebalance progress: 12.5
2021/12/15 09:34:35 Rebalance progress: 12.5
2021/12/15 09:34:40 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:34:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:34:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 09:34:40 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/15 09:34:42 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/15 09:34:47 Rebalance progress: 12.5
2021/12/15 09:34:52 Rebalance progress: 12.5
2021/12/15 09:34:57 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:34:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:34:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 09:34:57 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/15 09:35:02 Rebalance progress: 12.5
2021/12/15 09:35:07 Rebalance progress: 12.5
2021/12/15 09:35:12 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:35:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:35:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 09:35:12 Waiting for src bucket item count to get to: 50000 curr count: 49753
2021/12/15 09:35:17 Waiting for src bucket item count to get to: 50000 curr count: 49753
2021/12/15 09:35:27 Waiting for dst bucket item count to get to: 50000 curr count: 47058
2021/12/15 09:35:32 Waiting for dst bucket item count to get to: 50000 curr count: 47058
2021/12/15 09:35:32 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/15 09:35:32 Post SET ops. Expected item count: 50000 got 50000
2021/12/15 09:35:32 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 09:35:37 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/15 09:35:42 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/15 09:35:47 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/15 09:35:52 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/15 09:35:57 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/15 09:36:02 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/15 09:36:02 App: TestDuplicateTimerFiringEventingSwapReb got un-deployed
2021/12/15 09:36:02 No eventing-con process running
2021/12/15 09:36:02 Function: TestDuplicateTimerFiringEventingSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringEventingSwapReb deleting in the background"
}
2021/12/15 09:36:04 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:36:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 09:36:14 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/15 09:36:19 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/15 09:36:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 09:36:21 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:36:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 09:36:31 Waiting for dst bucket item count to get to: 0 curr count: 37608
2021/12/15 09:36:36 Waiting for dst bucket item count to get to: 0 curr count: 37608
2021/12/15 09:36:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringEventingSwapReb (180.74s)
=== RUN   TestDuplicateTimerFiringKVRebIn
2021/12/15 09:36:46 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/15 09:36:46 App: TestDuplicateTimerFiringKVRebIn got un-deployed
2021/12/15 09:36:46 No eventing-con process running
2021/12/15 09:36:46 Function: TestDuplicateTimerFiringKVRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringKVRebIn not found"
 }
}
2021/12/15 09:36:48 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:36:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 09:36:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 09:36:54 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:36:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 09:36:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:37:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 09:37:04 Deploying app: TestDuplicateTimerFiringKVRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/15 09:37:05 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringKVRebIn' in metakv",
  "warnings": null
 }
}
2021/12/15 09:37:11 Waiting for app: TestDuplicateTimerFiringKVRebIn to get deployed
2021/12/15 09:37:16 Waiting for app: TestDuplicateTimerFiringKVRebIn to get deployed
2021/12/15 09:37:21 Waiting for app: TestDuplicateTimerFiringKVRebIn to get deployed
2021/12/15 09:37:26 Waiting for app: TestDuplicateTimerFiringKVRebIn to get deployed
2021/12/15 09:37:26 App: TestDuplicateTimerFiringKVRebIn got deployed
2021/12/15 09:37:26 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/15 09:37:26 Starting bucket ops to source bucket
2021/12/15 09:37:26 Removing node(s): [] from the cluster
2021/12/15 09:37:31 Rebalance progress: 12.5
2021/12/15 09:37:36 Rebalance progress: 12.5
2021/12/15 09:37:41 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:37:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:37:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 09:37:41 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/15 09:37:46 Rebalance progress: 12.5
2021/12/15 09:37:51 Rebalance progress: 12.5
2021/12/15 09:37:56 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:37:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:37:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 09:37:57 Waiting for src bucket item count to get to: 50000 curr count: 29271
2021/12/15 09:38:02 Waiting for src bucket item count to get to: 50000 curr count: 40263
2021/12/15 09:38:07 Waiting for src bucket item count to get to: 50000 curr count: 40263
2021/12/15 09:38:17 Waiting for dst bucket item count to get to: 50000 curr count: 34112
2021/12/15 09:38:22 Waiting for dst bucket item count to get to: 50000 curr count: 34112
2021/12/15 09:38:27 Waiting for dst bucket item count to get to: 50000 curr count: 40972
2021/12/15 09:38:32 Waiting for dst bucket item count to get to: 50000 curr count: 40972
2021/12/15 09:38:32 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/15 09:38:32 Post SET ops. Expected item count: 50000 got 50000
2021/12/15 09:38:32 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 09:38:37 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/15 09:38:42 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/15 09:38:47 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/15 09:38:52 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/15 09:38:57 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/15 09:39:02 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/15 09:39:02 App: TestDuplicateTimerFiringKVRebIn got un-deployed
2021/12/15 09:39:02 No eventing-con process running
2021/12/15 09:39:02 Function: TestDuplicateTimerFiringKVRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringKVRebIn deleting in the background"
}
2021/12/15 09:39:04 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:39:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 09:39:14 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/15 09:39:19 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/15 09:39:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 09:39:20 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:39:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 09:39:30 Waiting for dst bucket item count to get to: 0 curr count: 27517
2021/12/15 09:39:35 Waiting for dst bucket item count to get to: 0 curr count: 27517
2021/12/15 09:39:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringKVRebIn (179.73s)
=== RUN   TestDuplicateTimerFiringKVRebOut
2021/12/15 09:39:40 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/15 09:39:41 Removing node(s): [] from the cluster
2021/12/15 09:39:46 Rebalance progress: 100
2021/12/15 09:39:51 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/15 09:39:51 App: TestDuplicateTimerFiringKVRebOut got un-deployed
2021/12/15 09:39:51 No eventing-con process running
2021/12/15 09:39:51 Function: TestDuplicateTimerFiringKVRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringKVRebOut not found"
 }
}
2021/12/15 09:39:53 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:39:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 09:39:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 09:40:00 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:40:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 09:40:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:40:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 09:40:10 Deploying app: TestDuplicateTimerFiringKVRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/15 09:40:11 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringKVRebOut' in metakv",
  "warnings": null
 }
}
2021/12/15 09:40:16 Waiting for app: TestDuplicateTimerFiringKVRebOut to get deployed
2021/12/15 09:40:21 Waiting for app: TestDuplicateTimerFiringKVRebOut to get deployed
2021/12/15 09:40:26 Waiting for app: TestDuplicateTimerFiringKVRebOut to get deployed
2021/12/15 09:40:31 Waiting for app: TestDuplicateTimerFiringKVRebOut to get deployed
2021/12/15 09:40:31 App: TestDuplicateTimerFiringKVRebOut got deployed
2021/12/15 09:40:31 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/15 09:40:31 Starting bucket ops to source bucket
2021/12/15 09:40:36 Rebalance progress: 12.5
2021/12/15 09:40:41 Rebalance progress: 12.5
2021/12/15 09:40:46 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:40:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:40:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 09:40:46 Waiting for src bucket item count to get to: 50000 curr count: 11083
2021/12/15 09:40:51 Waiting for src bucket item count to get to: 50000 curr count: 23551
2021/12/15 09:40:56 Waiting for src bucket item count to get to: 50000 curr count: 23551
2021/12/15 09:41:01 Waiting for src bucket item count to get to: 50000 curr count: 34061
2021/12/15 09:41:06 Waiting for src bucket item count to get to: 50000 curr count: 34061
2021/12/15 09:41:11 Waiting for src bucket item count to get to: 50000 curr count: 34061
2021/12/15 09:41:16 Waiting for src bucket item count to get to: 50000 curr count: 44927
2021/12/15 09:41:21 Waiting for src bucket item count to get to: 50000 curr count: 44927
2021/12/15 09:41:31 Waiting for dst bucket item count to get to: 50000 curr count: 39400
2021/12/15 09:41:36 Waiting for dst bucket item count to get to: 50000 curr count: 39400
2021/12/15 09:41:36 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/15 09:41:36 Post SET ops. Expected item count: 50000 got 50000
2021/12/15 09:41:36 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 09:41:41 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/15 09:41:46 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/15 09:41:51 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/15 09:41:56 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/15 09:42:01 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/15 09:42:06 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/15 09:42:06 App: TestDuplicateTimerFiringKVRebOut got un-deployed
2021/12/15 09:42:06 No eventing-con process running
2021/12/15 09:42:07 Function: TestDuplicateTimerFiringKVRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringKVRebOut deleting in the background"
}
2021/12/15 09:42:08 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:42:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 09:42:18 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/15 09:42:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 09:42:20 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:42:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 09:42:30 Waiting for dst bucket item count to get to: 0 curr count: 10816
2021/12/15 09:42:35 Waiting for dst bucket item count to get to: 0 curr count: 10816
2021/12/15 09:42:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringKVRebOut (179.70s)
=== RUN   TestDuplicateTimerFiringKVSwapReb
2021/12/15 09:42:45 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/15 09:42:45 App: TestDuplicateTimerFiringKVSwapReb got un-deployed
2021/12/15 09:42:45 No eventing-con process running
2021/12/15 09:42:45 Function: TestDuplicateTimerFiringKVSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringKVSwapReb not found"
 }
}
2021/12/15 09:42:47 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:42:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 09:42:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 09:42:54 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:42:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 09:42:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:43:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 09:43:04 Deploying app: TestDuplicateTimerFiringKVSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/15 09:43:05 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringKVSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/15 09:43:10 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get deployed
2021/12/15 09:43:15 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get deployed
2021/12/15 09:43:20 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get deployed
2021/12/15 09:43:25 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get deployed
2021/12/15 09:43:25 App: TestDuplicateTimerFiringKVSwapReb got deployed
2021/12/15 09:43:25 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/15 09:43:25 Starting bucket ops to source bucket
2021/12/15 09:43:25 Removing node(s): [] from the cluster
2021/12/15 09:43:30 Rebalance progress: 12.5
2021/12/15 09:43:35 Rebalance progress: 12.5
2021/12/15 09:43:40 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:43:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:43:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 09:43:40 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/15 09:43:41 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/15 09:43:46 Rebalance progress: 12.5
2021/12/15 09:43:51 Rebalance progress: 12.5
2021/12/15 09:43:56 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:43:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:43:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 09:43:56 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/15 09:44:01 Rebalance progress: 12.5
2021/12/15 09:44:06 Rebalance progress: 12.5
2021/12/15 09:44:11 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:44:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:44:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 09:44:11 Waiting for src bucket item count to get to: 50000 curr count: 49176
2021/12/15 09:44:16 Waiting for src bucket item count to get to: 50000 curr count: 49176
2021/12/15 09:44:26 Waiting for dst bucket item count to get to: 50000 curr count: 45980
2021/12/15 09:44:31 Waiting for dst bucket item count to get to: 50000 curr count: 45980
2021/12/15 09:44:31 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/15 09:44:31 Post SET ops. Expected item count: 50000 got 50000
2021/12/15 09:44:31 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 09:44:36 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/15 09:44:41 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/15 09:44:46 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/15 09:44:51 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/15 09:44:56 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/15 09:45:01 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/15 09:45:01 App: TestDuplicateTimerFiringKVSwapReb got un-deployed
2021/12/15 09:45:01 No eventing-con process running
2021/12/15 09:45:01 Function: TestDuplicateTimerFiringKVSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringKVSwapReb deleting in the background"
}
2021/12/15 09:45:03 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:45:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 09:45:13 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/15 09:45:18 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/15 09:45:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 09:45:20 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:45:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 09:45:30 Waiting for dst bucket item count to get to: 0 curr count: 2838
2021/12/15 09:45:35 Waiting for dst bucket item count to get to: 0 curr count: 2838
2021/12/15 09:45:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringKVSwapReb (179.80s)
=== RUN   TestKVRebOnUpdateBucketOpOneByOne
2021/12/15 09:45:45 Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 09:45:45 App: bucket_op_on_update got un-deployed
2021/12/15 09:45:45 No eventing-con process running
2021/12/15 09:45:45 Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/15 09:45:47 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:45:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 09:45:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 09:45:54 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:45:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 09:45:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:46:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 09:46:04 Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/15 09:46:05 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/15 09:46:10 Starting bucket ops to source bucket
2021/12/15 09:46:15 Waiting for app: bucket_op_on_update to get deployed
2021/12/15 09:46:20 Waiting for app: bucket_op_on_update to get deployed
2021/12/15 09:46:25 Waiting for app: bucket_op_on_update to get deployed
2021/12/15 09:46:25 App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:46:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:46:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 09:46:25 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/15 09:46:25 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/15 09:46:30 Rebalance progress: 12.5
2021/12/15 09:46:35 Rebalance progress: 12.5
2021/12/15 09:46:40 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:46:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:46:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 09:46:40 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/15 09:46:41 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/15 09:46:46 Rebalance progress: 12.5
2021/12/15 09:46:51 Rebalance progress: 12.5
2021/12/15 09:46:56 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:46:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:46:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 09:46:56 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: kv to the cluster
2021/12/15 09:46:56 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/15 09:47:01 Rebalance progress: 12.5
2021/12/15 09:47:06 Rebalance progress: 12.5
2021/12/15 09:47:11 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:47:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:47:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 09:47:11 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/15 09:47:16 Rebalance progress: 12.5
2021/12/15 09:47:21 Rebalance progress: 12.5
2021/12/15 09:47:26 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:47:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:47:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 09:47:26 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/15 09:47:31 Rebalance progress: 12.5
2021/12/15 09:47:36 Rebalance progress: 12.5
2021/12/15 09:47:41 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:47:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:47:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 09:47:42 clusterSetup.go:253: Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/15 09:47:47 Rebalance progress: 12.5
2021/12/15 09:47:52 Rebalance progress: 12.5
2021/12/15 09:47:57 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:47:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:47:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 09:47:57 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 09:48:02 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 09:48:07 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 09:48:12 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 09:48:17 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 09:48:22 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 09:48:27 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 09:48:27 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/15 09:48:27 eventingSetup.go:46: No eventing-con process running
2021/12/15 09:48:27 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/15 09:48:29 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:48:34 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 09:48:39 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 10686
2021/12/15 09:48:39 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 09:48:41 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:48:46 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 09:48:51 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7142
2021/12/15 09:48:56 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7142
2021/12/15 09:48:56 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebOnUpdateBucketOpOneByOne (200.67s)
=== RUN   TestKVForceFailover
2021/12/15 09:49:11 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 09:49:11 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/15 09:49:11 eventingSetup.go:46: No eventing-con process running
2021/12/15 09:49:11 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/15 09:49:12 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:49:17 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 09:49:17 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 09:49:19 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:49:24 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 09:49:24 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 09:49:24 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/15 09:49:25 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/15 09:49:30 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:49:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 09:49:30 Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/15 09:49:31 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/15 09:49:31 Starting bucket ops to source bucket
2021/12/15 09:49:36 Waiting for app: bucket_op_on_update to get deployed
2021/12/15 09:49:41 Waiting for app: bucket_op_on_update to get deployed
2021/12/15 09:49:46 Waiting for app: bucket_op_on_update to get deployed
2021/12/15 09:49:51 Waiting for app: bucket_op_on_update to get deployed
2021/12/15 09:49:51 App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:49:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:49:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 09:49:51 clusterSetup.go:229: Failing over: [127.0.0.1:19001]
2021/12/15 09:50:11 clusterSetup.go:237: Kicking off failover recovery, type: full
2021/12/15 09:50:11 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/15 09:50:16 Rebalance progress: 12.5
2021/12/15 09:50:21 Rebalance progress: 12.5
2021/12/15 09:50:26 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:50:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:50:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 09:50:27 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002] from the cluster
2021/12/15 09:50:32 Rebalance progress: 12.5
2021/12/15 09:50:37 Rebalance progress: 12.5
2021/12/15 09:50:42 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:50:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:50:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 09:50:42 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 09:50:47 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 09:50:52 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 09:50:57 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 09:51:02 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 09:51:07 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 09:51:12 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 09:51:12 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/15 09:51:12 eventingSetup.go:46: No eventing-con process running
2021/12/15 09:51:12 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/15 09:51:14 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:51:19 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 09:51:24 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7043
2021/12/15 09:51:29 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7043
2021/12/15 09:51:29 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 09:51:31 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:51:36 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 09:51:41 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4275
2021/12/15 09:51:46 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4275
2021/12/15 09:51:46 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVForceFailover (169.94s)
=== RUN   TestKVHardFailoverRecovery
2021/12/15 09:51:56 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 09:51:56 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/15 09:51:56 eventingSetup.go:46: No eventing-con process running
2021/12/15 09:51:56 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/15 09:51:57 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:52:02 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 09:52:02 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 09:52:04 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:52:09 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 09:52:09 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:52:14 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 09:52:14 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/15 09:52:15 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/15 09:52:20 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/15 09:52:25 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/15 09:52:30 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/15 09:52:35 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/15 09:52:35 clusterSetup.go:354: App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:52:35 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:52:35 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 09:52:35 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/15 09:52:36 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/15 09:52:41 Rebalance progress: 12.5
2021/12/15 09:52:46 Rebalance progress: 12.5
2021/12/15 09:52:51 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:52:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:52:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 09:52:51 clusterSetup.go:229: Failing over: [127.0.0.1:19001]
2021/12/15 09:53:51 clusterSetup.go:237: Kicking off failover recovery, type: full
2021/12/15 09:53:56 Rebalance progress: 12.5
2021/12/15 09:54:01 Rebalance progress: 12.5
2021/12/15 09:54:06 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:54:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:54:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 09:54:06 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/15 09:54:11 Rebalance progress: 12.5
2021/12/15 09:54:16 Rebalance progress: 12.5
2021/12/15 09:54:21 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:54:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:54:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 09:54:21 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 09:54:26 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 09:54:31 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 09:54:36 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 09:54:41 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 09:54:46 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 09:54:51 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 09:54:51 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/15 09:54:51 eventingSetup.go:46: No eventing-con process running
2021/12/15 09:54:51 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/15 09:54:53 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:54:58 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 09:55:03 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12062
2021/12/15 09:55:08 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12062
2021/12/15 09:55:08 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 09:55:10 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:55:15 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 09:55:20 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1993
2021/12/15 09:55:25 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1993
2021/12/15 09:55:25 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVHardFailoverRecovery (219.52s)
=== RUN   TestKVRebOnUpdateBucketOpAllAtOnce
2021/12/15 09:55:35 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 09:55:35 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/15 09:55:35 eventingSetup.go:46: No eventing-con process running
2021/12/15 09:55:35 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/15 09:55:37 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:55:42 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 09:55:42 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 09:55:44 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:55:49 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 09:55:49 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:55:54 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 09:55:54 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/15 09:55:55 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/15 09:56:00 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/15 09:56:05 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/15 09:56:10 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/15 09:56:15 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/15 09:56:15 clusterSetup.go:354: App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:56:15 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:56:15 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 09:56:15 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/15 09:56:15 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/15 09:56:15 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: kv to the cluster
2021/12/15 09:56:15 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/15 09:56:21 Rebalance progress: 12.5
2021/12/15 09:56:26 Rebalance progress: 12.5
2021/12/15 09:56:31 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:56:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:56:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 09:56:31 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002 127.0.0.1:19003] from the cluster
2021/12/15 09:56:36 Rebalance progress: 12.5
2021/12/15 09:56:41 Rebalance progress: 12.5
2021/12/15 09:56:46 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:56:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:56:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 09:56:46 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 09:56:51 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 09:56:56 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 09:57:01 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 09:57:06 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 09:57:11 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 09:57:16 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 09:57:16 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/15 09:57:16 eventingSetup.go:46: No eventing-con process running
2021/12/15 09:57:16 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/15 09:57:18 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:57:23 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 09:57:28 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4577
2021/12/15 09:57:28 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 09:57:29 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:57:34 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 09:57:34 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebOnUpdateBucketOpAllAtOnce (129.39s)
=== RUN   TestKVSwapRebOnUpdateBucketOp
2021/12/15 09:57:44 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 09:57:44 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/15 09:57:45 eventingSetup.go:46: No eventing-con process running
2021/12/15 09:57:45 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/15 09:57:46 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:57:51 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 09:57:51 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 09:57:53 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:57:58 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 09:57:58 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:57:58 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 09:57:58 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/15 09:57:59 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/15 09:58:04 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/15 09:58:09 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/15 09:58:14 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/15 09:58:19 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/15 09:58:19 clusterSetup.go:354: App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:58:19 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:58:19 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 09:58:19 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/15 09:58:20 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/15 09:58:25 Rebalance progress: 12.5
2021/12/15 09:58:30 Rebalance progress: 12.5
2021/12/15 09:58:35 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:58:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:58:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 09:58:35 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/15 09:58:35 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/15 09:58:40 Rebalance progress: 12.5
2021/12/15 09:58:45 Rebalance progress: 12.5
2021/12/15 09:58:50 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:58:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:58:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 09:58:50 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/15 09:58:55 Rebalance progress: 12.5
2021/12/15 09:59:00 Rebalance progress: 12.5
2021/12/15 09:59:05 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:59:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:59:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 09:59:05 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 09:59:10 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 09:59:15 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 09:59:20 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 09:59:25 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 09:59:30 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 09:59:36 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 09:59:36 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/15 09:59:36 eventingSetup.go:46: No eventing-con process running
2021/12/15 09:59:36 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/15 09:59:37 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:59:42 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 09:59:47 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6079
2021/12/15 09:59:47 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 09:59:49 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 09:59:54 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 09:59:59 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6079
2021/12/15 09:59:59 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 10:00:04 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 10:00:04 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/15 10:00:04 eventingSetup.go:46: No eventing-con process running
2021/12/15 10:00:04 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/15 10:00:06 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 10:00:11 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 10:00:11 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 10:00:13 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 10:00:18 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 10:00:18 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVSwapRebOnUpdateBucketOp (163.27s)
=== RUN   TestKVRebalanceWithMultipleHandlers
2021/12/15 10:00:28 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 10:00:28 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/15 10:00:28 eventingSetup.go:46: No eventing-con process running
2021/12/15 10:00:28 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/15 10:00:30 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 10:00:35 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 10:00:35 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 10:00:36 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 10:00:41 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 10:00:41 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 10:00:46 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/15 10:00:46 clusterSetup.go:405: App: sys_test_bucket_op got un-deployed
2021/12/15 10:00:46 eventingSetup.go:46: No eventing-con process running
2021/12/15 10:00:46 eventingSetup.go:418: Function: sys_test_bucket_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: sys_test_bucket_op not found"
 }
}
2021/12/15 10:00:48 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 10:00:53 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 10:00:53 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 10:00:55 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 10:01:00 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 10:01:00 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 10:01:00 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 10:01:00 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/15 10:01:01 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 10:01:01 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 10:01:01 eventingSetup.go:125: Deploying app: sys_test_bucket_op. Item count src bucket: 0 dst bucket: 0
2021/12/15 10:01:02 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'sys_test_bucket_op' in metakv",
  "warnings": null
 }
}
2021/12/15 10:01:07 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/15 10:01:12 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/15 10:01:17 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/15 10:01:22 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/15 10:01:22 clusterSetup.go:354: App: bucket_op_on_update got deployed
2021/12/15 10:01:27 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/15 10:01:32 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/15 10:01:37 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/15 10:01:37 clusterSetup.go:354: App: sys_test_bucket_op got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 10:01:37 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 10:01:37 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 10:01:37 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/15 10:01:38 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/15 10:01:43 Rebalance progress: 12.5
2021/12/15 10:01:48 Rebalance progress: 12.5
2021/12/15 10:01:53 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 10:01:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 10:01:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 10:01:53 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/15 10:01:53 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/15 10:01:58 Rebalance progress: 12.5
2021/12/15 10:02:03 Rebalance progress: 12.5
2021/12/15 10:02:08 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 10:02:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 10:02:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 10:02:08 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: kv to the cluster
2021/12/15 10:02:08 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/15 10:02:14 Rebalance progress: 12.5
2021/12/15 10:02:19 Rebalance progress: 12.5
2021/12/15 10:02:24 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 10:02:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 10:02:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 10:02:24 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/15 10:02:29 Rebalance progress: 12.5
2021/12/15 10:02:34 Rebalance progress: 12.5
2021/12/15 10:02:39 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 10:02:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 10:02:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 10:02:39 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/15 10:02:44 Rebalance progress: 12.5
2021/12/15 10:02:49 Rebalance progress: 12.5
2021/12/15 10:02:54 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 10:02:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 10:02:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 10:02:54 clusterSetup.go:253: Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/15 10:02:59 Rebalance progress: 12.5
2021/12/15 10:03:04 Rebalance progress: 12.5
2021/12/15 10:03:09 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 10:03:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 10:03:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 10:03:09 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 10:03:14 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 10:03:19 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 10:03:24 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 10:03:29 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 10:03:34 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 10:03:39 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 10:03:39 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/15 10:03:39 eventingSetup.go:46: No eventing-con process running
2021/12/15 10:03:39 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/15 10:03:45 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 10:03:50 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 10:03:55 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12157
2021/12/15 10:03:55 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 10:03:57 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 10:04:02 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 10:04:07 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12157
2021/12/15 10:04:07 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 10:04:07 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 10:04:12 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/15 10:04:17 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/15 10:04:22 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/15 10:04:27 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/15 10:04:32 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/15 10:04:37 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/15 10:04:37 clusterSetup.go:405: App: sys_test_bucket_op got un-deployed
2021/12/15 10:04:37 eventingSetup.go:46: No eventing-con process running
2021/12/15 10:04:37 eventingSetup.go:418: Function: sys_test_bucket_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: sys_test_bucket_op deleting in the background"
}
2021/12/15 10:04:39 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 10:04:44 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 10:04:44 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 10:04:46 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 10:04:51 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 10:04:51 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebalanceWithMultipleHandlers (272.80s)
=== RUN   TestKVSwapRebalanceWithMultipleHandlers
2021/12/15 10:05:01 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 10:05:01 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/15 10:05:01 eventingSetup.go:46: No eventing-con process running
2021/12/15 10:05:01 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/15 10:05:02 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 10:05:07 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 10:05:07 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 10:05:09 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 10:05:14 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 10:05:14 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 10:05:19 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/15 10:05:19 clusterSetup.go:405: App: sys_test_bucket_op got un-deployed
2021/12/15 10:05:19 eventingSetup.go:46: No eventing-con process running
2021/12/15 10:05:19 eventingSetup.go:418: Function: sys_test_bucket_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: sys_test_bucket_op not found"
 }
}
2021/12/15 10:05:21 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 10:05:26 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 10:05:26 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 10:05:28 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 10:05:33 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 10:05:33 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 10:05:33 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 10:05:33 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/15 10:05:34 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 10:05:34 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 10:05:34 eventingSetup.go:125: Deploying app: sys_test_bucket_op. Item count src bucket: 0 dst bucket: 0
2021/12/15 10:05:35 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'sys_test_bucket_op' in metakv",
  "warnings": null
 }
}
2021/12/15 10:05:40 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/15 10:05:45 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/15 10:05:50 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/15 10:05:55 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/15 10:05:55 clusterSetup.go:354: App: bucket_op_on_update got deployed
2021/12/15 10:06:00 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/15 10:06:05 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/15 10:06:10 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/15 10:06:10 clusterSetup.go:354: App: sys_test_bucket_op got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 10:06:10 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 10:06:10 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 10:06:10 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/15 10:06:11 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/15 10:06:16 Rebalance progress: 12.5
2021/12/15 10:06:21 Rebalance progress: 12.5
2021/12/15 10:06:26 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 10:06:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 10:06:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 10:06:26 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/15 10:06:26 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/15 10:06:31 Rebalance progress: 12.5
2021/12/15 10:06:36 Rebalance progress: 12.5
2021/12/15 10:06:41 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 10:06:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 10:06:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 10:06:41 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/15 10:06:46 Rebalance progress: 12.5
2021/12/15 10:06:51 Rebalance progress: 12.5
2021/12/15 10:06:56 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 10:06:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 10:06:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 10:06:56 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 10:07:01 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 10:07:06 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 10:07:11 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 10:07:17 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 10:07:22 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 10:07:27 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 10:07:27 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/15 10:07:27 eventingSetup.go:46: No eventing-con process running
2021/12/15 10:07:27 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/15 10:07:31 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 10:07:36 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 10:07:36 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 10:07:38 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 10:07:43 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 10:07:48 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7613
2021/12/15 10:07:48 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 10:07:48 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 10:07:53 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/15 10:07:58 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/15 10:08:03 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/15 10:08:08 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/15 10:08:13 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/15 10:08:18 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/15 10:08:18 clusterSetup.go:405: App: sys_test_bucket_op got un-deployed
2021/12/15 10:08:18 eventingSetup.go:46: No eventing-con process running
2021/12/15 10:08:18 eventingSetup.go:418: Function: sys_test_bucket_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: sys_test_bucket_op deleting in the background"
}
2021/12/15 10:08:20 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 10:08:25 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 10:08:25 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 10:08:27 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 10:08:32 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 10:08:32 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVSwapRebalanceWithMultipleHandlers (221.08s)
=== RUN   TestKVRebStopStartKVOpsOnUpdateBucketOpOneByOne
2021/12/15 10:08:42 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 10:08:42 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/15 10:08:42 eventingSetup.go:46: No eventing-con process running
2021/12/15 10:08:42 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/15 10:08:43 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 10:08:48 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 10:08:48 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 10:08:50 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 10:08:55 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 10:08:55 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 10:09:00 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 10:09:00 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/15 10:09:01 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/15 10:09:06 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/15 10:09:11 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/15 10:09:16 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/15 10:09:21 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/15 10:09:21 clusterSetup.go:354: App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 10:09:21 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 10:09:21 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 10:09:21 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/15 10:09:22 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/15 10:09:42 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 10:09:42 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 10:09:42 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 10:09:52 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 10:09:52 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 10:09:52 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/15 10:09:57 Rebalance progress: 12.5
2021/12/15 10:10:02 Rebalance progress: 12.5
2021/12/15 10:10:07 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 10:10:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 10:10:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 10:10:07 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/15 10:10:08 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/15 10:10:13 Rebalance progress: 12.5
2021/12/15 10:10:18 Rebalance progress: 12.5
2021/12/15 10:10:23 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 10:10:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 10:10:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 10:10:23 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/15 10:10:28 Rebalance progress: 12.5
2021/12/15 10:10:33 Rebalance progress: 12.5
2021/12/15 10:10:38 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 10:10:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 10:10:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 10:10:38 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 10:10:43 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 10:10:48 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 10:10:53 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 10:10:58 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 10:11:03 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 10:11:08 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 10:11:08 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/15 10:11:08 eventingSetup.go:46: No eventing-con process running
2021/12/15 10:11:08 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/15 10:11:10 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 10:11:15 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 10:11:20 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1015
2021/12/15 10:11:25 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1015
2021/12/15 10:11:25 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 10:11:27 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 10:11:32 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 10:11:37 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 9135
2021/12/15 10:11:37 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebStopStartKVOpsOnUpdateBucketOpOneByOne (185.22s)
=== RUN   TestKVFailoverOnUpdateBucketOp
2021/12/15 10:11:47 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 10:11:47 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/15 10:11:47 eventingSetup.go:46: No eventing-con process running
2021/12/15 10:11:47 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/15 10:11:49 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 10:11:54 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 10:11:54 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 10:11:55 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 10:12:00 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 10:12:00 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 10:12:05 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 10:12:05 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/15 10:12:07 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/15 10:12:12 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/15 10:12:17 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/15 10:12:22 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/15 10:12:27 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/15 10:12:27 clusterSetup.go:354: App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 10:12:27 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 10:12:27 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 10:12:27 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/15 10:12:27 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/15 10:12:32 Rebalance progress: 12.5
2021/12/15 10:12:37 Rebalance progress: 12.5
2021/12/15 10:12:42 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 10:12:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 10:12:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 10:12:42 clusterSetup.go:229: Failing over: [127.0.0.1:19001]
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 10:12:52 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 10:12:52 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 10:12:52 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/15 10:12:57 Rebalance progress: 12.5
2021/12/15 10:13:02 Rebalance progress: 12.5
2021/12/15 10:13:07 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 10:13:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 10:13:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 10:13:07 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/15 10:13:07 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/15 10:13:13 Rebalance progress: 12.5
2021/12/15 10:13:18 Rebalance progress: 12.5
2021/12/15 10:13:23 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 10:13:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 10:13:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 10:13:23 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/15 10:13:28 Rebalance progress: 12.5
2021/12/15 10:13:33 Rebalance progress: 12.5
2021/12/15 10:13:38 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 10:13:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 10:13:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 10:13:38 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 10:13:43 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 10:13:48 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 10:13:53 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 10:13:58 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 10:14:03 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 10:14:08 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 10:14:08 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/15 10:14:08 eventingSetup.go:46: No eventing-con process running
2021/12/15 10:14:08 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/15 10:14:10 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 10:14:15 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 10:14:20 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 886
2021/12/15 10:14:25 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 886
2021/12/15 10:14:25 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 10:14:27 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 10:14:32 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 10:14:37 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 8597
2021/12/15 10:14:37 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVFailoverOnUpdateBucketOp (179.98s)
=== RUN   TestBootstrapAfterKVHardFailover
2021/12/15 10:14:42 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/15 10:14:42 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/15 10:14:47 Rebalance progress: 100
2021/12/15 10:14:47 Failing over: [127.0.0.1:19001]
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 10:15:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 10:15:17 Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/15 10:15:18 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/15 10:15:23 Removing node(s): [] from the cluster
2021/12/15 10:15:23 Starting bucket ops to source bucket
2021/12/15 10:15:29 Rebalance failed. See logs for detailed reason. You can try again.
2021/12/15 10:15:34 Waiting for app: bucket_op_on_update to get deployed
2021/12/15 10:15:39 Waiting for app: bucket_op_on_update to get deployed
2021/12/15 10:15:39 App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 10:15:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 10:15:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 10:15:39 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 10:15:44 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 10:15:49 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 10:15:54 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 10:15:59 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 10:16:04 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 10:16:09 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 10:16:09 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/15 10:16:09 eventingSetup.go:46: No eventing-con process running
2021/12/15 10:16:09 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/15 10:16:11 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 10:16:16 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 10:16:21 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1167
2021/12/15 10:16:26 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1167
2021/12/15 10:16:26 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 10:16:28 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 10:16:33 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 10:16:38 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1507
2021/12/15 10:16:38 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestBootstrapAfterKVHardFailover (120.74s)
PASS
ok  	github.com/couchbase/eventing/tests/functional_tests	12246.705s

Rebalance tests took 3 hours, 24 minutes and 9 seconds.

Testing Succeeded


Version: versions-15.12.2021-02.27.cfg
Build Log: make-15.12.2021-02.27.log
Server Log: logs-15.12.2021-02.27.tar.gz

Finished