Building

Started building at 2021/12/11 01:20:32
Calculating base
Updating mirror
Basing run on 7.1.0-1891 b8242f809
Updating tree for run 11.12.2021-01.20
gocb is at 635fa6c, changes since last good build: none
libcouchbase is at f6822fc, changes since last good build: none
indexing is at 888c51e, changes since last good build: none
kv_engine is at 2499d82, changes since last good build: 
 2499d8239 MB-49767: Add extra Expects under in ADM
 1a1d5a327 MB-47604: Move NexusKVStore purgeSeqno for explicit compactions [3/3]
 335d6b4e5 MB-47604: [Nexus] Skip compaction checks if purging collections [2/3]
 94ee3a054 MB-49767: Add Expects to PDM::ctor to enforce order in trackedWrites
 c78e03a11 MB-49702: Log slow schedule times for AuxIO tasks
ns_server is at 5bd87dc, changes since last good build: 
 5bd87dc27 Add checks to make sure only JSON-ifiable terms are added as event logs
 6c8fb55b0 MB-49036: Add subdoc.ReviveDocument to bucketCapabilities
couchstore is at 5aa7c4c, changes since last good build: none
couchdb is at fca2e6b, changes since last good build: none
testrunner is at 8e115a1, changes since last good build: 
 8e115a195 no need key in azure template
 a51cf47c2 fix typo azure cloud
Switching eventing to unstable
eventing is at 9fce322, changes since last good build: none
Switching eventing-ee to unstable
eventing-ee is at de9f5f4, changes since last good build: none
Pulling in uncommitted change 160397 at refs/changes/97/160397/3
Total 23 (delta 15), reused 17 (delta 15)
[unstable d8e9434b] MB-100 : Do not fail n1ql timeout test if we are catching timeout exceptions successfully
 Author: abhijpes 
 Date: Mon Aug 30 08:24:38 2021 +0530
 1 file changed, 2 insertions(+), 2 deletions(-)
Pulling in uncommitted change 163449 at refs/changes/49/163449/1
Total 48 (delta 27), reused 39 (delta 27)
[unstable 0fcdab2d] MB-100: collect data when test fails
 Author: AnkitPrabhu 
 Date: Tue Oct 12 14:02:31 2021 +0530
 1 file changed, 2 insertions(+)
Pulling in uncommitted change 165976 at refs/changes/76/165976/17
Total 4 (delta 3), reused 4 (delta 3)
[unstable 299e7f30] MB-48538: Add new audit ids for different events
 Author: AnkitPrabhu 
 Date: Tue Nov 23 14:29:10 2021 +0530
 1 file changed, 337 insertions(+), 61 deletions(-)
Pulling in uncommitted change 166697 at refs/changes/97/166697/4
Total 10 (delta 9), reused 10 (delta 9)
[unstable 321f6e75] MB-49547: Check bucket instance to get kv node for dcp request
 Author: AnkitPrabhu 
 Date: Thu Dec 2 12:22:17 2021 +0530
 6 files changed, 38 insertions(+), 109 deletions(-)
Pulling in uncommitted change 166698 at refs/changes/98/166698/3
Total 4 (delta 3), reused 4 (delta 3)
[unstable 00371f42] MB-48334 : Fail rebalance unless all functions deployed on current encryption level
 Author: abhijpes 
 Date: Wed Dec 1 14:51:29 2021 +0530
 1 file changed, 9 insertions(+)
Pulling in uncommitted change 166812 at refs/changes/12/166812/5
Total 24 (delta 20), reused 24 (delta 20)
[unstable d0052ec2] MB-49547: Refresh bucket info from streaming endpoint instance
 Author: AnkitPrabhu 
 Date: Thu Dec 2 13:55:55 2021 +0530
 7 files changed, 147 insertions(+), 235 deletions(-)
Pulling in uncommitted change 166999 at refs/changes/99/166999/5
Total 56 (delta 47), reused 56 (delta 47)
[unstable b7ebcc07] MB-46304: Add functionality to disable eventing exposed javascript features
 Author: AnkitPrabhu 
 Date: Mon Dec 6 13:26:12 2021 +0530
 20 files changed, 217 insertions(+), 30 deletions(-)
Pulling in uncommitted change 167196 at refs/changes/96/167196/6
Total 67 (delta 57), reused 67 (delta 57)
[unstable d0922c26] MB-49720: Store failover log in checkpoint blob
 Author: AnkitPrabhu 
 Date: Wed Dec 8 15:16:22 2021 +0530
 5 files changed, 147 insertions(+), 72 deletions(-)
Pulling in uncommitted change 167183 at refs/changes/83/167183/2
Total 5 (delta 2), reused 4 (delta 2)
[unstable 86a164e] MB-46304: Throw runtime exception when curl feature is disabled
 Author: AnkitPrabhu 
 Date: Wed Dec 8 10:39:19 2021 +0530
 1 file changed, 9 insertions(+), 3 deletions(-)
Building cmakefiles and deps
Building main product

Testing

Started testing at 2021/12/11 01:36:52
Testing mode: rebalance,n1ql,functional
Starting server
Starting server: attempt 1

Functional tests

2021/12/11 01:38:23 Started functional
2021/12/11 01:38:32 Adding node: https://127.0.0.1:19001 with role: kv,index,n1ql to the cluster
2021/12/11 01:38:37 Error in adding nodes : , response: {"otpNode":"n_1@127.0.0.1"}
2021/12/11 01:38:37 Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/11 01:38:42 Removing node(s): [] from the cluster
2021/12/11 01:38:47 Rebalance progress: 0
2021/12/11 01:38:53 Rebalance progress: 0
2021/12/11 01:38:57 Rebalance progress: 4.62962962962963
2021/12/11 01:39:02 Rebalance progress: 9.25925925925926
2021/12/11 01:39:07 Rebalance progress: 12.73148148148148
2021/12/11 01:39:12 Rebalance progress: 17.50578703703703
2021/12/11 01:39:17 Rebalance progress: 20.83333333333333
2021/12/11 01:39:22 Rebalance progress: 25.46296296296296
2021/12/11 01:39:27 Rebalance progress: 100
2021/12/11 01:39:30 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/11 01:39:45 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/11 01:40:00 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/11 01:40:15 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/11 01:40:30 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/11 01:40:45 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/11 01:41:00 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/11 01:41:15 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/11 01:41:30 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/11 01:41:45 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
=== RUN   TestAdvancedGetOps
2021/12/11 01:42:03 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/11 01:42:18 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/11 01:42:33 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/11 01:42:48 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/11 01:43:03 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/11 01:43:18 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/11 01:43:33 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/11 01:43:48 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/11 01:44:03 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/11 01:44:18 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/11 01:44:33 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/11 01:44:48 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/11 01:45:03 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/11 01:45:18 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/11 01:45:33 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/11 01:45:48 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/11 01:46:03 Deploying app: TestAdvancedGetOps. Item count src bucket: 0 dst bucket: 0
2021/12/11 01:46:04 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedGetOps' in metakv",
  "warnings": null
 }
}
2021/12/11 01:46:09 Waiting for app: TestAdvancedGetOps to get deployed
2021/12/11 01:46:14 Waiting for app: TestAdvancedGetOps to get deployed
2021/12/11 01:46:19 Waiting for app: TestAdvancedGetOps to get deployed
2021/12/11 01:46:24 Waiting for app: TestAdvancedGetOps to get deployed
2021/12/11 01:46:24 App: TestAdvancedGetOps got deployed
2021/12/11 01:46:24 Starting bucket ops to source bucket
2021/12/11 01:46:29 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/11 01:46:34 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/11 01:46:34 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/11 01:46:34 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 01:46:34 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/11 01:46:34 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 01:46:39 Waiting for app: TestAdvancedGetOps to get un-deployed
2021/12/11 01:46:44 Waiting for app: TestAdvancedGetOps to get un-deployed
2021/12/11 01:46:49 Waiting for app: TestAdvancedGetOps to get un-deployed
2021/12/11 01:46:54 Waiting for app: TestAdvancedGetOps to get un-deployed
2021/12/11 01:46:54 App: TestAdvancedGetOps got un-deployed
2021/12/11 01:46:54 No eventing-con process running
2021/12/11 01:46:54 Function: TestAdvancedGetOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedGetOps deleting in the background"
}
2021/12/11 01:46:56 Context: Bucket flush, response code: 200 dump: 
2021/12/11 01:47:06 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/11 01:47:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 01:47:08 Context: Bucket flush, response code: 200 dump: 
2021/12/11 01:47:18 Waiting for dst bucket item count to get to: 0 curr count: 61
2021/12/11 01:47:23 Waiting for dst bucket item count to get to: 0 curr count: 11
2021/12/11 01:47:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedGetOps (320.12s)
=== RUN   TestAdvancedInsertOps
2021/12/11 01:47:23 Deploying app: TestAdvancedInsertOps. Item count src bucket: 0 dst bucket: 0
2021/12/11 01:47:24 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedInsertOps' in metakv",
  "warnings": null
 }
}
2021/12/11 01:47:29 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/11 01:47:34 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/11 01:47:39 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/11 01:47:44 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/11 01:47:44 App: TestAdvancedInsertOps got deployed
2021/12/11 01:47:44 Starting bucket ops to source bucket
2021/12/11 01:47:50 Waiting for dst bucket item count to get to: 1024 curr count: 0
2021/12/11 01:47:55 Waiting for dst bucket item count to get to: 1024 curr count: 512
2021/12/11 01:47:55 src & dst bucket item count matched up. src bucket count: 1024 dst bucket count: 1024
2021/12/11 01:47:55 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 01:47:55 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/11 01:47:55 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 01:48:00 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/11 01:48:05 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/11 01:48:10 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/11 01:48:15 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/11 01:48:15 App: TestAdvancedInsertOps got un-deployed
2021/12/11 01:48:15 No eventing-con process running
2021/12/11 01:48:15 Function: TestAdvancedInsertOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedInsertOps deleting in the background"
}
2021/12/11 01:48:17 Context: Bucket flush, response code: 200 dump: 
2021/12/11 01:48:27 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/11 01:48:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 01:48:29 Context: Bucket flush, response code: 200 dump: 
2021/12/11 01:48:39 Waiting for dst bucket item count to get to: 0 curr count: 919
2021/12/11 01:48:44 Waiting for dst bucket item count to get to: 0 curr count: 407
2021/12/11 01:48:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 01:48:44 Testing insert operation on source bucket
2021/12/11 01:48:44 Deploying app: TestAdvancedInsertOps. Item count src bucket: 0 dst bucket: 0
2021/12/11 01:48:45 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedInsertOps' in metakv",
  "warnings": null
 }
}
2021/12/11 01:48:50 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/11 01:48:55 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/11 01:49:00 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/11 01:49:05 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/11 01:49:05 App: TestAdvancedInsertOps got deployed
2021/12/11 01:49:05 Starting bucket ops to source bucket
2021/12/11 01:49:11 Waiting for dst bucket item count to get to: 2048 curr count: 0
2021/12/11 01:49:16 Waiting for dst bucket item count to get to: 2048 curr count: 1024
2021/12/11 01:49:16 src & dst bucket item count matched up. src bucket count: 2048 dst bucket count: 2048
2021/12/11 01:49:16 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 01:49:16 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/11 01:49:16 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 01:49:21 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/11 01:49:26 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/11 01:49:31 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/11 01:49:36 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/11 01:49:36 App: TestAdvancedInsertOps got un-deployed
2021/12/11 01:49:36 No eventing-con process running
2021/12/11 01:49:36 Function: TestAdvancedInsertOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedInsertOps deleting in the background"
}
2021/12/11 01:49:38 Context: Bucket flush, response code: 200 dump: 
2021/12/11 01:49:48 Waiting for dst bucket item count to get to: 0 curr count: 1050
2021/12/11 01:49:53 Waiting for dst bucket item count to get to: 0 curr count: 26
2021/12/11 01:49:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 01:49:54 Context: Bucket flush, response code: 200 dump: 
2021/12/11 01:49:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedInsertOps (156.48s)
=== RUN   TestAdvancedUpsertOps
2021/12/11 01:49:59 Starting bucket ops to source bucket
2021/12/11 01:50:00 Deploying app: TestAdvancedUpsertOps. Item count src bucket: 0 dst bucket: 0
2021/12/11 01:50:01 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedUpsertOps' in metakv",
  "warnings": null
 }
}
2021/12/11 01:50:06 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/11 01:50:11 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/11 01:50:16 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/11 01:50:21 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/11 01:50:21 App: TestAdvancedUpsertOps got deployed
2021/12/11 01:50:21 Starting bucket ops to source bucket
2021/12/11 01:50:27 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/11 01:50:32 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/11 01:50:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 01:50:32 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 01:50:32 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/11 01:50:32 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 01:50:37 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/11 01:50:42 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/11 01:50:47 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/11 01:50:52 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/11 01:50:52 App: TestAdvancedUpsertOps got un-deployed
2021/12/11 01:50:52 No eventing-con process running
2021/12/11 01:50:52 Function: TestAdvancedUpsertOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedUpsertOps deleting in the background"
}
2021/12/11 01:50:54 Context: Bucket flush, response code: 200 dump: 
2021/12/11 01:51:04 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/11 01:51:09 Waiting for dst bucket item count to get to: 0 curr count: 512
2021/12/11 01:51:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 01:51:11 Context: Bucket flush, response code: 200 dump: 
2021/12/11 01:51:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 01:51:16 Testing upsert operation on source bucket
2021/12/11 01:51:16 Deploying app: TestAdvancedUpsertOps. Item count src bucket: 0 dst bucket: 0
2021/12/11 01:51:17 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedUpsertOps' in metakv",
  "warnings": null
 }
}
2021/12/11 01:51:22 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/11 01:51:27 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/11 01:51:32 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/11 01:51:37 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/11 01:51:37 App: TestAdvancedUpsertOps got deployed
2021/12/11 01:51:37 Starting bucket ops to source bucket
2021/12/11 01:51:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 01:51:38 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 01:51:38 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/11 01:51:38 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 01:51:43 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/11 01:51:48 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/11 01:51:53 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/11 01:51:58 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/11 01:51:58 App: TestAdvancedUpsertOps got un-deployed
2021/12/11 01:51:58 No eventing-con process running
2021/12/11 01:51:58 Function: TestAdvancedUpsertOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedUpsertOps deleting in the background"
}
2021/12/11 01:52:00 Context: Bucket flush, response code: 200 dump: 
2021/12/11 01:52:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 01:52:07 Context: Bucket flush, response code: 200 dump: 
2021/12/11 01:52:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedUpsertOps (132.67s)
=== RUN   TestAdvancedReplaceOps
2021/12/11 01:52:12 Starting bucket ops to source bucket
2021/12/11 01:52:13 Deploying app: TestAdvancedReplaceOps. Item count src bucket: 0 dst bucket: 0
2021/12/11 01:52:14 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedReplaceOps' in metakv",
  "warnings": null
 }
}
2021/12/11 01:52:19 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/11 01:52:24 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/11 01:52:29 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/11 01:52:34 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/11 01:52:34 App: TestAdvancedReplaceOps got deployed
2021/12/11 01:52:34 Starting bucket ops to source bucket
2021/12/11 01:52:40 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/11 01:52:45 Waiting for dst bucket item count to get to: 0 curr count: 512
2021/12/11 01:52:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 01:52:45 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 01:52:45 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/11 01:52:45 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 01:52:50 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/11 01:52:55 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/11 01:53:00 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/11 01:53:05 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/11 01:53:05 App: TestAdvancedReplaceOps got un-deployed
2021/12/11 01:53:05 No eventing-con process running
2021/12/11 01:53:05 Function: TestAdvancedReplaceOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedReplaceOps deleting in the background"
}
2021/12/11 01:53:07 Context: Bucket flush, response code: 200 dump: 
2021/12/11 01:53:17 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/11 01:53:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 01:53:19 Context: Bucket flush, response code: 200 dump: 
2021/12/11 01:53:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 01:53:24 Testing upsert operation on source bucket
2021/12/11 01:53:24 Deploying app: TestAdvancedReplaceOps. Item count src bucket: 0 dst bucket: 0
2021/12/11 01:53:25 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedReplaceOps' in metakv",
  "warnings": null
 }
}
2021/12/11 01:53:30 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/11 01:53:35 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/11 01:53:40 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/11 01:53:45 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/11 01:53:45 App: TestAdvancedReplaceOps got deployed
2021/12/11 01:53:45 Starting bucket ops to source bucket
2021/12/11 01:53:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 01:53:46 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 01:53:46 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/11 01:53:46 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 01:53:51 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/11 01:53:56 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/11 01:54:01 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/11 01:54:06 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/11 01:54:06 App: TestAdvancedReplaceOps got un-deployed
2021/12/11 01:54:06 No eventing-con process running
2021/12/11 01:54:06 Function: TestAdvancedReplaceOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedReplaceOps deleting in the background"
}
2021/12/11 01:54:08 Context: Bucket flush, response code: 200 dump: 
2021/12/11 01:54:18 Waiting for dst bucket item count to get to: 0 curr count: 236
2021/12/11 01:54:23 Waiting for dst bucket item count to get to: 0 curr count: 69
2021/12/11 01:54:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 01:54:25 Context: Bucket flush, response code: 200 dump: 
2021/12/11 01:54:30 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedReplaceOps (137.70s)
=== RUN   TestAdvancedDeleteOps
2021/12/11 01:54:30 Starting bucket ops to source bucket
2021/12/11 01:54:30 Deploying app: TestAdvancedDeleteOps. Item count src bucket: 0 dst bucket: 0
2021/12/11 01:54:31 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedDeleteOps' in metakv",
  "warnings": null
 }
}
2021/12/11 01:54:36 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/11 01:54:41 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/11 01:54:46 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/11 01:54:51 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/11 01:54:51 App: TestAdvancedDeleteOps got deployed
2021/12/11 01:54:51 Starting bucket ops to source bucket
2021/12/11 01:54:56 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/11 01:55:01 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/11 01:55:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 01:55:01 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 01:55:01 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/11 01:55:01 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 01:55:06 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/11 01:55:11 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/11 01:55:16 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/11 01:55:21 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/11 01:55:21 App: TestAdvancedDeleteOps got un-deployed
2021/12/11 01:55:21 No eventing-con process running
2021/12/11 01:55:21 Function: TestAdvancedDeleteOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedDeleteOps deleting in the background"
}
2021/12/11 01:55:23 Context: Bucket flush, response code: 200 dump: 
2021/12/11 01:55:33 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/11 01:55:38 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/11 01:55:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 01:55:40 Context: Bucket flush, response code: 200 dump: 
2021/12/11 01:55:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 01:55:45 Testing delete operation on source bucket
2021/12/11 01:55:45 Deploying app: TestAdvancedDeleteOps. Item count src bucket: 0 dst bucket: 0
2021/12/11 01:55:46 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedDeleteOps' in metakv",
  "warnings": null
 }
}
2021/12/11 01:55:51 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/11 01:55:56 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/11 01:56:01 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/11 01:56:06 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/11 01:56:06 App: TestAdvancedDeleteOps got deployed
2021/12/11 01:56:06 Starting bucket ops to source bucket
2021/12/11 01:56:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 01:56:06 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 01:56:06 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/11 01:56:06 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 01:56:11 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/11 01:56:16 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/11 01:56:21 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/11 01:56:26 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/11 01:56:26 App: TestAdvancedDeleteOps got un-deployed
2021/12/11 01:56:26 No eventing-con process running
2021/12/11 01:56:26 Function: TestAdvancedDeleteOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedDeleteOps deleting in the background"
}
2021/12/11 01:56:28 Context: Bucket flush, response code: 200 dump: 
2021/12/11 01:56:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 01:56:35 Context: Bucket flush, response code: 200 dump: 
2021/12/11 01:56:40 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedDeleteOps (130.58s)
=== RUN   TestEnoentAdvancedGet
2021/12/11 01:56:40 Deploying app: TestEnoentAdvancedGet. Item count src bucket: 0 dst bucket: 0
2021/12/11 01:56:41 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedGet' in metakv",
  "warnings": null
 }
}
2021/12/11 01:56:46 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/11 01:56:51 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/11 01:56:56 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/11 01:57:02 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/11 01:57:02 App: TestEnoentAdvancedGet got deployed
2021/12/11 01:57:02 Starting bucket ops to source bucket
2021/12/11 01:57:07 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/11 01:57:12 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/11 01:57:12 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/11 01:57:12 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 01:57:12 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/11 01:57:12 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 01:57:17 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/11 01:57:22 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/11 01:57:27 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/11 01:57:32 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/11 01:57:32 App: TestEnoentAdvancedGet got un-deployed
2021/12/11 01:57:32 No eventing-con process running
2021/12/11 01:57:32 Function: TestEnoentAdvancedGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedGet deleting in the background"
}
2021/12/11 01:57:34 Context: Bucket flush, response code: 200 dump: 
2021/12/11 01:57:44 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/11 01:57:49 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/11 01:57:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 01:57:50 Context: Bucket flush, response code: 200 dump: 
2021/12/11 01:58:00 Waiting for dst bucket item count to get to: 0 curr count: 74
2021/12/11 01:58:05 Waiting for dst bucket item count to get to: 0 curr count: 74
2021/12/11 01:58:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 01:58:05 Testing get enoent operation on source bucket
2021/12/11 01:58:05 Deploying app: TestEnoentAdvancedGet. Item count src bucket: 0 dst bucket: 0
2021/12/11 01:58:06 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedGet' in metakv",
  "warnings": null
 }
}
2021/12/11 01:58:11 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/11 01:58:17 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/11 01:58:22 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/11 01:58:27 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/11 01:58:27 App: TestEnoentAdvancedGet got deployed
2021/12/11 01:58:27 Starting bucket ops to source bucket
2021/12/11 01:58:32 Waiting for dst bucket item count to get to: 200 curr count: 0
2021/12/11 01:58:32 src & dst bucket item count matched up. src bucket count: 200 dst bucket count: 200
2021/12/11 01:58:32 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 01:58:32 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/11 01:58:32 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 01:58:37 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/11 01:58:42 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/11 01:58:47 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/11 01:58:52 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/11 01:58:52 App: TestEnoentAdvancedGet got un-deployed
2021/12/11 01:58:52 No eventing-con process running
2021/12/11 01:58:52 Function: TestEnoentAdvancedGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedGet deleting in the background"
}
2021/12/11 01:58:54 Context: Bucket flush, response code: 200 dump: 
2021/12/11 01:59:04 Waiting for dst bucket item count to get to: 0 curr count: 200
2021/12/11 01:59:09 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/11 01:59:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 01:59:10 Context: Bucket flush, response code: 200 dump: 
2021/12/11 01:59:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEnoentAdvancedGet (154.93s)
=== RUN   TestEnoentAdvancedReplace
2021/12/11 01:59:15 Deploying app: TestEnoentAdvancedReplace. Item count src bucket: 0 dst bucket: 0
2021/12/11 01:59:16 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedReplace' in metakv",
  "warnings": null
 }
}
2021/12/11 01:59:21 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/11 01:59:26 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/11 01:59:31 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/11 01:59:36 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/11 01:59:36 App: TestEnoentAdvancedReplace got deployed
2021/12/11 01:59:36 Starting bucket ops to source bucket
2021/12/11 01:59:42 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/11 01:59:42 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/11 01:59:42 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 01:59:42 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/11 01:59:42 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 01:59:47 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/11 01:59:52 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/11 01:59:57 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/11 02:00:02 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/11 02:00:02 App: TestEnoentAdvancedReplace got un-deployed
2021/12/11 02:00:02 No eventing-con process running
2021/12/11 02:00:02 Function: TestEnoentAdvancedReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedReplace deleting in the background"
}
2021/12/11 02:00:03 Context: Bucket flush, response code: 200 dump: 
2021/12/11 02:00:13 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/11 02:00:18 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/11 02:00:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 02:00:20 Context: Bucket flush, response code: 200 dump: 
2021/12/11 02:00:30 Waiting for dst bucket item count to get to: 0 curr count: 71
2021/12/11 02:00:35 Waiting for dst bucket item count to get to: 0 curr count: 71
2021/12/11 02:00:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 02:00:35 Testing get enoent operation on source bucket
2021/12/11 02:00:35 Deploying app: TestEnoentAdvancedReplace. Item count src bucket: 0 dst bucket: 0
2021/12/11 02:00:36 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedReplace' in metakv",
  "warnings": null
 }
}
2021/12/11 02:00:41 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/11 02:00:46 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/11 02:00:51 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/11 02:00:56 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/11 02:00:56 App: TestEnoentAdvancedReplace got deployed
2021/12/11 02:00:56 Starting bucket ops to source bucket
2021/12/11 02:01:01 Waiting for dst bucket item count to get to: 200 curr count: 0
2021/12/11 02:01:01 src & dst bucket item count matched up. src bucket count: 200 dst bucket count: 200
2021/12/11 02:01:01 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 02:01:01 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/11 02:01:02 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 02:01:07 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/11 02:01:12 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/11 02:01:17 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/11 02:01:22 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/11 02:01:22 App: TestEnoentAdvancedReplace got un-deployed
2021/12/11 02:01:22 No eventing-con process running
2021/12/11 02:01:22 Function: TestEnoentAdvancedReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedReplace deleting in the background"
}
2021/12/11 02:01:23 Context: Bucket flush, response code: 200 dump: 
2021/12/11 02:01:33 Waiting for dst bucket item count to get to: 0 curr count: 200
2021/12/11 02:01:38 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/11 02:01:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 02:01:40 Context: Bucket flush, response code: 200 dump: 
2021/12/11 02:01:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEnoentAdvancedReplace (149.76s)
=== RUN   TestEnoentAdvancedDelete
2021/12/11 02:01:45 Deploying app: TestEnoentAdvancedDelete. Item count src bucket: 0 dst bucket: 0
2021/12/11 02:01:46 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedDelete' in metakv",
  "warnings": null
 }
}
2021/12/11 02:01:51 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/11 02:01:56 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/11 02:02:01 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/11 02:02:06 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/11 02:02:06 App: TestEnoentAdvancedDelete got deployed
2021/12/11 02:02:06 Starting bucket ops to source bucket
2021/12/11 02:02:11 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/11 02:02:11 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/11 02:02:11 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 02:02:11 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/11 02:02:11 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 02:02:16 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/11 02:02:21 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/11 02:02:26 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/11 02:02:31 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/11 02:02:31 App: TestEnoentAdvancedDelete got un-deployed
2021/12/11 02:02:31 No eventing-con process running
2021/12/11 02:02:32 Function: TestEnoentAdvancedDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedDelete deleting in the background"
}
2021/12/11 02:02:33 Context: Bucket flush, response code: 200 dump: 
2021/12/11 02:02:43 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/11 02:02:48 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/11 02:02:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 02:02:50 Context: Bucket flush, response code: 200 dump: 
2021/12/11 02:03:00 Waiting for dst bucket item count to get to: 0 curr count: 60
2021/12/11 02:03:05 Waiting for dst bucket item count to get to: 0 curr count: 60
2021/12/11 02:03:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 02:03:05 Testing get enoent operation on source bucket
2021/12/11 02:03:05 Deploying app: TestEnoentAdvancedDelete. Item count src bucket: 0 dst bucket: 0
2021/12/11 02:03:06 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedDelete' in metakv",
  "warnings": null
 }
}
2021/12/11 02:03:11 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/11 02:03:16 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/11 02:03:21 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/11 02:03:26 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/11 02:03:26 App: TestEnoentAdvancedDelete got deployed
2021/12/11 02:03:26 Starting bucket ops to source bucket
2021/12/11 02:03:31 Waiting for dst bucket item count to get to: 200 curr count: 0
2021/12/11 02:03:31 src & dst bucket item count matched up. src bucket count: 200 dst bucket count: 200
2021/12/11 02:03:31 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 02:03:31 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/11 02:03:31 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 02:03:36 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/11 02:03:41 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/11 02:03:46 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/11 02:03:51 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/11 02:03:51 App: TestEnoentAdvancedDelete got un-deployed
2021/12/11 02:03:51 No eventing-con process running
2021/12/11 02:03:52 Function: TestEnoentAdvancedDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedDelete deleting in the background"
}
2021/12/11 02:03:53 Context: Bucket flush, response code: 200 dump: 
2021/12/11 02:04:03 Waiting for dst bucket item count to get to: 0 curr count: 200
2021/12/11 02:04:08 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/11 02:04:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 02:04:10 Context: Bucket flush, response code: 200 dump: 
2021/12/11 02:04:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEnoentAdvancedDelete (149.92s)
=== RUN   TestAdvancedInsertKeyExist
2021/12/11 02:04:15 Deploying app: TestAdvancedInsertKeyExist. Item count src bucket: 0 dst bucket: 0
2021/12/11 02:04:16 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedInsertKeyExist' in metakv",
  "warnings": null
 }
}
2021/12/11 02:04:21 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/11 02:04:26 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/11 02:04:31 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/11 02:04:36 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/11 02:04:36 App: TestAdvancedInsertKeyExist got deployed
2021/12/11 02:04:36 Starting bucket ops to source bucket
2021/12/11 02:04:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 02:04:36 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 02:04:36 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/11 02:04:36 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 02:04:41 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/11 02:04:46 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/11 02:04:51 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/11 02:04:56 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/11 02:04:56 App: TestAdvancedInsertKeyExist got un-deployed
2021/12/11 02:04:56 No eventing-con process running
2021/12/11 02:04:56 Function: TestAdvancedInsertKeyExist delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedInsertKeyExist deleting in the background"
}
2021/12/11 02:04:58 Context: Bucket flush, response code: 200 dump: 
2021/12/11 02:05:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 02:05:06 Context: Bucket flush, response code: 200 dump: 
2021/12/11 02:05:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 02:05:11 Pumping document in the destination bucket
2021/12/11 02:05:11 Starting bucket ops to source bucket
2021/12/11 02:05:11 Deploying app: TestAdvancedInsertKeyExist. Item count src bucket: 0 dst bucket: 0
2021/12/11 02:05:12 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedInsertKeyExist' in metakv",
  "warnings": null
 }
}
2021/12/11 02:05:17 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/11 02:05:22 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/11 02:05:27 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/11 02:05:32 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/11 02:05:32 App: TestAdvancedInsertKeyExist got deployed
2021/12/11 02:05:32 Starting bucket ops to source bucket
2021/12/11 02:05:37 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/11 02:05:42 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/11 02:05:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 02:05:42 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 02:05:42 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/11 02:05:42 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 02:05:47 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/11 02:05:52 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/11 02:05:57 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/11 02:06:02 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/11 02:06:02 App: TestAdvancedInsertKeyExist got un-deployed
2021/12/11 02:06:02 No eventing-con process running
2021/12/11 02:06:02 Function: TestAdvancedInsertKeyExist delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedInsertKeyExist deleting in the background"
}
2021/12/11 02:06:04 Context: Bucket flush, response code: 200 dump: 
2021/12/11 02:06:14 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/11 02:06:19 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/11 02:06:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 02:06:21 Context: Bucket flush, response code: 200 dump: 
2021/12/11 02:06:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedInsertKeyExist (130.61s)
=== RUN   TestExipryGet
2021/12/11 02:06:26 Deploying app: TestExipryGet. Item count src bucket: 0 dst bucket: 0
2021/12/11 02:06:27 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryGet' in metakv",
  "warnings": null
 }
}
2021/12/11 02:06:32 Waiting for app: TestExipryGet to get deployed
2021/12/11 02:06:37 Waiting for app: TestExipryGet to get deployed
2021/12/11 02:06:42 Waiting for app: TestExipryGet to get deployed
2021/12/11 02:06:47 Waiting for app: TestExipryGet to get deployed
2021/12/11 02:06:47 App: TestExipryGet got deployed
2021/12/11 02:06:47 Starting bucket ops to source bucket
2021/12/11 02:07:32 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/11 02:07:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 02:07:32 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 02:07:32 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/11 02:07:32 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 02:07:37 Waiting for app: TestExipryGet to get un-deployed
2021/12/11 02:07:42 Waiting for app: TestExipryGet to get un-deployed
2021/12/11 02:07:47 Waiting for app: TestExipryGet to get un-deployed
2021/12/11 02:07:52 Waiting for app: TestExipryGet to get un-deployed
2021/12/11 02:07:52 App: TestExipryGet got un-deployed
2021/12/11 02:07:52 No eventing-con process running
2021/12/11 02:07:52 Function: TestExipryGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryGet deleting in the background"
}
2021/12/11 02:07:54 Context: Bucket flush, response code: 200 dump: 
2021/12/11 02:07:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 02:08:01 Context: Bucket flush, response code: 200 dump: 
2021/12/11 02:08:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 02:08:06 Pumping document in the destination bucket for expiry get
2021/12/11 02:08:06 Starting bucket ops to source bucket
2021/12/11 02:08:06 Deploying app: TestExipryGet. Item count src bucket: 0 dst bucket: 0
2021/12/11 02:08:08 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryGet' in metakv",
  "warnings": null
 }
}
2021/12/11 02:08:13 Waiting for app: TestExipryGet to get deployed
2021/12/11 02:08:18 Waiting for app: TestExipryGet to get deployed
2021/12/11 02:08:23 Waiting for app: TestExipryGet to get deployed
2021/12/11 02:08:28 Waiting for app: TestExipryGet to get deployed
2021/12/11 02:08:28 App: TestExipryGet got deployed
2021/12/11 02:08:28 Starting bucket ops to source bucket
2021/12/11 02:09:13 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/11 02:09:18 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/11 02:09:23 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/11 02:09:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 02:09:23 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 02:09:23 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/11 02:09:23 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 02:09:28 Waiting for app: TestExipryGet to get un-deployed
2021/12/11 02:09:33 Waiting for app: TestExipryGet to get un-deployed
2021/12/11 02:09:38 Waiting for app: TestExipryGet to get un-deployed
2021/12/11 02:09:43 Waiting for app: TestExipryGet to get un-deployed
2021/12/11 02:09:43 App: TestExipryGet got un-deployed
2021/12/11 02:09:43 No eventing-con process running
2021/12/11 02:09:43 Function: TestExipryGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryGet deleting in the background"
}
2021/12/11 02:09:45 Context: Bucket flush, response code: 200 dump: 
2021/12/11 02:09:55 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/11 02:09:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 02:09:56 Context: Bucket flush, response code: 200 dump: 
2021/12/11 02:10:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestExipryGet (215.81s)
=== RUN   TestExipryInsert
2021/12/11 02:10:01 Deploying app: TestExipryInsert. Item count src bucket: 0 dst bucket: 0
2021/12/11 02:10:03 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryInsert' in metakv",
  "warnings": null
 }
}
2021/12/11 02:10:08 Waiting for app: TestExipryInsert to get deployed
2021/12/11 02:10:13 Waiting for app: TestExipryInsert to get deployed
2021/12/11 02:10:18 Waiting for app: TestExipryInsert to get deployed
2021/12/11 02:10:23 Waiting for app: TestExipryInsert to get deployed
2021/12/11 02:10:23 App: TestExipryInsert got deployed
2021/12/11 02:10:23 Starting bucket ops to source bucket
2021/12/11 02:11:08 Waiting for dst bucket item count to get to: 0 curr count: 200
2021/12/11 02:11:13 Waiting for dst bucket item count to get to: 0 curr count: 200
2021/12/11 02:11:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 02:11:13 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 02:11:13 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/11 02:11:13 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 02:11:18 Waiting for app: TestExipryInsert to get un-deployed
2021/12/11 02:11:23 Waiting for app: TestExipryInsert to get un-deployed
2021/12/11 02:11:28 Waiting for app: TestExipryInsert to get un-deployed
2021/12/11 02:11:33 Waiting for app: TestExipryInsert to get un-deployed
2021/12/11 02:11:33 App: TestExipryInsert got un-deployed
2021/12/11 02:11:33 No eventing-con process running
2021/12/11 02:11:33 Function: TestExipryInsert delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryInsert deleting in the background"
}
2021/12/11 02:11:35 Context: Bucket flush, response code: 200 dump: 
2021/12/11 02:11:40 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 02:11:42 Context: Bucket flush, response code: 200 dump: 
2021/12/11 02:11:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 02:11:47 Testing on the destination bucket for expiry insert
2021/12/11 02:11:47 Deploying app: TestExipryInsert. Item count src bucket: 0 dst bucket: 0
2021/12/11 02:11:48 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryInsert' in metakv",
  "warnings": null
 }
}
2021/12/11 02:11:53 Waiting for app: TestExipryInsert to get deployed
2021/12/11 02:11:58 Waiting for app: TestExipryInsert to get deployed
2021/12/11 02:12:03 Waiting for app: TestExipryInsert to get deployed
2021/12/11 02:12:08 Waiting for app: TestExipryInsert to get deployed
2021/12/11 02:12:08 App: TestExipryInsert got deployed
2021/12/11 02:12:08 Starting bucket ops to source bucket
2021/12/11 02:12:53 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/11 02:12:58 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/11 02:12:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 02:12:58 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 02:12:58 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/11 02:12:58 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 02:13:03 Waiting for app: TestExipryInsert to get un-deployed
2021/12/11 02:13:09 Waiting for app: TestExipryInsert to get un-deployed
2021/12/11 02:13:14 Waiting for app: TestExipryInsert to get un-deployed
2021/12/11 02:13:19 Waiting for app: TestExipryInsert to get un-deployed
2021/12/11 02:13:19 App: TestExipryInsert got un-deployed
2021/12/11 02:13:19 No eventing-con process running
2021/12/11 02:13:19 Function: TestExipryInsert delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryInsert deleting in the background"
}
2021/12/11 02:13:20 Context: Bucket flush, response code: 200 dump: 
2021/12/11 02:13:30 Waiting for dst bucket item count to get to: 0 curr count: 72
2021/12/11 02:13:35 Waiting for dst bucket item count to get to: 0 curr count: 72
2021/12/11 02:13:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 02:13:37 Context: Bucket flush, response code: 200 dump: 
2021/12/11 02:13:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestExipryInsert (220.61s)
=== RUN   TestExipryReplace
2021/12/11 02:13:42 Deploying app: TestExipryReplace. Item count src bucket: 0 dst bucket: 0
2021/12/11 02:13:43 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryReplace' in metakv",
  "warnings": null
 }
}
2021/12/11 02:13:48 Waiting for app: TestExipryReplace to get deployed
2021/12/11 02:13:53 Waiting for app: TestExipryReplace to get deployed
2021/12/11 02:13:58 Waiting for app: TestExipryReplace to get deployed
2021/12/11 02:14:03 Waiting for app: TestExipryReplace to get deployed
2021/12/11 02:14:03 App: TestExipryReplace got deployed
2021/12/11 02:14:03 Starting bucket ops to source bucket
2021/12/11 02:14:48 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/11 02:14:53 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/11 02:14:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 02:14:53 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 02:14:53 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/11 02:14:53 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 02:14:58 Waiting for app: TestExipryReplace to get un-deployed
2021/12/11 02:15:04 Waiting for app: TestExipryReplace to get un-deployed
2021/12/11 02:15:09 Waiting for app: TestExipryReplace to get un-deployed
2021/12/11 02:15:14 Waiting for app: TestExipryReplace to get un-deployed
2021/12/11 02:15:14 App: TestExipryReplace got un-deployed
2021/12/11 02:15:14 No eventing-con process running
2021/12/11 02:15:14 Function: TestExipryReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryReplace deleting in the background"
}
2021/12/11 02:15:15 Context: Bucket flush, response code: 200 dump: 
2021/12/11 02:15:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 02:15:23 Context: Bucket flush, response code: 200 dump: 
2021/12/11 02:15:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 02:15:28 Testing on the destination bucket for expiry insert
2021/12/11 02:15:28 Deploying app: TestExipryReplace. Item count src bucket: 0 dst bucket: 0
2021/12/11 02:15:29 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryReplace' in metakv",
  "warnings": null
 }
}
2021/12/11 02:15:34 Waiting for app: TestExipryReplace to get deployed
2021/12/11 02:15:39 Waiting for app: TestExipryReplace to get deployed
2021/12/11 02:15:44 Waiting for app: TestExipryReplace to get deployed
2021/12/11 02:15:49 Waiting for app: TestExipryReplace to get deployed
2021/12/11 02:15:49 App: TestExipryReplace got deployed
2021/12/11 02:15:49 Starting bucket ops to source bucket
2021/12/11 02:16:34 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/11 02:16:39 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/11 02:16:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 02:16:39 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 02:16:39 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/11 02:16:39 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 02:16:44 Waiting for app: TestExipryReplace to get un-deployed
2021/12/11 02:16:49 Waiting for app: TestExipryReplace to get un-deployed
2021/12/11 02:16:54 Waiting for app: TestExipryReplace to get un-deployed
2021/12/11 02:16:59 Waiting for app: TestExipryReplace to get un-deployed
2021/12/11 02:16:59 App: TestExipryReplace got un-deployed
2021/12/11 02:16:59 No eventing-con process running
2021/12/11 02:16:59 Function: TestExipryReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryReplace deleting in the background"
}
2021/12/11 02:17:01 Context: Bucket flush, response code: 200 dump: 
2021/12/11 02:17:11 Waiting for dst bucket item count to get to: 0 curr count: 91
2021/12/11 02:17:16 Waiting for dst bucket item count to get to: 0 curr count: 91
2021/12/11 02:17:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 02:17:17 Context: Bucket flush, response code: 200 dump: 
2021/12/11 02:17:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestExipryReplace (220.51s)
=== RUN   TestExipryUpsert
2021/12/11 02:17:23 Deploying app: TestExipryUpsert. Item count src bucket: 0 dst bucket: 0
2021/12/11 02:17:24 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryUpsert' in metakv",
  "warnings": null
 }
}
2021/12/11 02:17:29 Waiting for app: TestExipryUpsert to get deployed
2021/12/11 02:17:34 Waiting for app: TestExipryUpsert to get deployed
2021/12/11 02:17:39 Waiting for app: TestExipryUpsert to get deployed
2021/12/11 02:17:44 Waiting for app: TestExipryUpsert to get deployed
2021/12/11 02:17:44 App: TestExipryUpsert got deployed
2021/12/11 02:17:44 Starting bucket ops to source bucket
2021/12/11 02:18:29 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/11 02:18:34 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/11 02:18:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 02:18:34 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 02:18:34 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/11 02:18:34 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 02:18:39 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/11 02:18:44 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/11 02:18:49 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/11 02:18:54 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/11 02:18:54 App: TestExipryUpsert got un-deployed
2021/12/11 02:18:54 No eventing-con process running
2021/12/11 02:18:54 Function: TestExipryUpsert delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryUpsert deleting in the background"
}
2021/12/11 02:18:56 Context: Bucket flush, response code: 200 dump: 
2021/12/11 02:19:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 02:19:03 Context: Bucket flush, response code: 200 dump: 
2021/12/11 02:19:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 02:19:08 Testing on the destination bucket for expiry insert
2021/12/11 02:19:08 Deploying app: TestExipryUpsert. Item count src bucket: 0 dst bucket: 0
2021/12/11 02:19:09 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryUpsert' in metakv",
  "warnings": null
 }
}
2021/12/11 02:19:14 Waiting for app: TestExipryUpsert to get deployed
2021/12/11 02:19:19 Waiting for app: TestExipryUpsert to get deployed
2021/12/11 02:19:24 Waiting for app: TestExipryUpsert to get deployed
2021/12/11 02:19:29 Waiting for app: TestExipryUpsert to get deployed
2021/12/11 02:19:29 App: TestExipryUpsert got deployed
2021/12/11 02:19:29 Starting bucket ops to source bucket
2021/12/11 02:20:15 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/11 02:20:20 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/11 02:20:25 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/11 02:20:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 02:20:25 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 02:20:25 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/11 02:20:25 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 02:20:30 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/11 02:20:35 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/11 02:20:40 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/11 02:20:45 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/11 02:20:45 App: TestExipryUpsert got un-deployed
2021/12/11 02:20:45 No eventing-con process running
2021/12/11 02:20:45 Function: TestExipryUpsert delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryUpsert deleting in the background"
}
2021/12/11 02:20:46 Context: Bucket flush, response code: 200 dump: 
2021/12/11 02:20:56 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/11 02:20:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 02:20:58 Context: Bucket flush, response code: 200 dump: 
2021/12/11 02:21:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestExipryUpsert (220.59s)
=== RUN   TestCasReplace
2021/12/11 02:21:03 Deploying app: TestCasReplace. Item count src bucket: 0 dst bucket: 0
2021/12/11 02:21:04 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCasReplace' in metakv",
  "warnings": null
 }
}
2021/12/11 02:21:09 Waiting for app: TestCasReplace to get deployed
2021/12/11 02:21:14 Waiting for app: TestCasReplace to get deployed
2021/12/11 02:21:19 Waiting for app: TestCasReplace to get deployed
2021/12/11 02:21:24 Waiting for app: TestCasReplace to get deployed
2021/12/11 02:21:24 App: TestCasReplace got deployed
2021/12/11 02:21:24 Starting bucket ops to source bucket
2021/12/11 02:21:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 02:21:24 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 02:21:24 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/11 02:21:25 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 02:21:30 Waiting for app: TestCasReplace to get un-deployed
2021/12/11 02:21:35 Waiting for app: TestCasReplace to get un-deployed
2021/12/11 02:21:40 Waiting for app: TestCasReplace to get un-deployed
2021/12/11 02:21:45 Waiting for app: TestCasReplace to get un-deployed
2021/12/11 02:21:45 App: TestCasReplace got un-deployed
2021/12/11 02:21:45 No eventing-con process running
2021/12/11 02:21:45 Function: TestCasReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCasReplace deleting in the background"
}
2021/12/11 02:21:46 Context: Bucket flush, response code: 200 dump: 
2021/12/11 02:21:56 Waiting for dst bucket item count to get to: 0 curr count: 17
2021/12/11 02:21:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 02:21:58 Context: Bucket flush, response code: 200 dump: 
2021/12/11 02:22:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 02:22:03 Testing on the destination bucket for upsert with cas
2021/12/11 02:22:03 Starting bucket ops to source bucket
2021/12/11 02:22:03 Deploying app: TestCasReplace. Item count src bucket: 0 dst bucket: 0
2021/12/11 02:22:04 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCasReplace' in metakv",
  "warnings": null
 }
}
2021/12/11 02:22:09 Waiting for app: TestCasReplace to get deployed
2021/12/11 02:22:14 Waiting for app: TestCasReplace to get deployed
2021/12/11 02:22:19 Waiting for app: TestCasReplace to get deployed
2021/12/11 02:22:24 Waiting for app: TestCasReplace to get deployed
2021/12/11 02:22:24 App: TestCasReplace got deployed
2021/12/11 02:22:24 Starting bucket ops to source bucket
2021/12/11 02:22:30 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/11 02:22:35 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/11 02:22:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 02:22:35 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 02:22:35 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/11 02:22:35 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 02:22:40 Waiting for app: TestCasReplace to get un-deployed
2021/12/11 02:22:45 Waiting for app: TestCasReplace to get un-deployed
2021/12/11 02:22:50 Waiting for app: TestCasReplace to get un-deployed
2021/12/11 02:22:55 Waiting for app: TestCasReplace to get un-deployed
2021/12/11 02:22:55 App: TestCasReplace got un-deployed
2021/12/11 02:22:55 No eventing-con process running
2021/12/11 02:22:55 Function: TestCasReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCasReplace deleting in the background"
}
2021/12/11 02:22:57 Context: Bucket flush, response code: 200 dump: 
2021/12/11 02:23:07 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/11 02:23:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 02:23:08 Context: Bucket flush, response code: 200 dump: 
2021/12/11 02:23:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCasReplace (130.17s)
=== RUN   TestCasDelete
2021/12/11 02:23:13 Deploying app: TestCasDelete. Item count src bucket: 0 dst bucket: 0
2021/12/11 02:23:14 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCasDelete' in metakv",
  "warnings": null
 }
}
2021/12/11 02:23:19 Waiting for app: TestCasDelete to get deployed
2021/12/11 02:23:24 Waiting for app: TestCasDelete to get deployed
2021/12/11 02:23:29 Waiting for app: TestCasDelete to get deployed
2021/12/11 02:23:34 Waiting for app: TestCasDelete to get deployed
2021/12/11 02:23:34 App: TestCasDelete got deployed
2021/12/11 02:23:34 Starting bucket ops to source bucket
2021/12/11 02:23:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 02:23:35 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 02:23:35 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/11 02:23:35 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 02:23:40 Waiting for app: TestCasDelete to get un-deployed
2021/12/11 02:23:45 Waiting for app: TestCasDelete to get un-deployed
2021/12/11 02:23:50 Waiting for app: TestCasDelete to get un-deployed
2021/12/11 02:23:55 Waiting for app: TestCasDelete to get un-deployed
2021/12/11 02:23:55 App: TestCasDelete got un-deployed
2021/12/11 02:23:55 No eventing-con process running
2021/12/11 02:23:55 Function: TestCasDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCasDelete deleting in the background"
}
2021/12/11 02:23:56 Context: Bucket flush, response code: 200 dump: 
2021/12/11 02:24:07 Waiting for dst bucket item count to get to: 0 curr count: 2
2021/12/11 02:24:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 02:24:08 Context: Bucket flush, response code: 200 dump: 
2021/12/11 02:24:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 02:24:13 Testing on the destination bucket for delete with cas
2021/12/11 02:24:13 Starting bucket ops to source bucket
2021/12/11 02:24:13 Deploying app: TestCasDelete. Item count src bucket: 0 dst bucket: 0
2021/12/11 02:24:14 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCasDelete' in metakv",
  "warnings": null
 }
}
2021/12/11 02:24:19 Waiting for app: TestCasDelete to get deployed
2021/12/11 02:24:25 Waiting for app: TestCasDelete to get deployed
2021/12/11 02:24:30 Waiting for app: TestCasDelete to get deployed
2021/12/11 02:24:35 Waiting for app: TestCasDelete to get deployed
2021/12/11 02:24:35 App: TestCasDelete got deployed
2021/12/11 02:24:35 Starting bucket ops to source bucket
2021/12/11 02:24:40 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/11 02:24:45 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/11 02:24:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 02:24:45 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 02:24:45 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/11 02:24:45 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 02:24:50 Waiting for app: TestCasDelete to get un-deployed
2021/12/11 02:24:55 Waiting for app: TestCasDelete to get un-deployed
2021/12/11 02:25:00 Waiting for app: TestCasDelete to get un-deployed
2021/12/11 02:25:05 Waiting for app: TestCasDelete to get un-deployed
2021/12/11 02:25:05 App: TestCasDelete got un-deployed
2021/12/11 02:25:05 No eventing-con process running
2021/12/11 02:25:05 Function: TestCasDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCasDelete deleting in the background"
}
2021/12/11 02:25:07 Context: Bucket flush, response code: 200 dump: 
2021/12/11 02:25:17 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/11 02:25:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 02:25:18 Context: Bucket flush, response code: 200 dump: 
2021/12/11 02:25:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCasDelete (130.09s)
=== RUN   TestCountersIncrement
2021/12/11 02:25:23 Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/11 02:25:30 Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/11 02:25:31 Removing node(s): [] from the cluster
2021/12/11 02:25:36 Rebalance progress: 100
2021/12/11 02:25:36 Deploying app: TestCountersIncrement. Item count src bucket: 0 dst bucket: 0
2021/12/11 02:25:37 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCountersIncrement' in metakv",
  "warnings": null
 }
}
2021/12/11 02:25:42 Waiting for app: TestCountersIncrement to get deployed
2021/12/11 02:25:47 Waiting for app: TestCountersIncrement to get deployed
2021/12/11 02:25:52 Waiting for app: TestCountersIncrement to get deployed
2021/12/11 02:25:57 Waiting for app: TestCountersIncrement to get deployed
2021/12/11 02:25:57 App: TestCountersIncrement got deployed
2021/12/11 02:25:57 Starting bucket ops to source bucket
2021/12/11 02:26:03 Waiting for dst bucket item count to get to: 1024 curr count: 0
2021/12/11 02:26:08 Waiting for dst bucket item count to get to: 1024 curr count: 114
2021/12/11 02:26:13 Waiting for dst bucket item count to get to: 1024 curr count: 626
2021/12/11 02:26:13 src & dst bucket item count matched up. src bucket count: 1024 dst bucket count: 1024
2021/12/11 02:26:13 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 02:26:13 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 02:26:18 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/11 02:26:23 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/11 02:26:28 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/11 02:26:33 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/11 02:26:33 App: TestCountersIncrement got un-deployed
2021/12/11 02:26:33 No eventing-con process running
2021/12/11 02:26:33 Function: TestCountersIncrement delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCountersIncrement deleting in the background"
}
2021/12/11 02:26:35 Context: Bucket flush, response code: 200 dump: 
2021/12/11 02:26:45 Waiting for dst bucket item count to get to: 0 curr count: 1025
2021/12/11 02:26:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 02:26:47 Context: Bucket flush, response code: 200 dump: 
2021/12/11 02:26:57 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/11 02:26:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 02:26:57 Deploying app: TestCountersIncrement. Item count src bucket: 0 dst bucket: 0
2021/12/11 02:26:58 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCountersIncrement' in metakv",
  "warnings": null
 }
}
2021/12/11 02:27:03 Waiting for app: TestCountersIncrement to get deployed
2021/12/11 02:27:08 Waiting for app: TestCountersIncrement to get deployed
2021/12/11 02:27:13 Waiting for app: TestCountersIncrement to get deployed
2021/12/11 02:27:18 Waiting for app: TestCountersIncrement to get deployed
2021/12/11 02:27:18 App: TestCountersIncrement got deployed
2021/12/11 02:27:18 Starting bucket ops to source bucket
2021/12/11 02:27:24 Waiting for dst bucket item count to get to: 2048 curr count: 0
2021/12/11 02:27:29 Waiting for dst bucket item count to get to: 2048 curr count: 1024
2021/12/11 02:27:29 src & dst bucket item count matched up. src bucket count: 2048 dst bucket count: 2048
2021/12/11 02:27:29 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 02:27:29 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 02:27:34 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/11 02:27:39 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/11 02:27:44 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/11 02:27:49 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/11 02:27:49 App: TestCountersIncrement got un-deployed
2021/12/11 02:27:49 No eventing-con process running
2021/12/11 02:27:49 Function: TestCountersIncrement delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCountersIncrement deleting in the background"
}
2021/12/11 02:27:51 Context: Bucket flush, response code: 200 dump: 
2021/12/11 02:28:01 Waiting for dst bucket item count to get to: 0 curr count: 2048
2021/12/11 02:28:06 Waiting for dst bucket item count to get to: 0 curr count: 2048
2021/12/11 02:28:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 02:28:08 Context: Bucket flush, response code: 200 dump: 
2021/12/11 02:28:18 Waiting for dst bucket item count to get to: 0 curr count: 1
2021/12/11 02:28:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 02:28:18 Removing node(s): [http://127.0.0.1:19002 http://127.0.0.1:19003] from the cluster
2021/12/11 02:28:23 Rebalance progress: 100
--- PASS: TestCountersIncrement (179.65s)
=== RUN   TestCountersDecrement
2021/12/11 02:28:23 Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/11 02:28:23 Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/11 02:28:24 Removing node(s): [] from the cluster
2021/12/11 02:28:29 Rebalance progress: 100
2021/12/11 02:28:29 Deploying app: TestCountersDecrement. Item count src bucket: 0 dst bucket: 0
2021/12/11 02:28:30 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCountersDecrement' in metakv",
  "warnings": null
 }
}
2021/12/11 02:28:35 Waiting for app: TestCountersDecrement to get deployed
2021/12/11 02:28:40 Waiting for app: TestCountersDecrement to get deployed
2021/12/11 02:28:45 Waiting for app: TestCountersDecrement to get deployed
2021/12/11 02:28:50 Waiting for app: TestCountersDecrement to get deployed
2021/12/11 02:28:50 App: TestCountersDecrement got deployed
2021/12/11 02:28:50 Starting bucket ops to source bucket
2021/12/11 02:28:56 Waiting for dst bucket item count to get to: 1024 curr count: 0
2021/12/11 02:29:01 Waiting for dst bucket item count to get to: 1024 curr count: 0
2021/12/11 02:29:06 Waiting for dst bucket item count to get to: 1024 curr count: 512
2021/12/11 02:29:06 src & dst bucket item count matched up. src bucket count: 1024 dst bucket count: 1024
2021/12/11 02:29:06 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 02:29:07 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 02:29:12 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/11 02:29:17 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/11 02:29:22 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/11 02:29:27 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/11 02:29:27 App: TestCountersDecrement got un-deployed
2021/12/11 02:29:27 No eventing-con process running
2021/12/11 02:29:27 Function: TestCountersDecrement delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCountersDecrement deleting in the background"
}
2021/12/11 02:29:28 Context: Bucket flush, response code: 200 dump: 
2021/12/11 02:29:38 Waiting for dst bucket item count to get to: 0 curr count: 836
2021/12/11 02:29:43 Waiting for dst bucket item count to get to: 0 curr count: 323
2021/12/11 02:29:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 02:29:45 Context: Bucket flush, response code: 200 dump: 
2021/12/11 02:29:55 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/11 02:29:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 02:29:55 Deploying app: TestCountersDecrement. Item count src bucket: 0 dst bucket: 0
2021/12/11 02:29:56 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCountersDecrement' in metakv",
  "warnings": null
 }
}
2021/12/11 02:30:01 Waiting for app: TestCountersDecrement to get deployed
2021/12/11 02:30:06 Waiting for app: TestCountersDecrement to get deployed
2021/12/11 02:30:11 Waiting for app: TestCountersDecrement to get deployed
2021/12/11 02:30:16 Waiting for app: TestCountersDecrement to get deployed
2021/12/11 02:30:16 App: TestCountersDecrement got deployed
2021/12/11 02:30:16 Starting bucket ops to source bucket
2021/12/11 02:30:23 Waiting for dst bucket item count to get to: 2048 curr count: 0
2021/12/11 02:30:28 Waiting for dst bucket item count to get to: 2048 curr count: 1736
2021/12/11 02:30:33 Waiting for dst bucket item count to get to: 2048 curr count: 1736
2021/12/11 02:30:33 src & dst bucket item count matched up. src bucket count: 2048 dst bucket count: 2048
2021/12/11 02:30:33 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 02:30:33 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 02:30:38 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/11 02:30:43 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/11 02:30:48 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/11 02:30:53 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/11 02:30:53 App: TestCountersDecrement got un-deployed
2021/12/11 02:30:53 No eventing-con process running
2021/12/11 02:30:53 Function: TestCountersDecrement delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCountersDecrement deleting in the background"
}
2021/12/11 02:30:54 Context: Bucket flush, response code: 200 dump: 
2021/12/11 02:31:04 Waiting for dst bucket item count to get to: 0 curr count: 2048
2021/12/11 02:31:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 02:31:06 Context: Bucket flush, response code: 200 dump: 
2021/12/11 02:31:16 Waiting for dst bucket item count to get to: 0 curr count: 1
2021/12/11 02:31:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 02:31:16 Removing node(s): [http://127.0.0.1:19002 http://127.0.0.1:19003] from the cluster
2021/12/11 02:31:21 Rebalance progress: 100
--- PASS: TestCountersDecrement (178.28s)
=== RUN   TestCacheGet
2021/12/11 02:31:21 Deploying app: TestCacheGet. Item count src bucket: 0 dst bucket: 0
2021/12/11 02:31:23 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCacheGet' in metakv",
  "warnings": null
 }
}
2021/12/11 02:31:28 Waiting for app: TestCacheGet to get deployed
2021/12/11 02:31:33 Waiting for app: TestCacheGet to get deployed
2021/12/11 02:31:38 Waiting for app: TestCacheGet to get deployed
2021/12/11 02:31:43 Waiting for app: TestCacheGet to get deployed
2021/12/11 02:31:43 App: TestCacheGet got deployed
2021/12/11 02:31:43 Starting bucket ops to source bucket
2021/12/11 02:31:48 Waiting for dst bucket item count to get to: 20 curr count: 0
2021/12/11 02:31:53 Waiting for dst bucket item count to get to: 20 curr count: 0
2021/12/11 02:31:58 Waiting for dst bucket item count to get to: 20 curr count: 0
2021/12/11 02:32:03 Waiting for dst bucket item count to get to: 20 curr count: 0
2021/12/11 02:32:08 Waiting for dst bucket item count to get to: 20 curr count: 16
2021/12/11 02:32:13 Waiting for dst bucket item count to get to: 20 curr count: 16
2021/12/11 02:32:13 src & dst bucket item count matched up. src bucket count: 20 dst bucket count: 20
2021/12/11 02:32:13 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 02:32:13 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 02:32:18 Waiting for app: TestCacheGet to get un-deployed
2021/12/11 02:32:23 Waiting for app: TestCacheGet to get un-deployed
2021/12/11 02:32:28 Waiting for app: TestCacheGet to get un-deployed
2021/12/11 02:32:33 Waiting for app: TestCacheGet to get un-deployed
2021/12/11 02:32:33 App: TestCacheGet got un-deployed
2021/12/11 02:32:33 No eventing-con process running
2021/12/11 02:32:33 Function: TestCacheGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCacheGet deleting in the background"
}
2021/12/11 02:32:35 Context: Bucket flush, response code: 200 dump: 
2021/12/11 02:32:45 Waiting for dst bucket item count to get to: 0 curr count: 20
2021/12/11 02:32:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 02:32:46 Context: Bucket flush, response code: 200 dump: 
2021/12/11 02:32:56 Waiting for dst bucket item count to get to: 0 curr count: 20
2021/12/11 02:32:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCacheGet (95.16s)
=== RUN   TestCacheRYOW
2021/12/11 02:32:56 Deploying app: TestCacheRYOW. Item count src bucket: 0 dst bucket: 0
2021/12/11 02:32:58 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCacheRYOW' in metakv",
  "warnings": null
 }
}
2021/12/11 02:33:03 Waiting for app: TestCacheRYOW to get deployed
2021/12/11 02:33:08 Waiting for app: TestCacheRYOW to get deployed
2021/12/11 02:33:13 Waiting for app: TestCacheRYOW to get deployed
2021/12/11 02:33:18 Waiting for app: TestCacheRYOW to get deployed
2021/12/11 02:33:18 App: TestCacheRYOW got deployed
2021/12/11 02:33:18 Starting bucket ops to source bucket
2021/12/11 02:33:23 Waiting for dst bucket item count to get to: 50 curr count: 0
2021/12/11 02:33:28 Waiting for dst bucket item count to get to: 50 curr count: 25
2021/12/11 02:33:33 Waiting for dst bucket item count to get to: 50 curr count: 25
2021/12/11 02:33:33 src & dst bucket item count matched up. src bucket count: 50 dst bucket count: 50
2021/12/11 02:33:33 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 02:33:33 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 02:33:38 Waiting for app: TestCacheRYOW to get un-deployed
2021/12/11 02:33:43 Waiting for app: TestCacheRYOW to get un-deployed
2021/12/11 02:33:48 Waiting for app: TestCacheRYOW to get un-deployed
2021/12/11 02:33:53 Waiting for app: TestCacheRYOW to get un-deployed
2021/12/11 02:33:53 App: TestCacheRYOW got un-deployed
2021/12/11 02:33:53 No eventing-con process running
2021/12/11 02:33:53 Function: TestCacheRYOW delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCacheRYOW deleting in the background"
}
2021/12/11 02:33:55 Context: Bucket flush, response code: 200 dump: 
2021/12/11 02:34:05 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/11 02:34:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 02:34:07 Context: Bucket flush, response code: 200 dump: 
2021/12/11 02:34:17 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/11 02:34:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCacheRYOW (80.23s)
=== RUN   TestCacheExpiry
2021/12/11 02:34:17 Deploying app: TestCacheExpiry. Item count src bucket: 0 dst bucket: 0
2021/12/11 02:34:18 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCacheExpiry' in metakv",
  "warnings": null
 }
}
2021/12/11 02:34:23 Waiting for app: TestCacheExpiry to get deployed
2021/12/11 02:34:28 Waiting for app: TestCacheExpiry to get deployed
2021/12/11 02:34:33 Waiting for app: TestCacheExpiry to get deployed
2021/12/11 02:34:38 Waiting for app: TestCacheExpiry to get deployed
2021/12/11 02:34:38 App: TestCacheExpiry got deployed
2021/12/11 02:34:38 Starting bucket ops to source bucket
2021/12/11 02:34:43 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/11 02:34:48 Waiting for dst bucket item count to get to: 100 curr count: 3
2021/12/11 02:34:53 Waiting for dst bucket item count to get to: 100 curr count: 3
2021/12/11 02:34:58 Waiting for dst bucket item count to get to: 100 curr count: 86
2021/12/11 02:35:03 Waiting for dst bucket item count to get to: 100 curr count: 97
2021/12/11 02:35:03 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/11 02:35:03 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 02:35:03 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 02:35:08 Waiting for app: TestCacheExpiry to get un-deployed
2021/12/11 02:35:13 Waiting for app: TestCacheExpiry to get un-deployed
2021/12/11 02:35:18 Waiting for app: TestCacheExpiry to get un-deployed
2021/12/11 02:35:23 Waiting for app: TestCacheExpiry to get un-deployed
2021/12/11 02:35:23 App: TestCacheExpiry got un-deployed
2021/12/11 02:35:23 No eventing-con process running
2021/12/11 02:35:23 Function: TestCacheExpiry delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCacheExpiry deleting in the background"
}
2021/12/11 02:35:25 Context: Bucket flush, response code: 200 dump: 
2021/12/11 02:35:35 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/11 02:35:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 02:35:37 Context: Bucket flush, response code: 200 dump: 
2021/12/11 02:35:47 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/11 02:35:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCacheExpiry (90.20s)
=== RUN   TestCacheExpiryCustom
2021/12/11 02:35:47 Deploying app: TestCacheExpiryCustom. Item count src bucket: 0 dst bucket: 0
2021/12/11 02:35:48 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCacheExpiryCustom' in metakv",
  "warnings": null
 }
}
2021/12/11 02:35:53 Waiting for app: TestCacheExpiryCustom to get deployed
2021/12/11 02:35:58 Waiting for app: TestCacheExpiryCustom to get deployed
2021/12/11 02:36:03 Waiting for app: TestCacheExpiryCustom to get deployed
2021/12/11 02:36:08 Waiting for app: TestCacheExpiryCustom to get deployed
2021/12/11 02:36:08 App: TestCacheExpiryCustom got deployed
2021/12/11 02:36:08 Starting bucket ops to source bucket
2021/12/11 02:36:13 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/11 02:36:18 Waiting for dst bucket item count to get to: 100 curr count: 3
2021/12/11 02:36:23 Waiting for dst bucket item count to get to: 100 curr count: 42
2021/12/11 02:36:28 Waiting for dst bucket item count to get to: 100 curr count: 86
2021/12/11 02:36:33 Waiting for dst bucket item count to get to: 100 curr count: 97
2021/12/11 02:36:33 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/11 02:36:33 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 02:36:33 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 02:36:38 Waiting for app: TestCacheExpiryCustom to get un-deployed
2021/12/11 02:36:44 Waiting for app: TestCacheExpiryCustom to get un-deployed
2021/12/11 02:36:49 Waiting for app: TestCacheExpiryCustom to get un-deployed
2021/12/11 02:36:54 Waiting for app: TestCacheExpiryCustom to get un-deployed
2021/12/11 02:36:54 App: TestCacheExpiryCustom got un-deployed
2021/12/11 02:36:54 No eventing-con process running
2021/12/11 02:36:54 Function: TestCacheExpiryCustom delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCacheExpiryCustom deleting in the background"
}
2021/12/11 02:36:55 Context: Bucket flush, response code: 200 dump: 
2021/12/11 02:37:05 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/11 02:37:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 02:37:07 Context: Bucket flush, response code: 200 dump: 
2021/12/11 02:37:17 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/11 02:37:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCacheExpiryCustom (90.21s)
=== RUN   TestCacheOverflowCustom
2021/12/11 02:37:17 Deploying app: TestCacheOverflowCustom. Item count src bucket: 0 dst bucket: 0
2021/12/11 02:37:18 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCacheOverflowCustom' in metakv",
  "warnings": null
 }
}
2021/12/11 02:37:23 Waiting for app: TestCacheOverflowCustom to get deployed
2021/12/11 02:37:28 Waiting for app: TestCacheOverflowCustom to get deployed
2021/12/11 02:37:33 Waiting for app: TestCacheOverflowCustom to get deployed
2021/12/11 02:37:38 Waiting for app: TestCacheOverflowCustom to get deployed
2021/12/11 02:37:38 App: TestCacheOverflowCustom got deployed
2021/12/11 02:37:38 Starting bucket ops to source bucket
2021/12/11 02:37:44 Waiting for dst bucket item count to get to: 35 curr count: 0
2021/12/11 02:37:49 Waiting for dst bucket item count to get to: 35 curr count: 2
2021/12/11 02:37:54 Waiting for dst bucket item count to get to: 35 curr count: 11
2021/12/11 02:37:59 Waiting for dst bucket item count to get to: 35 curr count: 22
2021/12/11 02:38:04 Waiting for dst bucket item count to get to: 35 curr count: 30
2021/12/11 02:38:09 Waiting for dst bucket item count to get to: 35 curr count: 34
2021/12/11 02:38:14 Waiting for dst bucket item count to get to: 35 curr count: 34
2021/12/11 02:38:14 src & dst bucket item count matched up. src bucket count: 35 dst bucket count: 35
2021/12/11 02:38:14 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 02:38:14 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 02:38:19 Waiting for app: TestCacheOverflowCustom to get un-deployed
2021/12/11 02:38:24 Waiting for app: TestCacheOverflowCustom to get un-deployed
2021/12/11 02:38:29 Waiting for app: TestCacheOverflowCustom to get un-deployed
2021/12/11 02:38:34 Waiting for app: TestCacheOverflowCustom to get un-deployed
2021/12/11 02:38:34 App: TestCacheOverflowCustom got un-deployed
2021/12/11 02:38:34 No eventing-con process running
2021/12/11 02:38:34 Function: TestCacheOverflowCustom delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCacheOverflowCustom deleting in the background"
}
2021/12/11 02:38:35 Context: Bucket flush, response code: 200 dump: 
2021/12/11 02:38:45 Waiting for dst bucket item count to get to: 0 curr count: 5
2021/12/11 02:38:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 02:38:47 Context: Bucket flush, response code: 200 dump: 
2021/12/11 02:38:57 Waiting for dst bucket item count to get to: 0 curr count: 35
2021/12/11 02:38:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCacheOverflowCustom (100.21s)
=== RUN   TestStrictMode
2021/12/11 02:38:57 Deploying app: TestStrictMode. Item count src bucket: 0 dst bucket: 0
2021/12/11 02:38:58 Context: Post to main store, response code: 422 dump: {
 "name": "ERR_HANDLER_COMPILATION",
 "code": 27,
 "description": "Function compilation failed",
 "attributes": null,
 "runtime_info": {
  "code": 27,
  "info": {
   "area": "handlerCode",
   "column_number": 63,
   "compile_success": false,
   "description": "        destination[meta.id] = 00000000000000000000000000000101;\n                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nSyntaxError: Octal literals are not allowed in strict mode.\n",
   "index": 132,
   "language": "JavaScript",
   "line_number": 4
  }
 }
}
--- PASS: TestStrictMode (1.06s)
=== RUN   TestDataTypes
2021/12/11 02:38:58 Deploying app: TestDataTypes. Item count src bucket: 0 dst bucket: 0
2021/12/11 02:39:00 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDataTypes' in metakv",
  "warnings": null
 }
}
2021/12/11 02:39:05 Waiting for app: TestDataTypes to get deployed
2021/12/11 02:39:10 Waiting for app: TestDataTypes to get deployed
2021/12/11 02:39:15 Waiting for app: TestDataTypes to get deployed
2021/12/11 02:39:20 Waiting for app: TestDataTypes to get deployed
2021/12/11 02:39:20 App: TestDataTypes got deployed
2021/12/11 02:39:20 Starting bucket ops to source bucket
2021/12/11 02:39:25 Waiting for dst bucket item count to get to: 18 curr count: 0
2021/12/11 02:39:30 Waiting for dst bucket item count to get to: 18 curr count: 0
2021/12/11 02:39:35 Waiting for dst bucket item count to get to: 18 curr count: 8
2021/12/11 02:39:35 src & dst bucket item count matched up. src bucket count: 18 dst bucket count: 18
2021/12/11 02:39:35 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 02:39:35 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 02:39:40 Waiting for app: TestDataTypes to get un-deployed
2021/12/11 02:39:45 Waiting for app: TestDataTypes to get un-deployed
2021/12/11 02:39:50 Waiting for app: TestDataTypes to get un-deployed
2021/12/11 02:39:55 Waiting for app: TestDataTypes to get un-deployed
2021/12/11 02:39:55 App: TestDataTypes got un-deployed
2021/12/11 02:39:55 No eventing-con process running
2021/12/11 02:39:55 Function: TestDataTypes delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDataTypes deleting in the background"
}
2021/12/11 02:39:57 Context: Bucket flush, response code: 200 dump: 
2021/12/11 02:40:07 Waiting for dst bucket item count to get to: 0 curr count: 1
2021/12/11 02:40:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 02:40:08 Context: Bucket flush, response code: 200 dump: 
2021/12/11 02:40:18 Waiting for dst bucket item count to get to: 0 curr count: 15
2021/12/11 02:40:23 Waiting for dst bucket item count to get to: 0 curr count: 5
2021/12/11 02:40:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDataTypes (85.10s)
=== RUN   TestEnoentGet
2021/12/11 02:40:23 Deploying app: TestEnoentGet. Item count src bucket: 0 dst bucket: 0
2021/12/11 02:40:25 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentGet' in metakv",
  "warnings": null
 }
}
2021/12/11 02:40:30 Waiting for app: TestEnoentGet to get deployed
2021/12/11 02:40:35 Waiting for app: TestEnoentGet to get deployed
2021/12/11 02:40:40 Waiting for app: TestEnoentGet to get deployed
2021/12/11 02:40:45 Waiting for app: TestEnoentGet to get deployed
2021/12/11 02:40:45 App: TestEnoentGet got deployed
2021/12/11 02:40:45 Starting bucket ops to source bucket
2021/12/11 02:40:50 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/11 02:40:55 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/11 02:40:55 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/11 02:40:55 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 02:40:55 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 02:41:00 Waiting for app: TestEnoentGet to get un-deployed
2021/12/11 02:41:05 Waiting for app: TestEnoentGet to get un-deployed
2021/12/11 02:41:10 Waiting for app: TestEnoentGet to get un-deployed
2021/12/11 02:41:15 Waiting for app: TestEnoentGet to get un-deployed
2021/12/11 02:41:15 App: TestEnoentGet got un-deployed
2021/12/11 02:41:15 No eventing-con process running
2021/12/11 02:41:15 Function: TestEnoentGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentGet deleting in the background"
}
2021/12/11 02:41:17 Context: Bucket flush, response code: 200 dump: 
2021/12/11 02:41:27 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/11 02:41:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 02:41:29 Context: Bucket flush, response code: 200 dump: 
2021/12/11 02:41:39 Waiting for dst bucket item count to get to: 0 curr count: 86
2021/12/11 02:41:44 Waiting for dst bucket item count to get to: 0 curr count: 36
2021/12/11 02:41:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 02:41:44 Deploying app: TestEnoentGet. Item count src bucket: 0 dst bucket: 0
2021/12/11 02:41:45 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentGet' in metakv",
  "warnings": null
 }
}
2021/12/11 02:41:50 Waiting for app: TestEnoentGet to get deployed
2021/12/11 02:41:55 Waiting for app: TestEnoentGet to get deployed
2021/12/11 02:42:00 Waiting for app: TestEnoentGet to get deployed
2021/12/11 02:42:05 Waiting for app: TestEnoentGet to get deployed
2021/12/11 02:42:05 App: TestEnoentGet got deployed
2021/12/11 02:42:05 Starting bucket ops to source bucket
2021/12/11 02:42:10 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/11 02:42:15 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/11 02:42:15 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/11 02:42:15 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 02:42:15 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 02:42:20 Waiting for app: TestEnoentGet to get un-deployed
2021/12/11 02:42:25 Waiting for app: TestEnoentGet to get un-deployed
2021/12/11 02:42:30 Waiting for app: TestEnoentGet to get un-deployed
2021/12/11 02:42:35 Waiting for app: TestEnoentGet to get un-deployed
2021/12/11 02:42:35 App: TestEnoentGet got un-deployed
2021/12/11 02:42:35 No eventing-con process running
2021/12/11 02:42:35 Function: TestEnoentGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentGet deleting in the background"
}
2021/12/11 02:42:37 Context: Bucket flush, response code: 200 dump: 
2021/12/11 02:42:47 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/11 02:42:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 02:42:49 Context: Bucket flush, response code: 200 dump: 
2021/12/11 02:42:59 Waiting for dst bucket item count to get to: 0 curr count: 91
2021/12/11 02:43:04 Waiting for dst bucket item count to get to: 0 curr count: 41
2021/12/11 02:43:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEnoentGet (160.23s)
=== RUN   TestEnoentDelete
2021/12/11 02:43:04 Deploying app: TestEnoentDelete. Item count src bucket: 0 dst bucket: 0
2021/12/11 02:43:05 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentDelete' in metakv",
  "warnings": null
 }
}
2021/12/11 02:43:10 Waiting for app: TestEnoentDelete to get deployed
2021/12/11 02:43:15 Waiting for app: TestEnoentDelete to get deployed
2021/12/11 02:43:20 Waiting for app: TestEnoentDelete to get deployed
2021/12/11 02:43:25 Waiting for app: TestEnoentDelete to get deployed
2021/12/11 02:43:25 App: TestEnoentDelete got deployed
2021/12/11 02:43:25 Starting bucket ops to source bucket
2021/12/11 02:43:30 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/11 02:43:35 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/11 02:43:35 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/11 02:43:35 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 02:43:35 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 02:43:40 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/11 02:43:45 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/11 02:43:50 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/11 02:43:55 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/11 02:43:55 App: TestEnoentDelete got un-deployed
2021/12/11 02:43:55 No eventing-con process running
2021/12/11 02:43:55 Function: TestEnoentDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentDelete deleting in the background"
}
2021/12/11 02:43:57 Context: Bucket flush, response code: 200 dump: 
2021/12/11 02:44:07 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/11 02:44:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 02:44:09 Context: Bucket flush, response code: 200 dump: 
2021/12/11 02:44:19 Waiting for dst bucket item count to get to: 0 curr count: 94
2021/12/11 02:44:24 Waiting for dst bucket item count to get to: 0 curr count: 44
2021/12/11 02:44:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 02:44:24 Deploying app: TestEnoentDelete. Item count src bucket: 0 dst bucket: 0
2021/12/11 02:44:25 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentDelete' in metakv",
  "warnings": null
 }
}
2021/12/11 02:44:30 Waiting for app: TestEnoentDelete to get deployed
2021/12/11 02:44:35 Waiting for app: TestEnoentDelete to get deployed
2021/12/11 02:44:40 Waiting for app: TestEnoentDelete to get deployed
2021/12/11 02:44:45 Waiting for app: TestEnoentDelete to get deployed
2021/12/11 02:44:45 App: TestEnoentDelete got deployed
2021/12/11 02:44:45 Starting bucket ops to source bucket
2021/12/11 02:44:50 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/11 02:44:55 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/11 02:44:55 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/11 02:44:55 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 02:44:55 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 02:45:00 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/11 02:45:05 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/11 02:45:10 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/11 02:45:15 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/11 02:45:15 App: TestEnoentDelete got un-deployed
2021/12/11 02:45:15 No eventing-con process running
2021/12/11 02:45:15 Function: TestEnoentDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentDelete deleting in the background"
}
2021/12/11 02:45:17 Context: Bucket flush, response code: 200 dump: 
2021/12/11 02:45:27 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/11 02:45:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 02:45:29 Context: Bucket flush, response code: 200 dump: 
2021/12/11 02:45:39 Waiting for dst bucket item count to get to: 0 curr count: 99
2021/12/11 02:45:44 Waiting for dst bucket item count to get to: 0 curr count: 49
2021/12/11 02:45:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEnoentDelete (160.20s)
=== RUN   TestError
2021/12/11 02:45:54 Waiting for app: TestError to get un-deployed
2021/12/11 02:45:54 App: TestError got un-deployed
2021/12/11 02:45:54 No eventing-con process running
2021/12/11 02:45:54 Function: TestError delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestError not found"
 }
}
2021/12/11 02:45:56 Context: Bucket flush, response code: 200 dump: 
2021/12/11 02:46:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 02:46:02 Context: Bucket flush, response code: 200 dump: 
2021/12/11 02:46:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 02:46:07 Deploying app: TestError. Item count src bucket: 0 dst bucket: 0
2021/12/11 02:46:09 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestError' in metakv",
  "warnings": null
 }
}
2021/12/11 02:46:14 Waiting for app: TestError to get deployed
2021/12/11 02:46:19 Waiting for app: TestError to get deployed
2021/12/11 02:46:24 Waiting for app: TestError to get deployed
2021/12/11 02:46:29 Waiting for app: TestError to get deployed
2021/12/11 02:46:29 App: TestError got deployed
2021/12/11 02:46:29 Starting bucket ops to source bucket
2021/12/11 02:46:34 Waiting for dst bucket item count to get to: 300 curr count: 0
2021/12/11 02:46:39 Waiting for dst bucket item count to get to: 300 curr count: 144
2021/12/11 02:46:39 src & dst bucket item count matched up. src bucket count: 300 dst bucket count: 300
2021/12/11 02:46:39 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 02:46:39 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 02:46:44 Waiting for app: TestError to get un-deployed
2021/12/11 02:46:49 Waiting for app: TestError to get un-deployed
2021/12/11 02:46:54 Waiting for app: TestError to get un-deployed
2021/12/11 02:46:59 Waiting for app: TestError to get un-deployed
2021/12/11 02:46:59 App: TestError got un-deployed
2021/12/11 02:46:59 No eventing-con process running
2021/12/11 02:46:59 Function: TestError delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestError deleting in the background"
}
2021/12/11 02:47:01 Context: Bucket flush, response code: 200 dump: 
2021/12/11 02:47:11 Waiting for dst bucket item count to get to: 0 curr count: 95
2021/12/11 02:47:16 Waiting for dst bucket item count to get to: 0 curr count: 95
2021/12/11 02:47:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 02:47:18 Context: Bucket flush, response code: 200 dump: 
2021/12/11 02:47:28 Waiting for dst bucket item count to get to: 0 curr count: 146
2021/12/11 02:47:33 Waiting for dst bucket item count to get to: 0 curr count: 2
2021/12/11 02:47:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestError (108.79s)
=== RUN   TestCRLF
2021/12/11 02:47:43 Waiting for app: TestCRLF to get un-deployed
2021/12/11 02:47:43 App: TestCRLF got un-deployed
2021/12/11 02:47:43 No eventing-con process running
2021/12/11 02:47:43 Function: TestCRLF delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestCRLF not found"
 }
}
2021/12/11 02:47:44 Context: Bucket flush, response code: 200 dump: 
2021/12/11 02:47:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 02:47:51 Context: Bucket flush, response code: 200 dump: 
2021/12/11 02:47:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 02:47:56 Deploying app: TestCRLF. Item count src bucket: 0 dst bucket: 0
2021/12/11 02:47:57 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCRLF' in metakv",
  "warnings": null
 }
}
2021/12/11 02:48:02 Waiting for app: TestCRLF to get deployed
2021/12/11 02:48:07 Waiting for app: TestCRLF to get deployed
2021/12/11 02:48:12 Waiting for app: TestCRLF to get deployed
2021/12/11 02:48:17 Waiting for app: TestCRLF to get deployed
2021/12/11 02:48:17 App: TestCRLF got deployed
2021/12/11 02:48:17 Starting bucket ops to source bucket
2021/12/11 02:48:23 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/11 02:48:28 Waiting for dst bucket item count to get to: 100 curr count: 51
2021/12/11 02:48:33 Waiting for dst bucket item count to get to: 100 curr count: 51
2021/12/11 02:48:33 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/11 02:48:33 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 02:48:33 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 02:48:38 Waiting for app: TestCRLF to get un-deployed
2021/12/11 02:48:43 Waiting for app: TestCRLF to get un-deployed
2021/12/11 02:48:48 Waiting for app: TestCRLF to get un-deployed
2021/12/11 02:48:53 Waiting for app: TestCRLF to get un-deployed
2021/12/11 02:48:53 App: TestCRLF got un-deployed
2021/12/11 02:48:53 No eventing-con process running
2021/12/11 02:48:53 Function: TestCRLF delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCRLF deleting in the background"
}
2021/12/11 02:48:55 Context: Bucket flush, response code: 200 dump: 
2021/12/11 02:49:05 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/11 02:49:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 02:49:06 Context: Bucket flush, response code: 200 dump: 
2021/12/11 02:49:16 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/11 02:49:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCRLF (103.69s)
=== RUN   TestImportExport
2021/12/11 02:49:26 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/11 02:49:26 App: TestImportExport_n1ql_insert_on_update got un-deployed
2021/12/11 02:49:26 No eventing-con process running
2021/12/11 02:49:26 Function: TestImportExport_n1ql_insert_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestImportExport_n1ql_insert_on_update not found"
 }
}
2021/12/11 02:49:28 Context: Bucket flush, response code: 200 dump: 
2021/12/11 02:49:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 02:49:35 Context: Bucket flush, response code: 200 dump: 
2021/12/11 02:49:40 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 02:49:45 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/11 02:49:45 App: TestImportExport_bucket_op_on_update got un-deployed
2021/12/11 02:49:45 No eventing-con process running
2021/12/11 02:49:45 Function: TestImportExport_bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestImportExport_bucket_op_on_update not found"
 }
}
2021/12/11 02:49:47 Context: Bucket flush, response code: 200 dump: 
2021/12/11 02:49:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 02:49:53 Context: Bucket flush, response code: 200 dump: 
2021/12/11 02:49:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 02:49:58 Deploying app: TestImportExport_n1ql_insert_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/11 02:50:00 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestImportExport_n1ql_insert_on_update' in metakv",
  "warnings": null
 }
}
2021/12/11 02:50:00 Deploying app: TestImportExport_bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/11 02:50:01 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestImportExport_bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/11 02:50:06 Waiting for app: TestImportExport_n1ql_insert_on_update to get deployed
2021/12/11 02:50:11 Waiting for app: TestImportExport_n1ql_insert_on_update to get deployed
2021/12/11 02:50:16 Waiting for app: TestImportExport_n1ql_insert_on_update to get deployed
2021/12/11 02:50:21 Waiting for app: TestImportExport_n1ql_insert_on_update to get deployed
2021/12/11 02:50:21 App: TestImportExport_n1ql_insert_on_update got deployed
2021/12/11 02:50:26 Waiting for app: TestImportExport_bucket_op_on_update to get deployed
2021/12/11 02:50:31 Waiting for app: TestImportExport_bucket_op_on_update to get deployed
2021/12/11 02:50:36 Waiting for app: TestImportExport_bucket_op_on_update to get deployed
2021/12/11 02:50:36 App: TestImportExport_bucket_op_on_update got deployed
2021/12/11 02:50:36 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 02:50:41 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/11 02:50:46 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/11 02:50:51 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/11 02:50:56 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/11 02:50:56 App: TestImportExport_n1ql_insert_on_update got un-deployed
2021/12/11 02:50:56 No eventing-con process running
2021/12/11 02:50:56 Function: TestImportExport_n1ql_insert_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestImportExport_n1ql_insert_on_update deleting in the background"
}
2021/12/11 02:50:56 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 02:51:01 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/11 02:51:06 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/11 02:51:11 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/11 02:51:16 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/11 02:51:16 App: TestImportExport_bucket_op_on_update got un-deployed
2021/12/11 02:51:16 No eventing-con process running
2021/12/11 02:51:16 Function: TestImportExport_bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestImportExport_bucket_op_on_update deleting in the background"
}
2021/12/11 02:51:34 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/11 02:51:34 App: TestImportExport_bucket_op_on_update got un-deployed
2021/12/11 02:51:34 No eventing-con process running
2021/12/11 02:51:34 Function: TestImportExport_bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestImportExport_bucket_op_on_update deleting in the background"
}
2021/12/11 02:51:35 Context: Bucket flush, response code: 200 dump: 
2021/12/11 02:51:40 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 02:51:42 Context: Bucket flush, response code: 200 dump: 
2021/12/11 02:51:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 02:51:52 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/11 02:51:52 App: TestImportExport_n1ql_insert_on_update got un-deployed
2021/12/11 02:51:52 No eventing-con process running
2021/12/11 02:51:52 Function: TestImportExport_n1ql_insert_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestImportExport_n1ql_insert_on_update deleting in the background"
}
2021/12/11 02:51:54 Context: Bucket flush, response code: 200 dump: 
2021/12/11 02:51:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 02:52:01 Context: Bucket flush, response code: 200 dump: 
2021/12/11 02:52:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestImportExport (169.47s)
=== RUN   TestDeployUndeployLoopNonDefaultSettings
2021/12/11 02:52:16 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/11 02:52:16 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/11 02:52:16 No eventing-con process running
2021/12/11 02:52:16 Function: TestDeployUndeployLoopNonDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDeployUndeployLoopNonDefaultSettings not found"
 }
}
2021/12/11 02:52:18 Context: Bucket flush, response code: 200 dump: 
2021/12/11 02:52:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 02:52:24 Context: Bucket flush, response code: 200 dump: 
2021/12/11 02:52:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 02:52:29 Deploying app: TestDeployUndeployLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/11 02:52:30 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/11 02:52:30 Starting bucket ops to source bucket
2021/12/11 02:52:38 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 02:52:43 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 02:52:49 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 02:52:54 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 02:52:54 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/11 02:52:59 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get deployed
2021/12/11 02:52:59 App: TestDeployUndeployLoopNonDefaultSettings got deployed
2021/12/11 02:52:59 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 02:52:59 Undeploying app: bucket_op_on_update
2021/12/11 02:52:59 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 02:53:04 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/11 02:53:09 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/11 02:53:14 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/11 02:53:19 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/11 02:53:19 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/11 02:53:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 02:53:19 No eventing-con process running
2021/12/11 02:53:21 Context: Bucket flush, response code: 200 dump: 
2021/12/11 02:53:28 Context: Bucket flush, response code: 200 dump: 
2021/12/11 02:54:03 Deploying app: TestDeployUndeployLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/11 02:54:04 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/11 02:54:04 Starting bucket ops to source bucket
2021/12/11 02:54:12 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 02:54:17 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 02:54:22 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 02:54:27 Waiting for dst bucket item count to get to: 5000 curr count: 1983
2021/12/11 02:54:32 Waiting for dst bucket item count to get to: 5000 curr count: 4483
2021/12/11 02:54:32 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/11 02:54:37 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get deployed
2021/12/11 02:54:37 App: TestDeployUndeployLoopNonDefaultSettings got deployed
2021/12/11 02:54:37 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 02:54:37 Undeploying app: bucket_op_on_update
2021/12/11 02:54:37 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 02:54:42 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/11 02:54:47 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/11 02:54:52 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/11 02:54:57 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/11 02:54:57 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/11 02:54:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 02:54:57 No eventing-con process running
2021/12/11 02:54:59 Context: Bucket flush, response code: 200 dump: 
2021/12/11 02:55:06 Context: Bucket flush, response code: 200 dump: 
2021/12/11 02:55:41 Deploying app: TestDeployUndeployLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/11 02:55:42 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/11 02:55:42 Starting bucket ops to source bucket
2021/12/11 02:55:50 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 02:55:55 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 02:56:00 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 02:56:05 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 02:56:05 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/11 02:56:10 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get deployed
2021/12/11 02:56:10 App: TestDeployUndeployLoopNonDefaultSettings got deployed
2021/12/11 02:56:10 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 02:56:10 Undeploying app: bucket_op_on_update
2021/12/11 02:56:10 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 02:56:15 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/11 02:56:20 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/11 02:56:25 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/11 02:56:30 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/11 02:56:30 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/11 02:56:30 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 02:56:30 No eventing-con process running
2021/12/11 02:56:32 Context: Bucket flush, response code: 200 dump: 
2021/12/11 02:56:39 Context: Bucket flush, response code: 200 dump: 
2021/12/11 02:57:14 Deploying app: TestDeployUndeployLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/11 02:57:16 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/11 02:57:16 Starting bucket ops to source bucket
2021/12/11 02:57:23 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 02:57:28 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 02:57:34 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 02:57:39 Waiting for dst bucket item count to get to: 5000 curr count: 2492
2021/12/11 02:57:44 Waiting for dst bucket item count to get to: 5000 curr count: 2492
2021/12/11 02:57:44 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/11 02:57:49 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get deployed
2021/12/11 02:57:49 App: TestDeployUndeployLoopNonDefaultSettings got deployed
2021/12/11 02:57:49 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 02:57:49 Undeploying app: bucket_op_on_update
2021/12/11 02:57:49 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 02:57:54 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/11 02:57:59 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/11 02:58:04 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/11 02:58:09 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/11 02:58:09 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/11 02:58:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 02:58:09 No eventing-con process running
2021/12/11 02:58:10 Context: Bucket flush, response code: 200 dump: 
2021/12/11 02:58:18 Context: Bucket flush, response code: 200 dump: 
2021/12/11 02:58:53 Deploying app: TestDeployUndeployLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/11 02:58:54 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/11 02:58:54 Starting bucket ops to source bucket
2021/12/11 02:59:02 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 02:59:07 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 02:59:12 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 02:59:17 Waiting for dst bucket item count to get to: 5000 curr count: 2043
2021/12/11 02:59:22 Waiting for dst bucket item count to get to: 5000 curr count: 4543
2021/12/11 02:59:22 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/11 02:59:27 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get deployed
2021/12/11 02:59:27 App: TestDeployUndeployLoopNonDefaultSettings got deployed
2021/12/11 02:59:27 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 02:59:27 Undeploying app: bucket_op_on_update
2021/12/11 02:59:27 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 02:59:32 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/11 02:59:37 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/11 02:59:42 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/11 02:59:47 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/11 02:59:47 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/11 02:59:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 02:59:47 No eventing-con process running
2021/12/11 02:59:49 Context: Bucket flush, response code: 200 dump: 
2021/12/11 02:59:56 Context: Bucket flush, response code: 200 dump: 
2021/12/11 03:00:31 Function: TestDeployUndeployLoopNonDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDeployUndeployLoopNonDefaultSettings deleting in the background"
}
--- PASS: TestDeployUndeployLoopNonDefaultSettings (505.28s)
=== RUN   TestOnUpdateN1QLOp
2021/12/11 03:00:41 Waiting for app: TestOnUpdateN1QLOp to get un-deployed
2021/12/11 03:00:41 App: TestOnUpdateN1QLOp got un-deployed
2021/12/11 03:00:41 No eventing-con process running
2021/12/11 03:00:41 Function: TestOnUpdateN1QLOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateN1QLOp not found"
 }
}
2021/12/11 03:00:43 Context: Bucket flush, response code: 200 dump: 
2021/12/11 03:00:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 03:00:50 Context: Bucket flush, response code: 200 dump: 
2021/12/11 03:00:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 03:00:55 Deploying app: TestOnUpdateN1QLOp. Item count src bucket: 0 dst bucket: 0
2021/12/11 03:00:56 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateN1QLOp' in metakv",
  "warnings": null
 }
}
2021/12/11 03:00:56 Starting bucket ops to source bucket
2021/12/11 03:01:04 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 03:01:09 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 03:01:14 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 03:01:19 Waiting for dst bucket item count to get to: 5000 curr count: 914
2021/12/11 03:01:24 Waiting for dst bucket item count to get to: 5000 curr count: 914
2021/12/11 03:01:24 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/11 03:01:24 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 03:01:24 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 03:01:29 Waiting for app: TestOnUpdateN1QLOp to get un-deployed
2021/12/11 03:01:34 Waiting for app: TestOnUpdateN1QLOp to get un-deployed
2021/12/11 03:01:39 Waiting for app: TestOnUpdateN1QLOp to get un-deployed
2021/12/11 03:01:44 Waiting for app: TestOnUpdateN1QLOp to get un-deployed
2021/12/11 03:01:44 App: TestOnUpdateN1QLOp got un-deployed
2021/12/11 03:01:44 No eventing-con process running
2021/12/11 03:01:44 Function: TestOnUpdateN1QLOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateN1QLOp deleting in the background"
}
2021/12/11 03:01:46 Context: Bucket flush, response code: 200 dump: 
2021/12/11 03:01:56 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/11 03:01:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 03:01:58 Context: Bucket flush, response code: 200 dump: 
2021/12/11 03:02:08 Waiting for dst bucket item count to get to: 0 curr count: 2511
2021/12/11 03:02:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateN1QLOp (96.45s)
=== RUN   TestOnUpdateBucketOpDefaultSettings
2021/12/11 03:02:18 Waiting for app: TestOnUpdateBucketOpDefaultSettings to get un-deployed
2021/12/11 03:02:18 App: TestOnUpdateBucketOpDefaultSettings got un-deployed
2021/12/11 03:02:18 No eventing-con process running
2021/12/11 03:02:18 Function: TestOnUpdateBucketOpDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateBucketOpDefaultSettings not found"
 }
}
2021/12/11 03:02:19 Context: Bucket flush, response code: 200 dump: 
2021/12/11 03:02:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 03:02:26 Context: Bucket flush, response code: 200 dump: 
2021/12/11 03:02:31 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 03:02:31 Deploying app: TestOnUpdateBucketOpDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/11 03:02:32 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateBucketOpDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/11 03:02:32 Starting bucket ops to source bucket
2021/12/11 03:02:40 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 03:02:45 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 03:02:50 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 03:02:55 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 03:02:55 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/11 03:02:55 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 03:02:55 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 03:03:00 Waiting for app: TestOnUpdateBucketOpDefaultSettings to get un-deployed
2021/12/11 03:03:05 Waiting for app: TestOnUpdateBucketOpDefaultSettings to get un-deployed
2021/12/11 03:03:10 Waiting for app: TestOnUpdateBucketOpDefaultSettings to get un-deployed
2021/12/11 03:03:15 Waiting for app: TestOnUpdateBucketOpDefaultSettings to get un-deployed
2021/12/11 03:03:15 App: TestOnUpdateBucketOpDefaultSettings got un-deployed
2021/12/11 03:03:15 No eventing-con process running
2021/12/11 03:03:15 Function: TestOnUpdateBucketOpDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateBucketOpDefaultSettings deleting in the background"
}
2021/12/11 03:03:17 Context: Bucket flush, response code: 200 dump: 
2021/12/11 03:03:27 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/11 03:03:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 03:03:29 Context: Bucket flush, response code: 200 dump: 
2021/12/11 03:03:39 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/11 03:03:44 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/11 03:03:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateBucketOpDefaultSettings (96.40s)
=== RUN   TestOnUpdateBucketOpNonDefaultSettings
2021/12/11 03:03:54 Waiting for app: TestOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/11 03:03:54 App: TestOnUpdateBucketOpNonDefaultSettings got un-deployed
2021/12/11 03:03:54 No eventing-con process running
2021/12/11 03:03:54 Function: TestOnUpdateBucketOpNonDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateBucketOpNonDefaultSettings not found"
 }
}
2021/12/11 03:03:56 Context: Bucket flush, response code: 200 dump: 
2021/12/11 03:04:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 03:04:03 Context: Bucket flush, response code: 200 dump: 
2021/12/11 03:04:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 03:04:08 Deploying app: TestOnUpdateBucketOpNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/11 03:04:09 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateBucketOpNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/11 03:04:09 Starting bucket ops to source bucket
2021/12/11 03:04:17 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 03:04:22 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 03:04:27 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 03:04:32 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 03:04:37 Waiting for dst bucket item count to get to: 5000 curr count: 2500
2021/12/11 03:04:37 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/11 03:04:37 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 03:04:37 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 03:04:42 Waiting for app: TestOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/11 03:04:47 Waiting for app: TestOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/11 03:04:52 Waiting for app: TestOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/11 03:04:57 Waiting for app: TestOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/11 03:04:57 App: TestOnUpdateBucketOpNonDefaultSettings got un-deployed
2021/12/11 03:04:57 No eventing-con process running
2021/12/11 03:04:57 Function: TestOnUpdateBucketOpNonDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateBucketOpNonDefaultSettings deleting in the background"
}
2021/12/11 03:04:59 Context: Bucket flush, response code: 200 dump: 
2021/12/11 03:05:09 Waiting for dst bucket item count to get to: 0 curr count: 4872
2021/12/11 03:05:14 Waiting for dst bucket item count to get to: 0 curr count: 2372
2021/12/11 03:05:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 03:05:16 Context: Bucket flush, response code: 200 dump: 
2021/12/11 03:05:26 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/11 03:05:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateBucketOpNonDefaultSettings (101.69s)
=== RUN   TestOnUpdateBucketOpDefaultSettings10K
2021/12/11 03:05:36 Waiting for app: TestOnUpdateBucketOpDefaultSettings10K to get un-deployed
2021/12/11 03:05:36 App: TestOnUpdateBucketOpDefaultSettings10K got un-deployed
2021/12/11 03:05:36 No eventing-con process running
2021/12/11 03:05:36 Function: TestOnUpdateBucketOpDefaultSettings10K delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateBucketOpDefaultSettings10K not found"
 }
}
2021/12/11 03:05:37 Context: Bucket flush, response code: 200 dump: 
2021/12/11 03:05:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 03:05:44 Context: Bucket flush, response code: 200 dump: 
2021/12/11 03:05:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 03:05:49 Deploying app: TestOnUpdateBucketOpDefaultSettings10K. Item count src bucket: 0 dst bucket: 0
2021/12/11 03:05:50 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateBucketOpDefaultSettings10K' in metakv",
  "warnings": null
 }
}
2021/12/11 03:05:50 Starting bucket ops to source bucket
2021/12/11 03:05:58 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 03:06:03 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 03:06:08 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 03:06:13 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 03:06:13 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/11 03:06:13 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 03:06:13 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 03:06:18 Waiting for app: TestOnUpdateBucketOpDefaultSettings10K to get un-deployed
2021/12/11 03:06:24 Waiting for app: TestOnUpdateBucketOpDefaultSettings10K to get un-deployed
2021/12/11 03:06:29 Waiting for app: TestOnUpdateBucketOpDefaultSettings10K to get un-deployed
2021/12/11 03:06:34 Waiting for app: TestOnUpdateBucketOpDefaultSettings10K to get un-deployed
2021/12/11 03:06:34 App: TestOnUpdateBucketOpDefaultSettings10K got un-deployed
2021/12/11 03:06:34 No eventing-con process running
2021/12/11 03:06:34 Function: TestOnUpdateBucketOpDefaultSettings10K delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateBucketOpDefaultSettings10K deleting in the background"
}
2021/12/11 03:06:35 Context: Bucket flush, response code: 200 dump: 
2021/12/11 03:06:45 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/11 03:06:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 03:06:47 Context: Bucket flush, response code: 200 dump: 
2021/12/11 03:06:57 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/11 03:06:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateBucketOpDefaultSettings10K (91.54s)
=== RUN   TestOnUpdateBucketOpDefaultSettings100K
2021/12/11 03:07:07 Waiting for app: TestOnUpdateBucketOpDefaultSettings100K to get un-deployed
2021/12/11 03:07:07 App: TestOnUpdateBucketOpDefaultSettings100K got un-deployed
2021/12/11 03:07:07 No eventing-con process running
2021/12/11 03:07:07 Function: TestOnUpdateBucketOpDefaultSettings100K delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateBucketOpDefaultSettings100K not found"
 }
}
2021/12/11 03:07:09 Context: Bucket flush, response code: 200 dump: 
2021/12/11 03:07:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 03:07:16 Context: Bucket flush, response code: 200 dump: 
2021/12/11 03:07:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 03:07:21 Deploying app: TestOnUpdateBucketOpDefaultSettings100K. Item count src bucket: 0 dst bucket: 0
2021/12/11 03:07:22 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateBucketOpDefaultSettings100K' in metakv",
  "warnings": null
 }
}
2021/12/11 03:07:22 Starting bucket ops to source bucket
2021/12/11 03:07:30 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 03:07:35 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 03:07:40 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 03:07:45 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 03:07:45 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/11 03:07:45 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 03:07:45 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 03:07:50 Waiting for app: TestOnUpdateBucketOpDefaultSettings100K to get un-deployed
2021/12/11 03:07:55 Waiting for app: TestOnUpdateBucketOpDefaultSettings100K to get un-deployed
2021/12/11 03:08:01 Waiting for app: TestOnUpdateBucketOpDefaultSettings100K to get un-deployed
2021/12/11 03:08:06 Waiting for app: TestOnUpdateBucketOpDefaultSettings100K to get un-deployed
2021/12/11 03:08:06 App: TestOnUpdateBucketOpDefaultSettings100K got un-deployed
2021/12/11 03:08:06 No eventing-con process running
2021/12/11 03:08:06 Function: TestOnUpdateBucketOpDefaultSettings100K delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateBucketOpDefaultSettings100K deleting in the background"
}
2021/12/11 03:08:07 Context: Bucket flush, response code: 200 dump: 
2021/12/11 03:08:17 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/11 03:08:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 03:08:19 Context: Bucket flush, response code: 200 dump: 
2021/12/11 03:08:29 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/11 03:08:34 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/11 03:08:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateBucketOpDefaultSettings100K (96.93s)
=== RUN   TestOnDeleteBucketOp
2021/12/11 03:08:44 Waiting for app: TestOnDeleteBucketOp to get un-deployed
2021/12/11 03:08:44 App: TestOnDeleteBucketOp got un-deployed
2021/12/11 03:08:44 No eventing-con process running
2021/12/11 03:08:44 Function: TestOnDeleteBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnDeleteBucketOp not found"
 }
}
2021/12/11 03:08:46 Context: Bucket flush, response code: 200 dump: 
2021/12/11 03:08:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 03:08:53 Context: Bucket flush, response code: 200 dump: 
2021/12/11 03:08:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 03:08:58 Deploying app: TestOnDeleteBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/11 03:08:59 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnDeleteBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/11 03:08:59 Starting bucket ops to source bucket
2021/12/11 03:09:10 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 03:09:15 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 03:09:20 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 03:09:25 Waiting for dst bucket item count to get to: 5000 curr count: 2504
2021/12/11 03:09:25 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/11 03:09:25 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 03:09:25 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 03:09:30 Waiting for app: TestOnDeleteBucketOp to get un-deployed
2021/12/11 03:09:35 Waiting for app: TestOnDeleteBucketOp to get un-deployed
2021/12/11 03:09:40 Waiting for app: TestOnDeleteBucketOp to get un-deployed
2021/12/11 03:09:45 Waiting for app: TestOnDeleteBucketOp to get un-deployed
2021/12/11 03:09:45 App: TestOnDeleteBucketOp got un-deployed
2021/12/11 03:09:45 No eventing-con process running
2021/12/11 03:09:45 Function: TestOnDeleteBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnDeleteBucketOp deleting in the background"
}
2021/12/11 03:09:47 Context: Bucket flush, response code: 200 dump: 
2021/12/11 03:09:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 03:09:54 Context: Bucket flush, response code: 200 dump: 
2021/12/11 03:10:04 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/11 03:10:09 Waiting for dst bucket item count to get to: 0 curr count: 2496
2021/12/11 03:10:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnDeleteBucketOp (94.78s)
=== RUN   TestOnDeleteBucketOp5K
2021/12/11 03:10:19 Waiting for app: TestOnDeleteBucketOp5K to get un-deployed
2021/12/11 03:10:19 App: TestOnDeleteBucketOp5K got un-deployed
2021/12/11 03:10:19 No eventing-con process running
2021/12/11 03:10:19 Function: TestOnDeleteBucketOp5K delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnDeleteBucketOp5K not found"
 }
}
2021/12/11 03:10:21 Context: Bucket flush, response code: 200 dump: 
2021/12/11 03:10:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 03:10:27 Context: Bucket flush, response code: 200 dump: 
2021/12/11 03:10:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 03:10:32 Deploying app: TestOnDeleteBucketOp5K. Item count src bucket: 0 dst bucket: 0
2021/12/11 03:10:34 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnDeleteBucketOp5K' in metakv",
  "warnings": null
 }
}
2021/12/11 03:10:34 Starting bucket ops to source bucket
2021/12/11 03:10:44 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 03:10:49 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 03:10:54 Waiting for dst bucket item count to get to: 5000 curr count: 2243
2021/12/11 03:10:59 Waiting for dst bucket item count to get to: 5000 curr count: 4739
2021/12/11 03:10:59 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/11 03:10:59 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 03:11:00 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 03:11:05 Waiting for app: TestOnDeleteBucketOp5K to get un-deployed
2021/12/11 03:11:10 Waiting for app: TestOnDeleteBucketOp5K to get un-deployed
2021/12/11 03:11:15 Waiting for app: TestOnDeleteBucketOp5K to get un-deployed
2021/12/11 03:11:20 Waiting for app: TestOnDeleteBucketOp5K to get un-deployed
2021/12/11 03:11:20 App: TestOnDeleteBucketOp5K got un-deployed
2021/12/11 03:11:20 No eventing-con process running
2021/12/11 03:11:20 Function: TestOnDeleteBucketOp5K delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnDeleteBucketOp5K deleting in the background"
}
2021/12/11 03:11:21 Context: Bucket flush, response code: 200 dump: 
2021/12/11 03:11:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 03:11:29 Context: Bucket flush, response code: 200 dump: 
2021/12/11 03:11:39 Waiting for dst bucket item count to get to: 0 curr count: 4130
2021/12/11 03:11:44 Waiting for dst bucket item count to get to: 0 curr count: 1634
2021/12/11 03:11:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnDeleteBucketOp5K (94.76s)
=== RUN   TestTimerBucketOp
2021/12/11 03:11:54 Waiting for app: TestTimerBucketOp to get un-deployed
2021/12/11 03:11:54 App: TestTimerBucketOp got un-deployed
2021/12/11 03:11:54 No eventing-con process running
2021/12/11 03:11:54 Function: TestTimerBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestTimerBucketOp not found"
 }
}
2021/12/11 03:11:55 Context: Bucket flush, response code: 200 dump: 
2021/12/11 03:12:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 03:12:02 Context: Bucket flush, response code: 200 dump: 
2021/12/11 03:12:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 03:12:07 Deploying app: TestTimerBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/11 03:12:08 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestTimerBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/11 03:12:08 Starting bucket ops to source bucket
2021/12/11 03:12:16 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 03:12:21 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 03:12:26 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 03:12:31 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 03:12:36 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 03:12:41 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 03:12:46 Waiting for dst bucket item count to get to: 5000 curr count: 2492
2021/12/11 03:12:51 Waiting for dst bucket item count to get to: 5000 curr count: 4992
2021/12/11 03:12:51 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/11 03:12:51 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 03:12:51 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 03:12:56 Waiting for app: TestTimerBucketOp to get un-deployed
2021/12/11 03:13:01 Waiting for app: TestTimerBucketOp to get un-deployed
2021/12/11 03:13:06 Waiting for app: TestTimerBucketOp to get un-deployed
2021/12/11 03:13:11 Waiting for app: TestTimerBucketOp to get un-deployed
2021/12/11 03:13:11 App: TestTimerBucketOp got un-deployed
2021/12/11 03:13:11 No eventing-con process running
2021/12/11 03:13:11 Function: TestTimerBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestTimerBucketOp deleting in the background"
}
2021/12/11 03:13:13 Context: Bucket flush, response code: 200 dump: 
2021/12/11 03:13:23 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/11 03:13:28 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/11 03:13:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 03:13:30 Context: Bucket flush, response code: 200 dump: 
2021/12/11 03:13:40 Waiting for dst bucket item count to get to: 0 curr count: 3117
2021/12/11 03:13:45 Waiting for dst bucket item count to get to: 0 curr count: 3117
2021/12/11 03:13:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestTimerBucketOp (121.34s)
=== RUN   TestTimerInPastBucketOp
2021/12/11 03:13:55 Waiting for app: TestTimerInPastBucketOp to get un-deployed
2021/12/11 03:13:55 App: TestTimerInPastBucketOp got un-deployed
2021/12/11 03:13:55 No eventing-con process running
2021/12/11 03:13:55 Function: TestTimerInPastBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestTimerInPastBucketOp not found"
 }
}
2021/12/11 03:13:57 Context: Bucket flush, response code: 200 dump: 
2021/12/11 03:14:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 03:14:04 Context: Bucket flush, response code: 200 dump: 
2021/12/11 03:14:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 03:14:09 Deploying app: TestTimerInPastBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/11 03:14:10 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestTimerInPastBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/11 03:14:10 Starting bucket ops to source bucket
2021/12/11 03:14:18 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 03:14:23 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 03:14:28 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 03:14:33 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 03:14:38 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 03:14:43 Waiting for dst bucket item count to get to: 5000 curr count: 302
2021/12/11 03:14:48 Waiting for dst bucket item count to get to: 5000 curr count: 1484
2021/12/11 03:14:53 Waiting for dst bucket item count to get to: 5000 curr count: 3682
2021/12/11 03:14:53 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/11 03:14:53 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 03:14:53 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 03:14:58 Waiting for app: TestTimerInPastBucketOp to get un-deployed
2021/12/11 03:15:03 Waiting for app: TestTimerInPastBucketOp to get un-deployed
2021/12/11 03:15:08 Waiting for app: TestTimerInPastBucketOp to get un-deployed
2021/12/11 03:15:13 Waiting for app: TestTimerInPastBucketOp to get un-deployed
2021/12/11 03:15:13 App: TestTimerInPastBucketOp got un-deployed
2021/12/11 03:15:13 No eventing-con process running
2021/12/11 03:15:13 Function: TestTimerInPastBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestTimerInPastBucketOp deleting in the background"
}
2021/12/11 03:15:15 Context: Bucket flush, response code: 200 dump: 
2021/12/11 03:15:25 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/11 03:15:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 03:15:26 Context: Bucket flush, response code: 200 dump: 
2021/12/11 03:15:36 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/11 03:15:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestTimerInPastBucketOp (111.34s)
=== RUN   TestDeployUndeployLoopTimer
2021/12/11 03:15:46 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/11 03:15:46 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/11 03:15:46 No eventing-con process running
2021/12/11 03:15:46 Function: TestDeployUndeployLoopTimer delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDeployUndeployLoopTimer not found"
 }
}
2021/12/11 03:15:48 Context: Bucket flush, response code: 200 dump: 
2021/12/11 03:15:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 03:15:55 Context: Bucket flush, response code: 200 dump: 
2021/12/11 03:16:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 03:16:00 Deploying app: TestDeployUndeployLoopTimer. Item count src bucket: 0 dst bucket: 0
2021/12/11 03:16:01 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopTimer' in metakv",
  "warnings": null
 }
}
2021/12/11 03:16:01 Starting bucket ops to source bucket
2021/12/11 03:16:09 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 03:16:14 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 03:16:19 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 03:16:24 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 03:16:29 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 03:16:34 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 03:16:39 Waiting for dst bucket item count to get to: 5000 curr count: 3140
2021/12/11 03:16:44 Waiting for dst bucket item count to get to: 5000 curr count: 3140
2021/12/11 03:16:44 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/11 03:16:49 Waiting for app: TestDeployUndeployLoopTimer to get deployed
2021/12/11 03:16:49 App: TestDeployUndeployLoopTimer got deployed
2021/12/11 03:16:49 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 03:16:49 Undeploying app: bucket_op_with_timer
2021/12/11 03:16:49 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 03:16:54 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/11 03:16:59 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/11 03:17:04 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/11 03:17:09 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/11 03:17:09 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/11 03:17:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 03:17:09 No eventing-con process running
2021/12/11 03:17:11 Context: Bucket flush, response code: 200 dump: 
2021/12/11 03:17:18 Context: Bucket flush, response code: 200 dump: 
2021/12/11 03:17:53 Deploying app: TestDeployUndeployLoopTimer. Item count src bucket: 0 dst bucket: 0
2021/12/11 03:17:54 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopTimer' in metakv",
  "warnings": null
 }
}
2021/12/11 03:17:54 Starting bucket ops to source bucket
2021/12/11 03:18:02 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 03:18:07 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 03:18:12 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 03:18:17 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 03:18:22 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 03:18:27 Waiting for dst bucket item count to get to: 5000 curr count: 55
2021/12/11 03:18:32 Waiting for dst bucket item count to get to: 5000 curr count: 119
2021/12/11 03:18:37 Waiting for dst bucket item count to get to: 5000 curr count: 2564
2021/12/11 03:18:37 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/11 03:18:42 Waiting for app: TestDeployUndeployLoopTimer to get deployed
2021/12/11 03:18:42 App: TestDeployUndeployLoopTimer got deployed
2021/12/11 03:18:42 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 03:18:42 Undeploying app: bucket_op_with_timer
2021/12/11 03:18:42 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 03:18:47 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/11 03:18:53 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/11 03:18:58 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/11 03:19:03 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/11 03:19:03 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/11 03:19:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 03:19:03 No eventing-con process running
2021/12/11 03:19:04 Context: Bucket flush, response code: 200 dump: 
2021/12/11 03:19:12 Context: Bucket flush, response code: 200 dump: 
2021/12/11 03:19:47 Deploying app: TestDeployUndeployLoopTimer. Item count src bucket: 0 dst bucket: 0
2021/12/11 03:19:48 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopTimer' in metakv",
  "warnings": null
 }
}
2021/12/11 03:19:48 Starting bucket ops to source bucket
2021/12/11 03:19:56 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 03:20:01 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 03:20:06 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 03:20:11 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 03:20:16 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 03:20:21 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 03:20:26 Waiting for dst bucket item count to get to: 5000 curr count: 2500
2021/12/11 03:20:26 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/11 03:20:31 Waiting for app: TestDeployUndeployLoopTimer to get deployed
2021/12/11 03:20:31 App: TestDeployUndeployLoopTimer got deployed
2021/12/11 03:20:31 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 03:20:31 Undeploying app: bucket_op_with_timer
2021/12/11 03:20:31 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 03:20:36 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/11 03:20:41 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/11 03:20:46 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/11 03:20:51 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/11 03:20:51 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/11 03:20:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 03:20:51 No eventing-con process running
2021/12/11 03:20:53 Context: Bucket flush, response code: 200 dump: 
2021/12/11 03:21:00 Context: Bucket flush, response code: 200 dump: 
2021/12/11 03:21:35 Deploying app: TestDeployUndeployLoopTimer. Item count src bucket: 0 dst bucket: 0
2021/12/11 03:21:36 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopTimer' in metakv",
  "warnings": null
 }
}
2021/12/11 03:21:36 Starting bucket ops to source bucket
2021/12/11 03:21:44 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 03:21:49 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 03:21:54 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 03:21:59 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 03:22:04 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 03:22:09 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 03:22:14 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 03:22:19 Waiting for dst bucket item count to get to: 5000 curr count: 4161
2021/12/11 03:22:24 Waiting for dst bucket item count to get to: 5000 curr count: 4161
2021/12/11 03:22:24 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/11 03:22:29 Waiting for app: TestDeployUndeployLoopTimer to get deployed
2021/12/11 03:22:29 App: TestDeployUndeployLoopTimer got deployed
2021/12/11 03:22:29 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 03:22:29 Undeploying app: bucket_op_with_timer
2021/12/11 03:22:29 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 03:22:34 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/11 03:22:39 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/11 03:22:44 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/11 03:22:49 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/11 03:22:49 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/11 03:22:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 03:22:49 No eventing-con process running
2021/12/11 03:22:51 Context: Bucket flush, response code: 200 dump: 
2021/12/11 03:22:58 Context: Bucket flush, response code: 200 dump: 
2021/12/11 03:23:33 Deploying app: TestDeployUndeployLoopTimer. Item count src bucket: 0 dst bucket: 0
2021/12/11 03:23:34 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopTimer' in metakv",
  "warnings": null
 }
}
2021/12/11 03:23:34 Starting bucket ops to source bucket
2021/12/11 03:23:42 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 03:23:47 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 03:23:52 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 03:23:57 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 03:24:02 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 03:24:07 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 03:24:12 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 03:24:17 Waiting for dst bucket item count to get to: 5000 curr count: 2500
2021/12/11 03:24:17 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/11 03:24:22 Waiting for app: TestDeployUndeployLoopTimer to get deployed
2021/12/11 03:24:22 App: TestDeployUndeployLoopTimer got deployed
2021/12/11 03:24:22 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 03:24:22 Undeploying app: bucket_op_with_timer
2021/12/11 03:24:22 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 03:24:27 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/11 03:24:32 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/11 03:24:37 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/11 03:24:42 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/11 03:24:42 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/11 03:24:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 03:24:42 No eventing-con process running
2021/12/11 03:24:44 Context: Bucket flush, response code: 200 dump: 
2021/12/11 03:24:51 Context: Bucket flush, response code: 200 dump: 
2021/12/11 03:25:26 Function: TestDeployUndeployLoopTimer delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDeployUndeployLoopTimer deleting in the background"
}
--- PASS: TestDeployUndeployLoopTimer (590.01s)
=== RUN   TestMultipleHandlers
2021/12/11 03:25:36 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get un-deployed
2021/12/11 03:25:36 App: TestMultipleHandlers_bucket_op_on_update got un-deployed
2021/12/11 03:25:36 No eventing-con process running
2021/12/11 03:25:36 Function: TestMultipleHandlers_bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMultipleHandlers_bucket_op_on_update not found"
 }
}
2021/12/11 03:25:38 Context: Bucket flush, response code: 200 dump: 
2021/12/11 03:25:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 03:25:45 Context: Bucket flush, response code: 200 dump: 
2021/12/11 03:25:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 03:25:55 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get un-deployed
2021/12/11 03:25:55 App: TestMultipleHandlers_bucket_op_on_delete got un-deployed
2021/12/11 03:25:55 No eventing-con process running
2021/12/11 03:25:55 Function: TestMultipleHandlers_bucket_op_on_delete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMultipleHandlers_bucket_op_on_delete not found"
 }
}
2021/12/11 03:25:57 Context: Bucket flush, response code: 200 dump: 
2021/12/11 03:26:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 03:26:03 Context: Bucket flush, response code: 200 dump: 
2021/12/11 03:26:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 03:26:09 Deploying app: TestMultipleHandlers_bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/11 03:26:10 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMultipleHandlers_bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/11 03:26:10 Deploying app: TestMultipleHandlers_bucket_op_on_delete. Item count src bucket: 0 dst bucket: 0
2021/12/11 03:26:11 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMultipleHandlers_bucket_op_on_delete' in metakv",
  "warnings": null
 }
}
2021/12/11 03:26:16 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get deployed
2021/12/11 03:26:21 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get deployed
2021/12/11 03:26:26 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get deployed
2021/12/11 03:26:31 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get deployed
2021/12/11 03:26:31 App: TestMultipleHandlers_bucket_op_on_update got deployed
2021/12/11 03:26:36 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get deployed
2021/12/11 03:26:41 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get deployed
2021/12/11 03:26:46 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get deployed
2021/12/11 03:26:46 App: TestMultipleHandlers_bucket_op_on_delete got deployed
2021/12/11 03:26:46 Starting bucket ops to source bucket
2021/12/11 03:26:56 Waiting for dst bucket item count to get to: 5000 curr count: 634
2021/12/11 03:27:01 Waiting for dst bucket item count to get to: 5000 curr count: 2362
2021/12/11 03:27:06 Waiting for dst bucket item count to get to: 5000 curr count: 4228
2021/12/11 03:27:06 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/11 03:27:06 Starting bucket ops to source bucket
2021/12/11 03:27:21 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/11 03:27:21 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/11 03:27:21 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 03:27:21 Function: TestMultipleHandlers_bucket_op_on_update update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/11 03:27:21 Function: TestMultipleHandlers_bucket_op_on_delete update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/11 03:27:21 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 03:27:26 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get un-deployed
2021/12/11 03:27:31 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get un-deployed
2021/12/11 03:27:36 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get un-deployed
2021/12/11 03:27:41 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get un-deployed
2021/12/11 03:27:41 App: TestMultipleHandlers_bucket_op_on_update got un-deployed
2021/12/11 03:27:41 No eventing-con process running
2021/12/11 03:27:41 Function: TestMultipleHandlers_bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMultipleHandlers_bucket_op_on_update deleting in the background"
}
2021/12/11 03:27:43 Context: Bucket flush, response code: 200 dump: 
2021/12/11 03:27:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 03:27:50 Context: Bucket flush, response code: 200 dump: 
2021/12/11 03:28:00 Waiting for dst bucket item count to get to: 0 curr count: 5684
2021/12/11 03:28:05 Waiting for dst bucket item count to get to: 0 curr count: 5684
2021/12/11 03:28:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 03:28:05 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 03:28:10 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get un-deployed
2021/12/11 03:28:15 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get un-deployed
2021/12/11 03:28:20 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get un-deployed
2021/12/11 03:28:25 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get un-deployed
2021/12/11 03:28:25 App: TestMultipleHandlers_bucket_op_on_delete got un-deployed
2021/12/11 03:28:25 No eventing-con process running
2021/12/11 03:28:25 Function: TestMultipleHandlers_bucket_op_on_delete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMultipleHandlers_bucket_op_on_delete deleting in the background"
}
2021/12/11 03:28:27 Context: Bucket flush, response code: 200 dump: 
2021/12/11 03:28:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 03:28:34 Context: Bucket flush, response code: 200 dump: 
2021/12/11 03:28:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestMultipleHandlers (192.52s)
=== RUN   TestPauseResumeLoopDefaultSettings
2021/12/11 03:28:49 Waiting for app: TestPauseResumeLoopDefaultSettings to get un-deployed
2021/12/11 03:28:49 App: TestPauseResumeLoopDefaultSettings got un-deployed
2021/12/11 03:28:49 No eventing-con process running
2021/12/11 03:28:49 Function: TestPauseResumeLoopDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestPauseResumeLoopDefaultSettings not found"
 }
}
2021/12/11 03:28:51 Context: Bucket flush, response code: 200 dump: 
2021/12/11 03:28:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 03:28:57 Context: Bucket flush, response code: 200 dump: 
2021/12/11 03:29:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 03:29:02 Deploying app: TestPauseResumeLoopDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/11 03:29:04 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestPauseResumeLoopDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/11 03:29:04 Starting bucket ops to source bucket
2021/12/11 03:29:12 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 03:29:17 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 03:29:22 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 03:29:27 Waiting for dst bucket item count to get to: 5000 curr count: 2275
2021/12/11 03:29:32 Waiting for dst bucket item count to get to: 5000 curr count: 4775
2021/12/11 03:29:32 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/11 03:29:32 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 03:29:37 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to deployed
2021/12/11 03:29:37 Function: TestPauseResumeLoopDefaultSettings status changed to deployed
2021/12/11 03:29:37 Pausing app: bucket_op_on_update
2021/12/11 03:29:37 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/11 03:29:42 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to paused
2021/12/11 03:29:42 Function: TestPauseResumeLoopDefaultSettings status changed to paused
2021/12/11 03:29:42 Resuming app: bucket_op_on_update
2021/12/11 03:29:43 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/11 03:29:43 Starting bucket ops to source bucket
2021/12/11 03:29:51 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/11 03:29:56 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/11 03:30:01 Waiting for dst bucket item count to get to: 10000 curr count: 5619
2021/12/11 03:30:06 Waiting for dst bucket item count to get to: 10000 curr count: 8119
2021/12/11 03:30:06 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/11 03:30:06 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 03:30:11 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to deployed
2021/12/11 03:30:11 Function: TestPauseResumeLoopDefaultSettings status changed to deployed
2021/12/11 03:30:11 Pausing app: bucket_op_on_update
2021/12/11 03:30:11 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/11 03:30:16 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to paused
2021/12/11 03:30:16 Function: TestPauseResumeLoopDefaultSettings status changed to paused
2021/12/11 03:30:16 Resuming app: bucket_op_on_update
2021/12/11 03:30:17 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/11 03:30:17 Starting bucket ops to source bucket
2021/12/11 03:30:25 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/11 03:30:30 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/11 03:30:35 Waiting for dst bucket item count to get to: 15000 curr count: 12500
2021/12/11 03:30:35 src & dst bucket item count matched up. src bucket count: 15000 dst bucket count: 15000
2021/12/11 03:30:35 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 03:30:40 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to deployed
2021/12/11 03:30:40 Function: TestPauseResumeLoopDefaultSettings status changed to deployed
2021/12/11 03:30:40 Pausing app: bucket_op_on_update
2021/12/11 03:30:40 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/11 03:30:45 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to paused
2021/12/11 03:30:45 Function: TestPauseResumeLoopDefaultSettings status changed to paused
2021/12/11 03:30:45 Resuming app: bucket_op_on_update
2021/12/11 03:30:46 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/11 03:30:46 Starting bucket ops to source bucket
2021/12/11 03:30:54 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/11 03:30:59 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/11 03:31:04 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/11 03:31:09 Waiting for dst bucket item count to get to: 20000 curr count: 17500
2021/12/11 03:31:09 src & dst bucket item count matched up. src bucket count: 20000 dst bucket count: 20000
2021/12/11 03:31:09 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 03:31:14 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to deployed
2021/12/11 03:31:14 Function: TestPauseResumeLoopDefaultSettings status changed to deployed
2021/12/11 03:31:14 Pausing app: bucket_op_on_update
2021/12/11 03:31:14 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/11 03:31:19 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to paused
2021/12/11 03:31:19 Function: TestPauseResumeLoopDefaultSettings status changed to paused
2021/12/11 03:31:19 Resuming app: bucket_op_on_update
2021/12/11 03:31:21 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/11 03:31:21 Starting bucket ops to source bucket
2021/12/11 03:31:28 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/11 03:31:33 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/11 03:31:38 Waiting for dst bucket item count to get to: 25000 curr count: 22297
2021/12/11 03:31:43 Waiting for dst bucket item count to get to: 25000 curr count: 22297
2021/12/11 03:31:43 src & dst bucket item count matched up. src bucket count: 25000 dst bucket count: 25000
2021/12/11 03:31:43 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 03:31:48 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to deployed
2021/12/11 03:31:48 Function: TestPauseResumeLoopDefaultSettings status changed to deployed
2021/12/11 03:31:48 Pausing app: bucket_op_on_update
2021/12/11 03:31:49 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/11 03:31:54 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to paused
2021/12/11 03:31:54 Function: TestPauseResumeLoopDefaultSettings status changed to paused
2021/12/11 03:31:54 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 03:31:59 Waiting for app: TestPauseResumeLoopDefaultSettings to get un-deployed
2021/12/11 03:32:04 Waiting for app: TestPauseResumeLoopDefaultSettings to get un-deployed
2021/12/11 03:32:09 Waiting for app: TestPauseResumeLoopDefaultSettings to get un-deployed
2021/12/11 03:32:14 Waiting for app: TestPauseResumeLoopDefaultSettings to get un-deployed
2021/12/11 03:32:14 App: TestPauseResumeLoopDefaultSettings got un-deployed
2021/12/11 03:32:14 No eventing-con process running
2021/12/11 03:32:14 Function: TestPauseResumeLoopDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestPauseResumeLoopDefaultSettings deleting in the background"
}
2021/12/11 03:32:15 Context: Bucket flush, response code: 200 dump: 
2021/12/11 03:32:25 Waiting for dst bucket item count to get to: 0 curr count: 25000
2021/12/11 03:32:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 03:32:27 Context: Bucket flush, response code: 200 dump: 
2021/12/11 03:32:37 Waiting for dst bucket item count to get to: 0 curr count: 25000
2021/12/11 03:32:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestPauseResumeLoopDefaultSettings (238.39s)
=== RUN   TestPauseResumeLoopNonDefaultSettings
2021/12/11 03:32:47 Waiting for app: TestPauseResumeLoopNonDefaultSettings to get un-deployed
2021/12/11 03:32:47 App: TestPauseResumeLoopNonDefaultSettings got un-deployed
2021/12/11 03:32:47 No eventing-con process running
2021/12/11 03:32:47 Function: TestPauseResumeLoopNonDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestPauseResumeLoopNonDefaultSettings not found"
 }
}
2021/12/11 03:32:49 Context: Bucket flush, response code: 200 dump: 
2021/12/11 03:32:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 03:32:56 Context: Bucket flush, response code: 200 dump: 
2021/12/11 03:33:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 03:33:01 Deploying app: TestPauseResumeLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/11 03:33:02 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestPauseResumeLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/11 03:33:02 Starting bucket ops to source bucket
2021/12/11 03:33:10 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 03:33:15 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 03:33:20 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 03:33:25 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 03:33:30 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 03:33:35 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 03:33:40 Waiting for dst bucket item count to get to: 5000 curr count: 2257
2021/12/11 03:33:45 Waiting for dst bucket item count to get to: 5000 curr count: 2257
2021/12/11 03:33:45 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/11 03:33:45 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 03:33:50 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to deployed
2021/12/11 03:33:50 Function: TestPauseResumeLoopNonDefaultSettings status changed to deployed
2021/12/11 03:33:50 Pausing app: bucket_op_with_timer
2021/12/11 03:33:50 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/11 03:33:55 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to paused
2021/12/11 03:33:55 Function: TestPauseResumeLoopNonDefaultSettings status changed to paused
2021/12/11 03:33:55 Resuming app: bucket_op_with_timer
2021/12/11 03:33:56 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/11 03:33:56 Starting bucket ops to source bucket
2021/12/11 03:34:04 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/11 03:34:09 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/11 03:34:14 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/11 03:34:19 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/11 03:34:24 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/11 03:34:29 Waiting for dst bucket item count to get to: 10000 curr count: 7952
2021/12/11 03:34:34 Waiting for dst bucket item count to get to: 10000 curr count: 7952
2021/12/11 03:34:34 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/11 03:34:34 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 03:34:39 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to deployed
2021/12/11 03:34:39 Function: TestPauseResumeLoopNonDefaultSettings status changed to deployed
2021/12/11 03:34:39 Pausing app: bucket_op_with_timer
2021/12/11 03:34:39 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/11 03:34:44 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to paused
2021/12/11 03:34:44 Function: TestPauseResumeLoopNonDefaultSettings status changed to paused
2021/12/11 03:34:44 Resuming app: bucket_op_with_timer
2021/12/11 03:34:46 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/11 03:34:46 Starting bucket ops to source bucket
2021/12/11 03:34:53 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/11 03:34:58 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/11 03:35:03 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/11 03:35:08 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/11 03:35:14 Waiting for dst bucket item count to get to: 15000 curr count: 11476
2021/12/11 03:35:19 Waiting for dst bucket item count to get to: 15000 curr count: 13976
2021/12/11 03:35:19 src & dst bucket item count matched up. src bucket count: 15000 dst bucket count: 15000
2021/12/11 03:35:19 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 03:35:24 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to deployed
2021/12/11 03:35:24 Function: TestPauseResumeLoopNonDefaultSettings status changed to deployed
2021/12/11 03:35:24 Pausing app: bucket_op_with_timer
2021/12/11 03:35:24 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/11 03:35:29 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to paused
2021/12/11 03:35:29 Function: TestPauseResumeLoopNonDefaultSettings status changed to paused
2021/12/11 03:35:29 Resuming app: bucket_op_with_timer
2021/12/11 03:35:30 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/11 03:35:30 Starting bucket ops to source bucket
2021/12/11 03:35:38 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/11 03:35:43 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/11 03:35:48 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/11 03:35:53 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/11 03:35:58 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/11 03:36:03 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/11 03:36:08 Waiting for dst bucket item count to get to: 20000 curr count: 17500
2021/12/11 03:36:08 src & dst bucket item count matched up. src bucket count: 20000 dst bucket count: 20000
2021/12/11 03:36:08 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 03:36:13 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to deployed
2021/12/11 03:36:13 Function: TestPauseResumeLoopNonDefaultSettings status changed to deployed
2021/12/11 03:36:13 Pausing app: bucket_op_with_timer
2021/12/11 03:36:13 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/11 03:36:18 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to paused
2021/12/11 03:36:18 Function: TestPauseResumeLoopNonDefaultSettings status changed to paused
2021/12/11 03:36:18 Resuming app: bucket_op_with_timer
2021/12/11 03:36:19 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/11 03:36:19 Starting bucket ops to source bucket
2021/12/11 03:36:27 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/11 03:36:32 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/11 03:36:37 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/11 03:36:42 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/11 03:36:47 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/11 03:36:52 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/11 03:36:57 Waiting for dst bucket item count to get to: 25000 curr count: 22500
2021/12/11 03:36:57 src & dst bucket item count matched up. src bucket count: 25000 dst bucket count: 25000
2021/12/11 03:36:57 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 03:37:02 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to deployed
2021/12/11 03:37:02 Function: TestPauseResumeLoopNonDefaultSettings status changed to deployed
2021/12/11 03:37:02 Pausing app: bucket_op_with_timer
2021/12/11 03:37:02 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/11 03:37:07 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to paused
2021/12/11 03:37:07 Function: TestPauseResumeLoopNonDefaultSettings status changed to paused
2021/12/11 03:37:07 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 03:37:12 Waiting for app: TestPauseResumeLoopNonDefaultSettings to get un-deployed
2021/12/11 03:37:17 Waiting for app: TestPauseResumeLoopNonDefaultSettings to get un-deployed
2021/12/11 03:37:22 Waiting for app: TestPauseResumeLoopNonDefaultSettings to get un-deployed
2021/12/11 03:37:28 Waiting for app: TestPauseResumeLoopNonDefaultSettings to get un-deployed
2021/12/11 03:37:28 App: TestPauseResumeLoopNonDefaultSettings got un-deployed
2021/12/11 03:37:28 No eventing-con process running
2021/12/11 03:37:28 Function: TestPauseResumeLoopNonDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestPauseResumeLoopNonDefaultSettings deleting in the background"
}
2021/12/11 03:37:29 Context: Bucket flush, response code: 200 dump: 
2021/12/11 03:37:39 Waiting for dst bucket item count to get to: 0 curr count: 25000
2021/12/11 03:37:44 Waiting for dst bucket item count to get to: 0 curr count: 12500
2021/12/11 03:37:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 03:37:46 Context: Bucket flush, response code: 200 dump: 
2021/12/11 03:37:56 Waiting for dst bucket item count to get to: 0 curr count: 25000
2021/12/11 03:37:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestPauseResumeLoopNonDefaultSettings (318.79s)
=== RUN   TestPauseAndResumeWithWorkerCountChange
2021/12/11 03:38:06 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get un-deployed
2021/12/11 03:38:06 App: TestPauseAndResumeWithWorkerCountChange got un-deployed
2021/12/11 03:38:06 No eventing-con process running
2021/12/11 03:38:06 Function: TestPauseAndResumeWithWorkerCountChange delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestPauseAndResumeWithWorkerCountChange not found"
 }
}
2021/12/11 03:38:08 Context: Bucket flush, response code: 200 dump: 
2021/12/11 03:38:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 03:38:15 Context: Bucket flush, response code: 200 dump: 
2021/12/11 03:38:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 03:38:20 Deploying app: TestPauseAndResumeWithWorkerCountChange. Item count src bucket: 0 dst bucket: 0
2021/12/11 03:38:21 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestPauseAndResumeWithWorkerCountChange' in metakv",
  "warnings": null
 }
}
2021/12/11 03:38:26 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get deployed
2021/12/11 03:38:31 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get deployed
2021/12/11 03:38:36 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get deployed
2021/12/11 03:38:41 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get deployed
2021/12/11 03:38:41 App: TestPauseAndResumeWithWorkerCountChange got deployed
2021/12/11 03:38:41 Starting bucket ops to source bucket
2021/12/11 03:38:50 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 03:38:55 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 03:39:00 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 03:39:05 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 03:39:10 Waiting for dst bucket item count to get to: 5000 curr count: 4386
2021/12/11 03:39:15 Waiting for dst bucket item count to get to: 5000 curr count: 4386
2021/12/11 03:39:15 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/11 03:39:15 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 03:39:15 Pausing app: bucket_op_with_timer
2021/12/11 03:39:15 Function: TestPauseAndResumeWithWorkerCountChange update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/11 03:39:20 Waiting for function: TestPauseAndResumeWithWorkerCountChange status to change to paused
2021/12/11 03:39:20 Function: TestPauseAndResumeWithWorkerCountChange status changed to paused
2021/12/11 03:39:20 Resuming app: bucket_op_with_timer
2021/12/11 03:39:21 Function: TestPauseAndResumeWithWorkerCountChange update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:6] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/11 03:39:21 Starting bucket ops to source bucket
2021/12/11 03:39:33 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/11 03:39:38 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/11 03:39:43 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/11 03:39:48 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/11 03:39:53 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/11 03:39:53 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/11 03:39:53 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 03:39:58 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get un-deployed
2021/12/11 03:40:03 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get un-deployed
2021/12/11 03:40:08 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get un-deployed
2021/12/11 03:40:13 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get un-deployed
2021/12/11 03:40:13 App: TestPauseAndResumeWithWorkerCountChange got un-deployed
2021/12/11 03:40:13 No eventing-con process running
2021/12/11 03:40:13 Function: TestPauseAndResumeWithWorkerCountChange delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestPauseAndResumeWithWorkerCountChange deleting in the background"
}
2021/12/11 03:40:15 Context: Bucket flush, response code: 200 dump: 
2021/12/11 03:40:25 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/11 03:40:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 03:40:26 Context: Bucket flush, response code: 200 dump: 
2021/12/11 03:40:36 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/11 03:40:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestPauseAndResumeWithWorkerCountChange (160.39s)
=== RUN   TestPauseResumeWithEventingReb
2021/12/11 03:40:46 Waiting for app: TestPauseResumeWithEventingReb to get un-deployed
2021/12/11 03:40:46 App: TestPauseResumeWithEventingReb got un-deployed
2021/12/11 03:40:46 No eventing-con process running
2021/12/11 03:40:46 Function: TestPauseResumeWithEventingReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestPauseResumeWithEventingReb not found"
 }
}
2021/12/11 03:40:48 Context: Bucket flush, response code: 200 dump: 
2021/12/11 03:40:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 03:40:55 Context: Bucket flush, response code: 200 dump: 
2021/12/11 03:41:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 03:41:00 Deploying app: TestPauseResumeWithEventingReb. Item count src bucket: 0 dst bucket: 0
2021/12/11 03:41:01 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestPauseResumeWithEventingReb' in metakv",
  "warnings": null
 }
}
2021/12/11 03:41:06 Waiting for app: TestPauseResumeWithEventingReb to get deployed
2021/12/11 03:41:11 Waiting for app: TestPauseResumeWithEventingReb to get deployed
2021/12/11 03:41:16 Waiting for app: TestPauseResumeWithEventingReb to get deployed
2021/12/11 03:41:21 Waiting for app: TestPauseResumeWithEventingReb to get deployed
2021/12/11 03:41:21 App: TestPauseResumeWithEventingReb got deployed
2021/12/11 03:41:21 Starting bucket ops to source bucket
2021/12/11 03:41:30 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 03:41:35 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 03:41:40 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 03:41:45 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 03:41:45 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/11 03:41:45 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 03:41:45 Pausing app: bucket_op_with_timer
2021/12/11 03:41:45 Function: TestPauseResumeWithEventingReb update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/11 03:41:50 Waiting for function: TestPauseResumeWithEventingReb status to change to paused
2021/12/11 03:41:50 Function: TestPauseResumeWithEventingReb status changed to paused
2021/12/11 03:41:50 Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/11 03:41:51 Removing node(s): [] from the cluster
2021/12/11 03:41:56 Rebalance progress: 29.16666666666666
2021/12/11 03:42:01 Rebalance progress: 29.16666666666666
2021/12/11 03:42:06 Rebalance progress: 100
2021/12/11 03:42:06 Resuming app: bucket_op_with_timer
2021/12/11 03:42:07 Function: TestPauseResumeWithEventingReb update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:6] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/11 03:42:07 Starting bucket ops to source bucket
2021/12/11 03:42:19 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/11 03:42:24 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/11 03:42:29 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/11 03:42:34 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/11 03:42:39 Waiting for dst bucket item count to get to: 10000 curr count: 5736
2021/12/11 03:42:44 Waiting for dst bucket item count to get to: 10000 curr count: 5736
2021/12/11 03:42:44 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/11 03:42:44 Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/11 03:42:49 Rebalance progress: 29.16666666666666
2021/12/11 03:42:54 Rebalance progress: 29.16666666666666
2021/12/11 03:42:59 Rebalance progress: 100

VB distribution across eventing nodes and their workers::

2021/12/11 03:43:04 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:60} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/11 03:43:19 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:60} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/11 03:43:34 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:60} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/11 03:43:49 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:60} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/11 03:44:04 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:60} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/11 03:44:19 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:60} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/11 03:44:34 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:60} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/11 03:44:49 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:60} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/11 03:45:04 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:60} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/11 03:45:19 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:60} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/11 03:45:34 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:60} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/11 03:45:49 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:60} {Name:#primary Bucket:hello-world Completion:100}]

DCP Stream statuses:
	stream status: running
	len: 1024
	vb list dump: [0-1023]

vbucket curr owner:
Producer node: 127.0.0.1:9302	Node UUID: ff39ea7c3d7f9701d93e7c48208f6b74
	workerID: worker_TestPauseResumeWithEventingReb_4
	len: 57
	v: [228-284]
	workerID: worker_TestPauseResumeWithEventingReb_5
	len: 57
	v: [285-341]
	workerID: worker_TestPauseResumeWithEventingReb_0
	len: 57
	v: [0-56]
	workerID: worker_TestPauseResumeWithEventingReb_1
	len: 57
	v: [57-113]
	workerID: worker_TestPauseResumeWithEventingReb_2
	len: 57
	v: [114-170]
	workerID: worker_TestPauseResumeWithEventingReb_3
	len: 57
	v: [171-227]
Producer node: 172.31.2.134:9300	Node UUID: 8cb6f622ccb07b97de85c9bedb4434c2
	workerID: worker_TestPauseResumeWithEventingReb_4
	len: 57
	v: [911-967]
	workerID: worker_TestPauseResumeWithEventingReb_5
	len: 56
	v: [968-1023]
	workerID: worker_TestPauseResumeWithEventingReb_0
	len: 57
	v: [683-739]
	workerID: worker_TestPauseResumeWithEventingReb_1
	len: 57
	v: [740-796]
	workerID: worker_TestPauseResumeWithEventingReb_2
	len: 57
	v: [797-853]
	workerID: worker_TestPauseResumeWithEventingReb_3
	len: 57
	v: [854-910]
Producer node: 127.0.0.1:9303	Node UUID: 46adfc38c81b9f50741786c7f6948222
	workerID: worker_TestPauseResumeWithEventingReb_0
	len: 57
	v: [342-398]
	workerID: worker_TestPauseResumeWithEventingReb_1
	len: 57
	v: [399-455]
	workerID: worker_TestPauseResumeWithEventingReb_2
	len: 57
	v: [456-512]
	workerID: worker_TestPauseResumeWithEventingReb_3
	len: 57
	v: [513-569]
	workerID: worker_TestPauseResumeWithEventingReb_4
	len: 57
	v: [570-626]
	workerID: worker_TestPauseResumeWithEventingReb_5
	len: 56
	v: [627-682]


2021/12/11 03:46:04 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 03:46:04 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 03:46:09 clusterSetup.go:397: Waiting for app: TestPauseResumeWithEventingReb to get un-deployed
2021/12/11 03:46:14 clusterSetup.go:397: Waiting for app: TestPauseResumeWithEventingReb to get un-deployed
2021/12/11 03:46:19 clusterSetup.go:397: Waiting for app: TestPauseResumeWithEventingReb to get un-deployed
2021/12/11 03:46:24 clusterSetup.go:397: Waiting for app: TestPauseResumeWithEventingReb to get un-deployed
2021/12/11 03:46:24 clusterSetup.go:405: App: TestPauseResumeWithEventingReb got un-deployed
2021/12/11 03:46:24 eventingSetup.go:46: No eventing-con process running
2021/12/11 03:46:24 eventingSetup.go:418: Function: TestPauseResumeWithEventingReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestPauseResumeWithEventingReb deleting in the background"
}
2021/12/11 03:46:26 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/11 03:46:36 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/11 03:46:36 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 03:46:38 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/11 03:46:48 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6329
2021/12/11 03:46:53 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1329
2021/12/11 03:46:53 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestPauseResumeWithEventingReb (376.38s)
=== RUN   TestBucketDeleteAfterPause
2021/12/11 03:47:03 clusterSetup.go:397: Waiting for app: TestBucketDeleteAfterPause to get un-deployed
2021/12/11 03:47:03 clusterSetup.go:405: App: TestBucketDeleteAfterPause got un-deployed
2021/12/11 03:47:03 eventingSetup.go:46: No eventing-con process running
2021/12/11 03:47:03 eventingSetup.go:418: Function: TestBucketDeleteAfterPause delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestBucketDeleteAfterPause not found"
 }
}
2021/12/11 03:47:05 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/11 03:47:10 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 03:47:11 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/11 03:47:16 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 03:47:16 eventingSetup.go:125: Deploying app: TestBucketDeleteAfterPause. Item count src bucket: 0 dst bucket: 0
2021/12/11 03:47:18 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestBucketDeleteAfterPause' in metakv",
  "warnings": null
 }
}
2021/12/11 03:47:23 clusterSetup.go:346: Waiting for app: TestBucketDeleteAfterPause to get deployed
2021/12/11 03:47:28 clusterSetup.go:346: Waiting for app: TestBucketDeleteAfterPause to get deployed
2021/12/11 03:47:33 clusterSetup.go:346: Waiting for app: TestBucketDeleteAfterPause to get deployed
2021/12/11 03:47:38 clusterSetup.go:346: Waiting for app: TestBucketDeleteAfterPause to get deployed
2021/12/11 03:47:38 clusterSetup.go:354: App: TestBucketDeleteAfterPause got deployed
2021/12/11 03:47:38 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/11 03:47:47 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 903
2021/12/11 03:47:52 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 903
2021/12/11 03:47:57 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 903
2021/12/11 03:47:57 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/11 03:47:57 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 03:47:57 handler_test.go:712: Pausing app: bucket_op_on_update
2021/12/11 03:47:57 eventingSetup.go:270: Function: TestBucketDeleteAfterPause update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/11 03:48:02 eventingSetup.go:719: Waiting for function: TestBucketDeleteAfterPause status to change to paused
2021/12/11 03:48:02 eventingSetup.go:727: Function: TestBucketDeleteAfterPause status changed to paused
2021/12/11 03:48:02 handler_test.go:716: Deleting source bucket: default
2021/12/11 03:48:09 eventingSetup.go:719: Waiting for function: TestBucketDeleteAfterPause status to change to undeployed
2021/12/11 03:48:09 eventingSetup.go:727: Function: TestBucketDeleteAfterPause status changed to undeployed
2021/12/11 03:48:14 clusterSetup.go:397: Waiting for app: TestBucketDeleteAfterPause to get un-deployed
2021/12/11 03:48:14 clusterSetup.go:405: App: TestBucketDeleteAfterPause got un-deployed
2021/12/11 03:48:14 eventingSetup.go:46: No eventing-con process running
2021/12/11 03:48:14 eventingSetup.go:418: Function: TestBucketDeleteAfterPause delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestBucketDeleteAfterPause deleting in the background"
}
2021/12/11 03:48:15 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/11 03:48:25 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/11 03:48:25 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 03:48:27 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/11 03:48:37 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/11 03:48:37 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestBucketDeleteAfterPause (104.25s)
=== RUN   TestChangeFnCodeBetweenPauseResume
2021/12/11 03:48:47 clusterSetup.go:397: Waiting for app: TestChangeFnCodeBetweenPauseResume to get un-deployed
2021/12/11 03:48:47 clusterSetup.go:405: App: TestChangeFnCodeBetweenPauseResume got un-deployed
2021/12/11 03:48:47 eventingSetup.go:46: No eventing-con process running
2021/12/11 03:48:47 eventingSetup.go:418: Function: TestChangeFnCodeBetweenPauseResume delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestChangeFnCodeBetweenPauseResume not found"
 }
}
2021/12/11 03:48:48 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/11 03:48:53 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 03:48:55 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/11 03:49:00 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 03:49:00 eventingSetup.go:125: Deploying app: TestChangeFnCodeBetweenPauseResume. Item count src bucket: 0 dst bucket: 0
2021/12/11 03:49:01 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestChangeFnCodeBetweenPauseResume' in metakv",
  "warnings": null
 }
}
2021/12/11 03:49:06 clusterSetup.go:346: Waiting for app: TestChangeFnCodeBetweenPauseResume to get deployed
2021/12/11 03:49:11 clusterSetup.go:346: Waiting for app: TestChangeFnCodeBetweenPauseResume to get deployed
2021/12/11 03:49:16 clusterSetup.go:346: Waiting for app: TestChangeFnCodeBetweenPauseResume to get deployed
2021/12/11 03:49:21 clusterSetup.go:346: Waiting for app: TestChangeFnCodeBetweenPauseResume to get deployed
2021/12/11 03:49:21 clusterSetup.go:354: App: TestChangeFnCodeBetweenPauseResume got deployed
2021/12/11 03:49:21 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/11 03:49:56 handler_test.go:739: Pausing function: TestChangeFnCodeBetweenPauseResume
2021/12/11 03:49:56 eventingSetup.go:270: Function: TestChangeFnCodeBetweenPauseResume update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/11 03:50:01 eventingSetup.go:719: Waiting for function: TestChangeFnCodeBetweenPauseResume status to change to paused
2021/12/11 03:50:01 eventingSetup.go:727: Function: TestChangeFnCodeBetweenPauseResume status changed to paused
2021/12/11 03:53:01 handler_test.go:748: Resuming function: TestChangeFnCodeBetweenPauseResume
2021/12/11 03:53:01 eventingSetup.go:125: Deploying app: TestChangeFnCodeBetweenPauseResume. Item count src bucket: 5000 dst bucket: 0
2021/12/11 03:53:03 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestChangeFnCodeBetweenPauseResume' in metakv",
  "warnings": null
 }
}
2021/12/11 03:53:08 eventingSetup.go:795: Waiting for function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter to get to 5000
2021/12/11 03:53:08 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 03:53:08 eventingSetup.go:799: Function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter got to 0 expected 5000
2021/12/11 03:53:13 eventingSetup.go:795: Waiting for function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter to get to 5000
2021/12/11 03:53:13 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 03:53:13 eventingSetup.go:799: Function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter got to 2452 expected 5000
2021/12/11 03:53:18 eventingSetup.go:795: Waiting for function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter to get to 5000
2021/12/11 03:53:18 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 03:53:18 eventingSetup.go:799: Function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter got to 3126 expected 5000
2021/12/11 03:53:23 eventingSetup.go:795: Waiting for function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter to get to 5000
2021/12/11 03:53:23 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 03:53:23 eventingSetup.go:804: Function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter got to 5000
2021/12/11 03:53:23 handler_test.go:753: Undeploying function: TestChangeFnCodeBetweenPauseResume
2021/12/11 03:53:23 eventingSetup.go:270: Function: TestChangeFnCodeBetweenPauseResume update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/11 03:53:33 clusterSetup.go:397: Waiting for app: TestChangeFnCodeBetweenPauseResume to get un-deployed
2021/12/11 03:53:38 clusterSetup.go:397: Waiting for app: TestChangeFnCodeBetweenPauseResume to get un-deployed
2021/12/11 03:53:43 clusterSetup.go:397: Waiting for app: TestChangeFnCodeBetweenPauseResume to get un-deployed
2021/12/11 03:53:43 clusterSetup.go:405: App: TestChangeFnCodeBetweenPauseResume got un-deployed
2021/12/11 03:53:43 eventingSetup.go:46: No eventing-con process running
2021/12/11 03:53:43 eventingSetup.go:418: Function: TestChangeFnCodeBetweenPauseResume delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestChangeFnCodeBetweenPauseResume deleting in the background"
}
2021/12/11 03:53:44 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/11 03:53:54 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/11 03:53:59 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/11 03:53:59 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 03:54:01 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/11 03:54:06 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestChangeFnCodeBetweenPauseResume (328.83s)
=== RUN   TestCommentUnCommentOnDelete
2021/12/11 03:54:16 clusterSetup.go:397: Waiting for app: TestCommentUnCommentOnDelete to get un-deployed
2021/12/11 03:54:16 clusterSetup.go:405: App: TestCommentUnCommentOnDelete got un-deployed
2021/12/11 03:54:16 eventingSetup.go:46: No eventing-con process running
2021/12/11 03:54:16 eventingSetup.go:418: Function: TestCommentUnCommentOnDelete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestCommentUnCommentOnDelete not found"
 }
}
2021/12/11 03:54:17 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/11 03:54:22 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 03:54:24 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/11 03:54:29 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 03:54:29 eventingSetup.go:125: Deploying app: TestCommentUnCommentOnDelete. Item count src bucket: 0 dst bucket: 0
2021/12/11 03:54:30 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCommentUnCommentOnDelete' in metakv",
  "warnings": null
 }
}
2021/12/11 03:54:35 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/11 03:54:40 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/11 03:54:45 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/11 03:54:50 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/11 03:54:50 clusterSetup.go:354: App: TestCommentUnCommentOnDelete got deployed
2021/12/11 03:54:50 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/11 03:54:59 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 03:55:04 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 03:55:04 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/11 03:55:04 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 03:55:04 handler_test.go:780: Undeploying app: TestCommentUnCommentOnDelete
2021/12/11 03:55:04 eventingSetup.go:270: Function: TestCommentUnCommentOnDelete update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/11 03:55:34 eventingSetup.go:125: Deploying app: TestCommentUnCommentOnDelete. Item count src bucket: 5000 dst bucket: 5000
2021/12/11 03:55:35 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCommentUnCommentOnDelete' in metakv",
  "warnings": null
 }
}
2021/12/11 03:55:40 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/11 03:55:45 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/11 03:55:50 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/11 03:55:55 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/11 03:55:55 clusterSetup.go:354: App: TestCommentUnCommentOnDelete got deployed
2021/12/11 03:55:55 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/11 03:56:08 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/11 03:56:13 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/11 03:56:13 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 03:56:13 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 03:56:13 handler_test.go:799: Undeploying app: TestCommentUnCommentOnDelete
2021/12/11 03:56:13 eventingSetup.go:270: Function: TestCommentUnCommentOnDelete update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/11 03:56:23 clusterSetup.go:397: Waiting for app: TestCommentUnCommentOnDelete to get un-deployed
2021/12/11 03:56:28 clusterSetup.go:397: Waiting for app: TestCommentUnCommentOnDelete to get un-deployed
2021/12/11 03:56:33 clusterSetup.go:397: Waiting for app: TestCommentUnCommentOnDelete to get un-deployed
2021/12/11 03:56:33 clusterSetup.go:405: App: TestCommentUnCommentOnDelete got un-deployed
2021/12/11 03:56:33 eventingSetup.go:46: No eventing-con process running
2021/12/11 03:56:33 eventingSetup.go:418: Function: TestCommentUnCommentOnDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCommentUnCommentOnDelete deleting in the background"
}
2021/12/11 03:56:35 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/11 03:56:40 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 03:56:41 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/11 03:56:46 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCommentUnCommentOnDelete (160.44s)
=== RUN   TestCPPWorkerCleanup
2021/12/11 03:56:56 clusterSetup.go:397: Waiting for app: TestCPPWorkerCleanup to get un-deployed
2021/12/11 03:56:56 clusterSetup.go:405: App: TestCPPWorkerCleanup got un-deployed
2021/12/11 03:56:56 eventingSetup.go:46: No eventing-con process running
2021/12/11 03:56:56 eventingSetup.go:418: Function: TestCPPWorkerCleanup delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestCPPWorkerCleanup not found"
 }
}
2021/12/11 03:56:58 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/11 03:57:03 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 03:57:04 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/11 03:57:09 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 03:57:09 eventingSetup.go:125: Deploying app: TestCPPWorkerCleanup. Item count src bucket: 0 dst bucket: 0
2021/12/11 03:57:11 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCPPWorkerCleanup' in metakv",
  "warnings": null
 }
}
2021/12/11 03:57:11 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/11 03:57:18 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 03:57:23 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 03:57:28 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 03:57:33 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 03:57:38 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 4405
2021/12/11 03:57:43 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 4405
2021/12/11 03:57:43 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/11 03:57:43 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 03:57:43 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 03:57:48 clusterSetup.go:397: Waiting for app: TestCPPWorkerCleanup to get un-deployed
2021/12/11 03:57:53 clusterSetup.go:397: Waiting for app: TestCPPWorkerCleanup to get un-deployed
2021/12/11 03:57:58 clusterSetup.go:397: Waiting for app: TestCPPWorkerCleanup to get un-deployed
2021/12/11 03:58:03 clusterSetup.go:397: Waiting for app: TestCPPWorkerCleanup to get un-deployed
2021/12/11 03:58:03 clusterSetup.go:405: App: TestCPPWorkerCleanup got un-deployed
2021/12/11 03:58:03 eventingSetup.go:46: No eventing-con process running
2021/12/11 03:58:03 eventingSetup.go:418: Function: TestCPPWorkerCleanup delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCPPWorkerCleanup deleting in the background"
}
2021/12/11 03:58:04 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/11 03:58:14 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/11 03:58:19 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/11 03:58:19 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 03:58:21 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/11 03:58:31 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4948
2021/12/11 03:58:36 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4948
2021/12/11 03:58:36 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCPPWorkerCleanup (139.63s)
=== RUN   TestWithUserXattrs
2021/12/11 03:59:16 clusterSetup.go:397: Waiting for app: TestWithUserXattrs to get un-deployed
2021/12/11 03:59:16 clusterSetup.go:405: App: TestWithUserXattrs got un-deployed
2021/12/11 03:59:16 eventingSetup.go:46: No eventing-con process running
2021/12/11 03:59:16 eventingSetup.go:418: Function: TestWithUserXattrs delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestWithUserXattrs not found"
 }
}
2021/12/11 03:59:17 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/11 03:59:22 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 03:59:24 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/11 03:59:29 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 03:59:29 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/11 03:59:31 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/11 03:59:33 eventingSetup.go:125: Deploying app: TestWithUserXattrs. Item count src bucket: 0 dst bucket: 0
2021/12/11 03:59:34 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestWithUserXattrs' in metakv",
  "warnings": null
 }
}
2021/12/11 03:59:39 clusterSetup.go:346: Waiting for app: TestWithUserXattrs to get deployed
2021/12/11 03:59:44 clusterSetup.go:346: Waiting for app: TestWithUserXattrs to get deployed
2021/12/11 03:59:49 clusterSetup.go:346: Waiting for app: TestWithUserXattrs to get deployed
2021/12/11 03:59:54 clusterSetup.go:346: Waiting for app: TestWithUserXattrs to get deployed
2021/12/11 03:59:54 clusterSetup.go:354: App: TestWithUserXattrs got deployed
2021/12/11 03:59:54 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/11 04:00:04 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 04:00:09 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 4129
2021/12/11 04:00:14 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 4129
2021/12/11 04:00:14 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/11 04:00:14 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 04:00:14 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 04:00:19 clusterSetup.go:397: Waiting for app: TestWithUserXattrs to get un-deployed
2021/12/11 04:00:24 clusterSetup.go:397: Waiting for app: TestWithUserXattrs to get un-deployed
2021/12/11 04:00:29 clusterSetup.go:397: Waiting for app: TestWithUserXattrs to get un-deployed
2021/12/11 04:00:34 clusterSetup.go:397: Waiting for app: TestWithUserXattrs to get un-deployed
2021/12/11 04:00:34 clusterSetup.go:405: App: TestWithUserXattrs got un-deployed
2021/12/11 04:00:34 eventingSetup.go:46: No eventing-con process running
2021/12/11 04:00:34 eventingSetup.go:418: Function: TestWithUserXattrs delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestWithUserXattrs deleting in the background"
}
2021/12/11 04:00:35 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/11 04:00:45 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/11 04:00:45 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 04:00:47 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/11 04:00:57 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/11 04:00:57 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestWithUserXattrs (110.90s)
=== RUN   TestEventProcessingPostBucketFlush
2021/12/11 04:01:07 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/11 04:01:07 clusterSetup.go:405: App: TestEventProcessingPostBucketFlush got un-deployed
2021/12/11 04:01:07 eventingSetup.go:46: No eventing-con process running
2021/12/11 04:01:07 eventingSetup.go:418: Function: TestEventProcessingPostBucketFlush delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventProcessingPostBucketFlush not found"
 }
}
2021/12/11 04:01:08 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/11 04:01:13 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 04:01:15 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/11 04:01:20 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 04:01:20 eventingSetup.go:125: Deploying app: TestEventProcessingPostBucketFlush. Item count src bucket: 0 dst bucket: 0
2021/12/11 04:01:21 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventProcessingPostBucketFlush' in metakv",
  "warnings": null
 }
}
2021/12/11 04:01:26 clusterSetup.go:346: Waiting for app: TestEventProcessingPostBucketFlush to get deployed
2021/12/11 04:01:31 clusterSetup.go:346: Waiting for app: TestEventProcessingPostBucketFlush to get deployed
2021/12/11 04:01:36 clusterSetup.go:346: Waiting for app: TestEventProcessingPostBucketFlush to get deployed
2021/12/11 04:01:41 clusterSetup.go:346: Waiting for app: TestEventProcessingPostBucketFlush to get deployed
2021/12/11 04:01:41 clusterSetup.go:354: App: TestEventProcessingPostBucketFlush got deployed
2021/12/11 04:01:41 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/11 04:01:50 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 04:01:55 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 04:01:55 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/11 04:01:55 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 04:01:59 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/11 04:02:09 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3261
2021/12/11 04:02:14 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 761
2021/12/11 04:02:14 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 04:02:16 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/11 04:02:26 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/11 04:02:26 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 04:02:26 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/11 04:02:51 eventingSetup.go:441: Waiting for dst bucket item count to get to: 25000 curr count: 14948
2021/12/11 04:02:56 eventingSetup.go:441: Waiting for dst bucket item count to get to: 25000 curr count: 14948
2021/12/11 04:02:56 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 25000 dst bucket count: 25000
2021/12/11 04:02:56 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 04:02:56 eventingSetup.go:270: Function: TestEventProcessingPostBucketFlush update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/11 04:03:01 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/11 04:03:06 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/11 04:03:11 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/11 04:03:16 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/11 04:03:16 clusterSetup.go:405: App: TestEventProcessingPostBucketFlush got un-deployed
2021/12/11 04:03:16 eventingSetup.go:46: No eventing-con process running
2021/12/11 04:03:21 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/11 04:03:21 clusterSetup.go:405: App: TestEventProcessingPostBucketFlush got un-deployed
2021/12/11 04:03:21 eventingSetup.go:46: No eventing-con process running
2021/12/11 04:03:21 eventingSetup.go:418: Function: TestEventProcessingPostBucketFlush delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventProcessingPostBucketFlush deleting in the background"
}
2021/12/11 04:03:23 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/11 04:03:33 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 25000
2021/12/11 04:03:38 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12500
2021/12/11 04:03:38 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 04:03:39 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/11 04:03:50 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12500
2021/12/11 04:03:55 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12500
2021/12/11 04:03:55 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventProcessingPostBucketFlush (177.64s)
=== RUN   TestMetaBucketDelete
2021/12/11 04:04:05 clusterSetup.go:397: Waiting for app: TestMetaBucketDelete to get un-deployed
2021/12/11 04:04:05 clusterSetup.go:405: App: TestMetaBucketDelete got un-deployed
2021/12/11 04:04:05 eventingSetup.go:46: No eventing-con process running
2021/12/11 04:04:05 eventingSetup.go:418: Function: TestMetaBucketDelete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMetaBucketDelete not found"
 }
}
2021/12/11 04:04:06 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/11 04:04:11 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 04:04:13 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/11 04:04:18 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 04:04:18 eventingSetup.go:125: Deploying app: TestMetaBucketDelete. Item count src bucket: 0 dst bucket: 0
2021/12/11 04:04:19 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMetaBucketDelete' in metakv",
  "warnings": null
 }
}
2021/12/11 04:04:24 clusterSetup.go:346: Waiting for app: TestMetaBucketDelete to get deployed
2021/12/11 04:04:29 clusterSetup.go:346: Waiting for app: TestMetaBucketDelete to get deployed
2021/12/11 04:04:34 clusterSetup.go:346: Waiting for app: TestMetaBucketDelete to get deployed
2021/12/11 04:04:39 clusterSetup.go:346: Waiting for app: TestMetaBucketDelete to get deployed
2021/12/11 04:04:39 clusterSetup.go:354: App: TestMetaBucketDelete got deployed
2021/12/11 04:04:39 handler_test.go:902: Deleting metadata bucket: eventing
2021/12/11 04:04:40 handler_test.go:904: Deleted metadata bucket: eventing
2021/12/11 04:04:45 clusterSetup.go:397: Waiting for app: TestMetaBucketDelete to get un-deployed
2021/12/11 04:04:45 clusterSetup.go:405: App: TestMetaBucketDelete got un-deployed
2021/12/11 04:05:00 clusterSetup.go:397: Waiting for app: TestMetaBucketDelete to get un-deployed
2021/12/11 04:05:00 clusterSetup.go:405: App: TestMetaBucketDelete got un-deployed
2021/12/11 04:05:00 eventingSetup.go:46: No eventing-con process running
2021/12/11 04:05:00 eventingSetup.go:418: Function: TestMetaBucketDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMetaBucketDelete deleting in the background"
}
2021/12/11 04:05:01 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/11 04:05:06 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 04:05:08 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/11 04:05:13 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestMetaBucketDelete (78.63s)
=== RUN   TestMetaBucketDeleteWithBootstrap
2021/12/11 04:05:23 clusterSetup.go:397: Waiting for app: TestMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/11 04:05:23 clusterSetup.go:405: App: TestMetaBucketDeleteWithBootstrap got un-deployed
2021/12/11 04:05:23 eventingSetup.go:46: No eventing-con process running
2021/12/11 04:05:23 eventingSetup.go:418: Function: TestMetaBucketDeleteWithBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMetaBucketDeleteWithBootstrap not found"
 }
}
2021/12/11 04:05:24 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/11 04:05:29 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 04:05:31 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/11 04:05:36 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 04:05:36 eventingSetup.go:125: Deploying app: TestMetaBucketDeleteWithBootstrap. Item count src bucket: 0 dst bucket: 0
2021/12/11 04:05:37 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMetaBucketDeleteWithBootstrap' in metakv",
  "warnings": null
 }
}
2021/12/11 04:05:56 handler_test.go:923: Deleting metadata bucket: eventing
2021/12/11 04:05:57 handler_test.go:925: Deleted metadata bucket: eventing
2021/12/11 04:06:12 clusterSetup.go:397: Waiting for app: TestMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/11 04:06:12 clusterSetup.go:405: App: TestMetaBucketDeleteWithBootstrap got un-deployed
2021/12/11 04:06:27 clusterSetup.go:397: Waiting for app: TestMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/11 04:06:27 clusterSetup.go:405: App: TestMetaBucketDeleteWithBootstrap got un-deployed
2021/12/11 04:06:27 eventingSetup.go:46: No eventing-con process running
2021/12/11 04:06:27 eventingSetup.go:418: Function: TestMetaBucketDeleteWithBootstrap delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMetaBucketDeleteWithBootstrap deleting in the background"
}
2021/12/11 04:06:28 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/11 04:06:33 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 04:06:35 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/11 04:06:40 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestMetaBucketDeleteWithBootstrap (86.69s)
=== RUN   TestSourceBucketDelete
2021/12/11 04:06:50 clusterSetup.go:397: Waiting for app: TestSourceBucketDelete to get un-deployed
2021/12/11 04:06:50 clusterSetup.go:405: App: TestSourceBucketDelete got un-deployed
2021/12/11 04:06:50 eventingSetup.go:46: No eventing-con process running
2021/12/11 04:06:50 eventingSetup.go:418: Function: TestSourceBucketDelete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestSourceBucketDelete not found"
 }
}
2021/12/11 04:06:51 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/11 04:06:56 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 04:06:58 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/11 04:07:03 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 04:07:03 eventingSetup.go:125: Deploying app: TestSourceBucketDelete. Item count src bucket: 0 dst bucket: 0
2021/12/11 04:07:04 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestSourceBucketDelete' in metakv",
  "warnings": null
 }
}
2021/12/11 04:07:09 clusterSetup.go:346: Waiting for app: TestSourceBucketDelete to get deployed
2021/12/11 04:07:14 clusterSetup.go:346: Waiting for app: TestSourceBucketDelete to get deployed
2021/12/11 04:07:19 clusterSetup.go:346: Waiting for app: TestSourceBucketDelete to get deployed
2021/12/11 04:07:24 clusterSetup.go:346: Waiting for app: TestSourceBucketDelete to get deployed
2021/12/11 04:07:24 clusterSetup.go:354: App: TestSourceBucketDelete got deployed
2021/12/11 04:07:24 handler_test.go:945: Deleting source bucket: default
2021/12/11 04:07:26 handler_test.go:947: Deleted source bucket: default
2021/12/11 04:07:31 clusterSetup.go:397: Waiting for app: TestSourceBucketDelete to get un-deployed
2021/12/11 04:07:31 clusterSetup.go:405: App: TestSourceBucketDelete got un-deployed
2021/12/11 04:07:46 clusterSetup.go:397: Waiting for app: TestSourceBucketDelete to get un-deployed
2021/12/11 04:07:46 clusterSetup.go:405: App: TestSourceBucketDelete got un-deployed
2021/12/11 04:07:46 eventingSetup.go:46: No eventing-con process running
2021/12/11 04:07:46 eventingSetup.go:418: Function: TestSourceBucketDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestSourceBucketDelete deleting in the background"
}
2021/12/11 04:07:47 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/11 04:07:52 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 04:07:54 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/11 04:07:59 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestSourceBucketDelete (78.98s)
=== RUN   TestSourceBucketDeleteWithBootstrap
2021/12/11 04:08:09 clusterSetup.go:397: Waiting for app: TestSourceBucketDeleteWithBootstrap to get un-deployed
2021/12/11 04:08:09 clusterSetup.go:405: App: TestSourceBucketDeleteWithBootstrap got un-deployed
2021/12/11 04:08:09 eventingSetup.go:46: No eventing-con process running
2021/12/11 04:08:09 eventingSetup.go:418: Function: TestSourceBucketDeleteWithBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestSourceBucketDeleteWithBootstrap not found"
 }
}
2021/12/11 04:08:10 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/11 04:08:15 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 04:08:17 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/11 04:08:22 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 04:08:22 eventingSetup.go:125: Deploying app: TestSourceBucketDeleteWithBootstrap. Item count src bucket: 0 dst bucket: 0
2021/12/11 04:08:23 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestSourceBucketDeleteWithBootstrap' in metakv",
  "warnings": null
 }
}
2021/12/11 04:08:42 handler_test.go:966: Deleting source bucket: default
2021/12/11 04:08:43 handler_test.go:968: Deleted source bucket: default
2021/12/11 04:08:58 clusterSetup.go:397: Waiting for app: TestSourceBucketDeleteWithBootstrap to get un-deployed
2021/12/11 04:08:58 clusterSetup.go:405: App: TestSourceBucketDeleteWithBootstrap got un-deployed
2021/12/11 04:09:13 clusterSetup.go:397: Waiting for app: TestSourceBucketDeleteWithBootstrap to get un-deployed
2021/12/11 04:09:13 clusterSetup.go:405: App: TestSourceBucketDeleteWithBootstrap got un-deployed
2021/12/11 04:09:13 eventingSetup.go:46: No eventing-con process running
2021/12/11 04:09:13 eventingSetup.go:418: Function: TestSourceBucketDeleteWithBootstrap delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestSourceBucketDeleteWithBootstrap deleting in the background"
}
2021/12/11 04:09:15 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/11 04:09:20 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 04:09:21 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/11 04:09:26 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestSourceBucketDeleteWithBootstrap (87.55s)
=== RUN   TestSourceAndMetaBucketDeleteWithBootstrap
2021/12/11 04:09:36 clusterSetup.go:397: Waiting for app: TestSourceAndMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/11 04:09:36 clusterSetup.go:405: App: TestSourceAndMetaBucketDeleteWithBootstrap got un-deployed
2021/12/11 04:09:36 eventingSetup.go:46: No eventing-con process running
2021/12/11 04:09:36 eventingSetup.go:418: Function: TestSourceAndMetaBucketDeleteWithBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestSourceAndMetaBucketDeleteWithBootstrap not found"
 }
}
2021/12/11 04:09:38 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/11 04:09:43 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 04:09:44 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/11 04:09:49 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 04:09:49 eventingSetup.go:125: Deploying app: TestSourceAndMetaBucketDeleteWithBootstrap. Item count src bucket: 0 dst bucket: 0
2021/12/11 04:09:51 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestSourceAndMetaBucketDeleteWithBootstrap' in metakv",
  "warnings": null
 }
}
2021/12/11 04:10:09 handler_test.go:988: Deleting source bucket: default
2021/12/11 04:10:11 handler_test.go:990: Deleted source bucket: default
2021/12/11 04:10:11 handler_test.go:992: Deleting metadata bucket: eventing
2021/12/11 04:10:12 handler_test.go:994: Deleted metadata bucket: eventing
2021/12/11 04:10:27 clusterSetup.go:397: Waiting for app: TestSourceAndMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/11 04:10:27 clusterSetup.go:405: App: TestSourceAndMetaBucketDeleteWithBootstrap got un-deployed
2021/12/11 04:10:42 clusterSetup.go:397: Waiting for app: TestSourceAndMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/11 04:10:42 clusterSetup.go:405: App: TestSourceAndMetaBucketDeleteWithBootstrap got un-deployed
2021/12/11 04:10:42 eventingSetup.go:46: No eventing-con process running
2021/12/11 04:10:42 eventingSetup.go:418: Function: TestSourceAndMetaBucketDeleteWithBootstrap delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestSourceAndMetaBucketDeleteWithBootstrap deleting in the background"
}
2021/12/11 04:10:43 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/11 04:10:48 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 04:10:50 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/11 04:10:55 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestSourceAndMetaBucketDeleteWithBootstrap (88.32s)
=== RUN   TestUndeployDuringBootstrap
2021/12/11 04:11:05 clusterSetup.go:397: Waiting for app: TestUndeployDuringBootstrap to get un-deployed
2021/12/11 04:11:05 clusterSetup.go:405: App: TestUndeployDuringBootstrap got un-deployed
2021/12/11 04:11:05 eventingSetup.go:46: No eventing-con process running
2021/12/11 04:11:05 eventingSetup.go:418: Function: TestUndeployDuringBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestUndeployDuringBootstrap not found"
 }
}
2021/12/11 04:11:06 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/11 04:11:11 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 04:11:13 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/11 04:11:18 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 04:11:18 eventingSetup.go:125: Deploying app: TestUndeployDuringBootstrap. Item count src bucket: 0 dst bucket: 0
2021/12/11 04:11:19 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestUndeployDuringBootstrap' in metakv",
  "warnings": null
 }
}
2021/12/11 04:11:19 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/11 04:11:31 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 04:11:31 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 04:11:31 clusterSetup.go:375: Apps undergoing bootstrap: true
2021/12/11 04:11:31 clusterSetup.go:378: App: TestUndeployDuringBootstrap started bootstrap
2021/12/11 04:11:31 clusterSetup.go:375: Apps undergoing bootstrap: true
2021/12/11 04:11:36 clusterSetup.go:367: No apps undergoing bootstrap
2021/12/11 04:11:41 clusterSetup.go:397: Waiting for app: TestUndeployDuringBootstrap to get un-deployed
2021/12/11 04:11:46 clusterSetup.go:397: Waiting for app: TestUndeployDuringBootstrap to get un-deployed
2021/12/11 04:11:51 clusterSetup.go:397: Waiting for app: TestUndeployDuringBootstrap to get un-deployed
2021/12/11 04:11:51 clusterSetup.go:405: App: TestUndeployDuringBootstrap got un-deployed
2021/12/11 04:11:51 eventingSetup.go:46: No eventing-con process running
2021/12/11 04:12:16 clusterSetup.go:397: Waiting for app: TestUndeployDuringBootstrap to get un-deployed
2021/12/11 04:12:16 clusterSetup.go:405: App: TestUndeployDuringBootstrap got un-deployed
2021/12/11 04:12:16 eventingSetup.go:46: No eventing-con process running
2021/12/11 04:12:16 eventingSetup.go:418: Function: TestUndeployDuringBootstrap delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestUndeployDuringBootstrap deleting in the background"
}
2021/12/11 04:12:17 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/11 04:12:27 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/11 04:12:27 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 04:12:29 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/11 04:12:39 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2709
2021/12/11 04:12:44 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 222
2021/12/11 04:12:44 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestUndeployDuringBootstrap (109.34s)
=== RUN   TestDeleteBeforeUndeploy
2021/12/11 04:12:49 eventingSetup.go:125: Deploying app: TestDeleteBeforeUndeploy. Item count src bucket: 0 dst bucket: 0
2021/12/11 04:12:50 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeleteBeforeUndeploy' in metakv",
  "warnings": null
 }
}
2021/12/11 04:12:55 clusterSetup.go:346: Waiting for app: TestDeleteBeforeUndeploy to get deployed
2021/12/11 04:13:00 clusterSetup.go:346: Waiting for app: TestDeleteBeforeUndeploy to get deployed
2021/12/11 04:13:05 clusterSetup.go:346: Waiting for app: TestDeleteBeforeUndeploy to get deployed
2021/12/11 04:13:10 clusterSetup.go:346: Waiting for app: TestDeleteBeforeUndeploy to get deployed
2021/12/11 04:13:10 clusterSetup.go:354: App: TestDeleteBeforeUndeploy got deployed
2021/12/11 04:13:10 eventingSetup.go:418: Function: TestDeleteBeforeUndeploy delete from main store request response code: 422 dump: {
 "name": "ERR_APP_NOT_UNDEPLOYED",
 "code": 22,
 "description": "Function hasn't undeployed",
 "attributes": null,
 "runtime_info": {
  "code": 22,
  "info": "Function: TestDeleteBeforeUndeploy skipping delete request from primary store, as it hasn't been undeployed"
 }
}
2021/12/11 04:13:10 eventingSetup.go:270: Function: TestDeleteBeforeUndeploy update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/11 04:13:10 eventingSetup.go:418: Function: TestDeleteBeforeUndeploy delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDeleteBeforeUndeploy deleting in the background"
}
2021/12/11 04:13:16 clusterSetup.go:397: Waiting for app: TestDeleteBeforeUndeploy to get un-deployed
2021/12/11 04:13:16 clusterSetup.go:405: App: TestDeleteBeforeUndeploy got un-deployed
2021/12/11 04:13:16 eventingSetup.go:46: No eventing-con process running
2021/12/11 04:13:16 eventingSetup.go:418: Function: TestDeleteBeforeUndeploy delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDeleteBeforeUndeploy not found"
 }
}
2021/12/11 04:13:17 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/11 04:13:22 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 04:13:24 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/11 04:13:29 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDeleteBeforeUndeploy (44.71s)
=== RUN   TestUndeployWhenTimersAreFired
2021/12/11 04:13:39 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/11 04:13:39 clusterSetup.go:405: App: TestUndeployWhenTimersAreFired got un-deployed
2021/12/11 04:13:39 eventingSetup.go:46: No eventing-con process running
2021/12/11 04:13:39 eventingSetup.go:418: Function: TestUndeployWhenTimersAreFired delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestUndeployWhenTimersAreFired not found"
 }
}
2021/12/11 04:13:40 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/11 04:13:45 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 04:13:47 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/11 04:13:52 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 04:13:52 eventingSetup.go:125: Deploying app: TestUndeployWhenTimersAreFired. Item count src bucket: 0 dst bucket: 0
2021/12/11 04:13:53 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestUndeployWhenTimersAreFired' in metakv",
  "warnings": null
 }
}
2021/12/11 04:13:58 clusterSetup.go:346: Waiting for app: TestUndeployWhenTimersAreFired to get deployed
2021/12/11 04:14:03 clusterSetup.go:346: Waiting for app: TestUndeployWhenTimersAreFired to get deployed
2021/12/11 04:14:08 clusterSetup.go:346: Waiting for app: TestUndeployWhenTimersAreFired to get deployed
2021/12/11 04:14:13 clusterSetup.go:346: Waiting for app: TestUndeployWhenTimersAreFired to get deployed
2021/12/11 04:14:13 clusterSetup.go:354: App: TestUndeployWhenTimersAreFired got deployed
2021/12/11 04:14:13 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/11 04:14:44 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 04:14:49 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/11 04:14:54 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/11 04:14:59 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/11 04:15:04 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/11 04:15:04 clusterSetup.go:405: App: TestUndeployWhenTimersAreFired got un-deployed
2021/12/11 04:15:04 eventingSetup.go:46: No eventing-con process running
2021/12/11 04:16:49 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/11 04:16:49 clusterSetup.go:405: App: TestUndeployWhenTimersAreFired got un-deployed
2021/12/11 04:16:49 eventingSetup.go:46: No eventing-con process running
2021/12/11 04:16:49 eventingSetup.go:418: Function: TestUndeployWhenTimersAreFired delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestUndeployWhenTimersAreFired deleting in the background"
}
2021/12/11 04:16:50 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/11 04:17:00 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2972
2021/12/11 04:17:05 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2972
2021/12/11 04:17:05 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 04:17:07 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/11 04:17:17 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/11 04:17:17 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestUndeployWhenTimersAreFired (228.00s)
=== RUN   TestUndeployWithKVFailover
2021/12/11 04:17:27 clusterSetup.go:397: Waiting for app: TestUndeployWithKVFailover to get un-deployed
2021/12/11 04:17:27 clusterSetup.go:405: App: TestUndeployWithKVFailover got un-deployed
2021/12/11 04:17:27 eventingSetup.go:46: No eventing-con process running
2021/12/11 04:17:27 eventingSetup.go:418: Function: TestUndeployWithKVFailover delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestUndeployWithKVFailover not found"
 }
}
2021/12/11 04:17:28 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/11 04:17:33 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 04:17:35 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/11 04:17:40 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 04:17:40 eventingSetup.go:125: Deploying app: TestUndeployWithKVFailover. Item count src bucket: 0 dst bucket: 0
2021/12/11 04:17:41 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestUndeployWithKVFailover' in metakv",
  "warnings": null
 }
}
2021/12/11 04:17:41 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/11 04:17:48 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 04:17:53 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 04:17:58 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 04:18:03 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 04:18:08 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 04:18:13 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 04:18:18 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 1902
2021/12/11 04:18:23 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 1902
2021/12/11 04:18:23 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/11 04:18:23 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: kv to the cluster
2021/12/11 04:18:23 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/11 04:18:28 Rebalance progress: 29.16666666666666
2021/12/11 04:18:33 Rebalance progress: 29.16666666666666
2021/12/11 04:18:38 Rebalance progress: 100
2021/12/11 04:18:38 Failing over: [127.0.0.1:19003]
2021/12/11 04:18:48 Function: TestUndeployWithKVFailover update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/11 04:19:48 Removing node(s): [] from the cluster
2021/12/11 04:19:54 Rebalance progress: 29.16666666666666
2021/12/11 04:19:59 Rebalance progress: 29.16666666666666
2021/12/11 04:20:04 Rebalance progress: 100
2021/12/11 04:20:09 Waiting for app: TestUndeployWithKVFailover to get un-deployed
2021/12/11 04:20:09 App: TestUndeployWithKVFailover got un-deployed
2021/12/11 04:20:09 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 04:20:14 Waiting for app: TestUndeployWithKVFailover to get un-deployed
2021/12/11 04:20:14 App: TestUndeployWithKVFailover got un-deployed
2021/12/11 04:20:14 No eventing-con process running
2021/12/11 04:20:14 Function: TestUndeployWithKVFailover delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestUndeployWithKVFailover deleting in the background"
}
2021/12/11 04:20:15 Context: Bucket flush, response code: 200 dump: 
2021/12/11 04:20:25 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/11 04:20:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 04:20:27 Context: Bucket flush, response code: 200 dump: 
2021/12/11 04:20:37 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/11 04:20:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestUndeployWithKVFailover (199.85s)
=== RUN   TestBucketFlushWhileFnDeployed
2021/12/11 04:20:47 Waiting for app: TestBucketFlushWhileFnDeployed to get un-deployed
2021/12/11 04:20:47 App: TestBucketFlushWhileFnDeployed got un-deployed
2021/12/11 04:20:47 No eventing-con process running
2021/12/11 04:20:47 Function: TestBucketFlushWhileFnDeployed delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestBucketFlushWhileFnDeployed not found"
 }
}
2021/12/11 04:20:48 Context: Bucket flush, response code: 200 dump: 
2021/12/11 04:20:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 04:20:55 Context: Bucket flush, response code: 200 dump: 
2021/12/11 04:21:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 04:21:00 Deploying app: TestBucketFlushWhileFnDeployed. Item count src bucket: 0 dst bucket: 0
2021/12/11 04:21:01 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestBucketFlushWhileFnDeployed' in metakv",
  "warnings": null
 }
}
2021/12/11 04:21:06 Waiting for app: TestBucketFlushWhileFnDeployed to get deployed
2021/12/11 04:21:11 Waiting for app: TestBucketFlushWhileFnDeployed to get deployed
2021/12/11 04:21:16 Waiting for app: TestBucketFlushWhileFnDeployed to get deployed
2021/12/11 04:21:21 Waiting for app: TestBucketFlushWhileFnDeployed to get deployed
2021/12/11 04:21:21 App: TestBucketFlushWhileFnDeployed got deployed
2021/12/11 04:21:21 Starting bucket ops to source bucket
2021/12/11 04:21:41 Waiting for dst bucket item count to get to: 20000 curr count: 9517
2021/12/11 04:21:41 src & dst bucket item count matched up. src bucket count: 20000 dst bucket count: 20000
2021/12/11 04:21:41 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 04:21:44 Context: Bucket flush, response code: 200 dump: 
2021/12/11 04:21:54 Waiting for dst bucket item count to get to: 0 curr count: 20000
2021/12/11 04:21:59 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/11 04:21:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 04:22:00 Context: Bucket flush, response code: 200 dump: 
2021/12/11 04:22:10 Waiting for dst bucket item count to get to: 0 curr count: 15194
2021/12/11 04:22:15 Waiting for dst bucket item count to get to: 0 curr count: 15194
2021/12/11 04:22:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 04:22:15 Starting bucket ops to source bucket
2021/12/11 04:22:28 Waiting for dst bucket item count to get to: 10000 curr count: 3660
2021/12/11 04:22:33 Waiting for dst bucket item count to get to: 10000 curr count: 3660
2021/12/11 04:22:33 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/11 04:22:33 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 04:22:33 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 04:22:38 Waiting for app: TestBucketFlushWhileFnDeployed to get un-deployed
2021/12/11 04:22:43 Waiting for app: TestBucketFlushWhileFnDeployed to get un-deployed
2021/12/11 04:22:48 Waiting for app: TestBucketFlushWhileFnDeployed to get un-deployed
2021/12/11 04:22:53 Waiting for app: TestBucketFlushWhileFnDeployed to get un-deployed
2021/12/11 04:22:53 App: TestBucketFlushWhileFnDeployed got un-deployed
2021/12/11 04:22:53 No eventing-con process running
2021/12/11 04:22:53 Function: TestBucketFlushWhileFnDeployed delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestBucketFlushWhileFnDeployed deleting in the background"
}
2021/12/11 04:22:55 Context: Bucket flush, response code: 200 dump: 
2021/12/11 04:23:05 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/11 04:23:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 04:23:06 Context: Bucket flush, response code: 200 dump: 
2021/12/11 04:23:16 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/11 04:23:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestBucketFlushWhileFnDeployed (159.72s)
=== RUN   TestTimerOverwrite
2021/12/11 04:23:26 Waiting for app: TestTimerOverwrite to get un-deployed
2021/12/11 04:23:26 App: TestTimerOverwrite got un-deployed
2021/12/11 04:23:26 No eventing-con process running
2021/12/11 04:23:26 Function: TestTimerOverwrite delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestTimerOverwrite not found"
 }
}
2021/12/11 04:23:28 Context: Bucket flush, response code: 200 dump: 
2021/12/11 04:23:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 04:23:34 Context: Bucket flush, response code: 200 dump: 
2021/12/11 04:23:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 04:23:39 Deploying app: TestTimerOverwrite. Item count src bucket: 0 dst bucket: 0
2021/12/11 04:23:41 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestTimerOverwrite' in metakv",
  "warnings": null
 }
}
2021/12/11 04:23:46 Waiting for app: TestTimerOverwrite to get deployed
2021/12/11 04:23:51 Waiting for app: TestTimerOverwrite to get deployed
2021/12/11 04:23:56 Waiting for app: TestTimerOverwrite to get deployed
2021/12/11 04:24:01 Waiting for app: TestTimerOverwrite to get deployed
2021/12/11 04:24:01 App: TestTimerOverwrite got deployed
2021/12/11 04:24:11 Starting bucket ops to source bucket
2021/12/11 04:24:20 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 04:24:25 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 04:24:30 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 04:24:35 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 04:24:40 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 04:24:45 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 04:24:50 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 04:24:55 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 04:25:00 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 04:25:05 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 04:25:10 Waiting for dst bucket item count to get to: 5000 curr count: 997
2021/12/11 04:25:15 Waiting for dst bucket item count to get to: 5000 curr count: 997
2021/12/11 04:25:15 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/11 04:25:25 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 04:25:25 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 04:25:30 Waiting for app: TestTimerOverwrite to get un-deployed
2021/12/11 04:25:35 Waiting for app: TestTimerOverwrite to get un-deployed
2021/12/11 04:25:40 Waiting for app: TestTimerOverwrite to get un-deployed
2021/12/11 04:25:45 Waiting for app: TestTimerOverwrite to get un-deployed
2021/12/11 04:25:45 App: TestTimerOverwrite got un-deployed
2021/12/11 04:25:45 No eventing-con process running
2021/12/11 04:25:45 Function: TestTimerOverwrite delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestTimerOverwrite deleting in the background"
}
2021/12/11 04:25:46 Context: Bucket flush, response code: 200 dump: 
2021/12/11 04:25:56 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/11 04:25:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 04:25:58 Context: Bucket flush, response code: 200 dump: 
2021/12/11 04:26:08 Waiting for dst bucket item count to get to: 0 curr count: 3332
2021/12/11 04:26:13 Waiting for dst bucket item count to get to: 0 curr count: 814
2021/12/11 04:26:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestTimerOverwrite (176.59s)
=== RUN   TestUndeployBackdoorDuringBootstrap
2021/12/11 04:26:18 Adding node: https://127.0.0.1:19003 with role: kv to the cluster
2021/12/11 04:26:19 Removing node(s): [] from the cluster
2021/12/11 04:26:24 Rebalance progress: 100
2021/12/11 04:26:24 Failing over: [127.0.0.1:19003]
2021/12/11 04:26:29 Waiting for app: TestUndeployBackdoorDuringBootstrap to get un-deployed
2021/12/11 04:26:29 App: TestUndeployBackdoorDuringBootstrap got un-deployed
2021/12/11 04:26:29 No eventing-con process running
2021/12/11 04:26:29 Function: TestUndeployBackdoorDuringBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestUndeployBackdoorDuringBootstrap not found"
 }
}
2021/12/11 04:26:30 Context: Bucket flush, response code: 200 dump: 
2021/12/11 04:26:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 04:26:37 Context: Bucket flush, response code: 200 dump: 
2021/12/11 04:26:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 04:26:42 Deploying app: TestUndeployBackdoorDuringBootstrap. Item count src bucket: 0 dst bucket: 0
2021/12/11 04:26:43 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestUndeployBackdoorDuringBootstrap' in metakv",
  "warnings": null
 }
}
2021/12/11 04:26:43 Starting bucket ops to source bucket
2021/12/11 04:26:53 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 04:26:53 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 04:26:53 Context: Post to set retry counter, response code: 200 dump: 
2021/12/11 04:26:53 set retry counter, response &{[] }
2021/12/11 04:27:58 Waiting for app: TestUndeployBackdoorDuringBootstrap to get un-deployed
2021/12/11 04:27:58 App: TestUndeployBackdoorDuringBootstrap got un-deployed
2021/12/11 04:27:58 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 04:27:58 Function: TestUndeployBackdoorDuringBootstrap delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestUndeployBackdoorDuringBootstrap deleting in the background"
}
2021/12/11 04:28:03 Waiting for app: TestUndeployBackdoorDuringBootstrap to get un-deployed
2021/12/11 04:28:03 App: TestUndeployBackdoorDuringBootstrap got un-deployed
2021/12/11 04:28:03 No eventing-con process running
2021/12/11 04:28:03 Function: TestUndeployBackdoorDuringBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestUndeployBackdoorDuringBootstrap not found"
 }
}
2021/12/11 04:28:04 Context: Bucket flush, response code: 200 dump: 
2021/12/11 04:28:14 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/11 04:28:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 04:28:16 Context: Bucket flush, response code: 200 dump: 
2021/12/11 04:28:26 Waiting for dst bucket item count to get to: 0 curr count: 1666
2021/12/11 04:28:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 04:28:26 Removing node(s): [] from the cluster
2021/12/11 04:28:31 Rebalance progress: 100
--- PASS: TestUndeployBackdoorDuringBootstrap (138.23s)
=== RUN   TestOnUpdateSrcMutation
2021/12/11 04:28:41 Waiting for app: TestOnUpdateSrcMutation to get un-deployed
2021/12/11 04:28:41 App: TestOnUpdateSrcMutation got un-deployed
2021/12/11 04:28:41 No eventing-con process running
2021/12/11 04:28:41 Function: TestOnUpdateSrcMutation delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateSrcMutation not found"
 }
}
2021/12/11 04:28:42 Context: Bucket flush, response code: 200 dump: 
2021/12/11 04:28:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 04:28:49 Context: Bucket flush, response code: 200 dump: 
2021/12/11 04:28:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 04:28:54 Deploying app: TestOnUpdateSrcMutation. Item count src bucket: 0 dst bucket: 0
2021/12/11 04:28:55 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateSrcMutation' in metakv",
  "warnings": null
 }
}
2021/12/11 04:28:55 Starting bucket ops to source bucket
2021/12/11 04:29:22 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/11 04:29:22 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 04:29:22 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 04:29:27 Waiting for app: TestOnUpdateSrcMutation to get un-deployed
2021/12/11 04:29:32 Waiting for app: TestOnUpdateSrcMutation to get un-deployed
2021/12/11 04:29:37 Waiting for app: TestOnUpdateSrcMutation to get un-deployed
2021/12/11 04:29:42 Waiting for app: TestOnUpdateSrcMutation to get un-deployed
2021/12/11 04:29:42 App: TestOnUpdateSrcMutation got un-deployed
2021/12/11 04:29:42 No eventing-con process running
2021/12/11 04:29:42 Function: TestOnUpdateSrcMutation delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateSrcMutation deleting in the background"
}
2021/12/11 04:29:44 Context: Bucket flush, response code: 200 dump: 
2021/12/11 04:29:54 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/11 04:29:59 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/11 04:29:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 04:30:00 Context: Bucket flush, response code: 200 dump: 
2021/12/11 04:30:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateSrcMutation (94.35s)
=== RUN   TestOnDeleteSrcMutation
2021/12/11 04:30:15 Waiting for app: TestOnDeleteSrcMutation to get un-deployed
2021/12/11 04:30:15 App: TestOnDeleteSrcMutation got un-deployed
2021/12/11 04:30:15 No eventing-con process running
2021/12/11 04:30:16 Function: TestOnDeleteSrcMutation delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnDeleteSrcMutation not found"
 }
}
2021/12/11 04:30:17 Context: Bucket flush, response code: 200 dump: 
2021/12/11 04:30:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 04:30:23 Context: Bucket flush, response code: 200 dump: 
2021/12/11 04:30:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 04:30:29 Deploying app: TestOnDeleteSrcMutation. Item count src bucket: 0 dst bucket: 0
2021/12/11 04:30:30 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnDeleteSrcMutation' in metakv",
  "warnings": null
 }
}
2021/12/11 04:30:30 Starting bucket ops to source bucket
2021/12/11 04:30:58 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/11 04:30:58 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 04:30:58 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 04:31:03 Waiting for app: TestOnDeleteSrcMutation to get un-deployed
2021/12/11 04:31:08 Waiting for app: TestOnDeleteSrcMutation to get un-deployed
2021/12/11 04:31:13 Waiting for app: TestOnDeleteSrcMutation to get un-deployed
2021/12/11 04:31:18 Waiting for app: TestOnDeleteSrcMutation to get un-deployed
2021/12/11 04:31:18 App: TestOnDeleteSrcMutation got un-deployed
2021/12/11 04:31:18 No eventing-con process running
2021/12/11 04:31:19 Function: TestOnDeleteSrcMutation delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnDeleteSrcMutation deleting in the background"
}
2021/12/11 04:31:20 Context: Bucket flush, response code: 200 dump: 
2021/12/11 04:31:30 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/11 04:31:35 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/11 04:31:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 04:31:37 Context: Bucket flush, response code: 200 dump: 
2021/12/11 04:31:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnDeleteSrcMutation (96.11s)
=== RUN   TestOnUpdateSrcMutationWithTimer
2021/12/11 04:31:52 Waiting for app: TestOnUpdateSrcMutationWithTimer to get un-deployed
2021/12/11 04:31:52 App: TestOnUpdateSrcMutationWithTimer got un-deployed
2021/12/11 04:31:52 No eventing-con process running
2021/12/11 04:31:52 Function: TestOnUpdateSrcMutationWithTimer delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateSrcMutationWithTimer not found"
 }
}
2021/12/11 04:31:53 Context: Bucket flush, response code: 200 dump: 
2021/12/11 04:31:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 04:32:00 Context: Bucket flush, response code: 200 dump: 
2021/12/11 04:32:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 04:32:05 Deploying app: TestOnUpdateSrcMutationWithTimer. Item count src bucket: 0 dst bucket: 0
2021/12/11 04:32:06 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateSrcMutationWithTimer' in metakv",
  "warnings": null
 }
}
2021/12/11 04:32:06 Starting bucket ops to source bucket
2021/12/11 04:32:43 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/11 04:32:43 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 04:32:43 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 04:32:48 Waiting for app: TestOnUpdateSrcMutationWithTimer to get un-deployed
2021/12/11 04:32:53 Waiting for app: TestOnUpdateSrcMutationWithTimer to get un-deployed
2021/12/11 04:32:58 Waiting for app: TestOnUpdateSrcMutationWithTimer to get un-deployed
2021/12/11 04:33:03 Waiting for app: TestOnUpdateSrcMutationWithTimer to get un-deployed
2021/12/11 04:33:03 App: TestOnUpdateSrcMutationWithTimer got un-deployed
2021/12/11 04:33:03 No eventing-con process running
2021/12/11 04:33:03 Function: TestOnUpdateSrcMutationWithTimer delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateSrcMutationWithTimer deleting in the background"
}
2021/12/11 04:33:04 Context: Bucket flush, response code: 200 dump: 
2021/12/11 04:33:14 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/11 04:33:19 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/11 04:33:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 04:33:21 Context: Bucket flush, response code: 200 dump: 
2021/12/11 04:33:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateSrcMutationWithTimer (104.46s)
=== RUN   TestOnDeleteSrcMutationsWithTimer
2021/12/11 04:33:36 Waiting for app: TestOnDeleteSrcMutationsWithTimer to get un-deployed
2021/12/11 04:33:36 App: TestOnDeleteSrcMutationsWithTimer got un-deployed
2021/12/11 04:33:36 No eventing-con process running
2021/12/11 04:33:36 Function: TestOnDeleteSrcMutationsWithTimer delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnDeleteSrcMutationsWithTimer not found"
 }
}
2021/12/11 04:33:37 Context: Bucket flush, response code: 200 dump: 
2021/12/11 04:33:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 04:33:44 Context: Bucket flush, response code: 200 dump: 
2021/12/11 04:33:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 04:33:49 Deploying app: TestOnDeleteSrcMutationsWithTimer. Item count src bucket: 0 dst bucket: 0
2021/12/11 04:33:50 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnDeleteSrcMutationsWithTimer' in metakv",
  "warnings": null
 }
}
2021/12/11 04:33:50 Starting bucket ops to source bucket
2021/12/11 04:34:34 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/11 04:34:34 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 04:34:34 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 04:34:39 Waiting for app: TestOnDeleteSrcMutationsWithTimer to get un-deployed
2021/12/11 04:34:44 Waiting for app: TestOnDeleteSrcMutationsWithTimer to get un-deployed
2021/12/11 04:34:49 Waiting for app: TestOnDeleteSrcMutationsWithTimer to get un-deployed
2021/12/11 04:34:54 Waiting for app: TestOnDeleteSrcMutationsWithTimer to get un-deployed
2021/12/11 04:34:54 App: TestOnDeleteSrcMutationsWithTimer got un-deployed
2021/12/11 04:34:54 No eventing-con process running
2021/12/11 04:34:54 Function: TestOnDeleteSrcMutationsWithTimer delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnDeleteSrcMutationsWithTimer deleting in the background"
}
2021/12/11 04:34:56 Context: Bucket flush, response code: 200 dump: 
2021/12/11 04:35:06 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/11 04:35:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 04:35:07 Context: Bucket flush, response code: 200 dump: 
2021/12/11 04:35:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnDeleteSrcMutationsWithTimer (106.25s)
=== RUN   TestInterHandlerRecursion
2021/12/11 04:35:22 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/11 04:35:22 App: TestInterHandlerRecursion_source_bucket_update_op got un-deployed
2021/12/11 04:35:22 No eventing-con process running
2021/12/11 04:35:22 Function: TestInterHandlerRecursion_source_bucket_update_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestInterHandlerRecursion_source_bucket_update_op not found"
 }
}
2021/12/11 04:35:24 Context: Bucket flush, response code: 200 dump: 
2021/12/11 04:35:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 04:35:30 Context: Bucket flush, response code: 200 dump: 
2021/12/11 04:35:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 04:35:40 Waiting for app: TestInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/11 04:35:40 App: TestInterHandlerRecursion_src_bucket_op_on_delete got un-deployed
2021/12/11 04:35:40 No eventing-con process running
2021/12/11 04:35:40 Function: TestInterHandlerRecursion_src_bucket_op_on_delete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestInterHandlerRecursion_src_bucket_op_on_delete not found"
 }
}
2021/12/11 04:35:42 Context: Bucket flush, response code: 200 dump: 
2021/12/11 04:35:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 04:35:48 Context: Bucket flush, response code: 200 dump: 
2021/12/11 04:35:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 04:35:53 Deploying app: TestInterHandlerRecursion_source_bucket_update_op. Item count src bucket: 0 dst bucket: 0
2021/12/11 04:35:55 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterHandlerRecursion_source_bucket_update_op' in metakv",
  "warnings": null
 }
}
2021/12/11 04:35:55 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterHandlerRecursion_source_bucket_update_op' in metakv",
  "warnings": null
 }
} err 
2021/12/11 04:36:00 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/11 04:36:05 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/11 04:36:10 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/11 04:36:15 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/11 04:36:15 App: TestInterHandlerRecursion_source_bucket_update_op got deployed
2021/12/11 04:36:15 Deploying app: TestInterHandlerRecursion_src_bucket_op_on_delete. Item count src bucket: 0 dst bucket: 0
2021/12/11 04:36:15 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_INTER_FUNCTION_RECURSION",
 "code": 50,
 "description": "Inter function recursion error, only one function is allowed to do source bucket mutation/delete on a bucket",
 "attributes": null,
 "runtime_info": {
  "code": 50,
  "info": "Inter handler recursion error"
 }
}
2021/12/11 04:36:15 response body {
 "name": "ERR_INTER_FUNCTION_RECURSION",
 "code": 50,
 "description": "Inter function recursion error, only one function is allowed to do source bucket mutation/delete on a bucket",
 "attributes": null,
 "runtime_info": {
  "code": 50,
  "info": "Inter handler recursion error"
 }
} err 
2021/12/11 04:36:15 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 04:36:20 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/11 04:36:25 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/11 04:36:30 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/11 04:36:35 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/11 04:36:35 App: TestInterHandlerRecursion_source_bucket_update_op got un-deployed
2021/12/11 04:36:35 No eventing-con process running
2021/12/11 04:36:35 Function: TestInterHandlerRecursion_source_bucket_update_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestInterHandlerRecursion_source_bucket_update_op deleting in the background"
}
2021/12/11 04:36:36 Context: Bucket flush, response code: 200 dump: 
2021/12/11 04:36:41 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 04:36:43 Context: Bucket flush, response code: 200 dump: 
2021/12/11 04:36:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 04:36:53 Waiting for app: TestInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/11 04:36:53 App: TestInterHandlerRecursion_src_bucket_op_on_delete got un-deployed
2021/12/11 04:36:53 No eventing-con process running
2021/12/11 04:36:53 Function: TestInterHandlerRecursion_src_bucket_op_on_delete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestInterHandlerRecursion_src_bucket_op_on_delete not found"
 }
}
2021/12/11 04:36:54 Context: Bucket flush, response code: 200 dump: 
2021/12/11 04:36:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 04:37:01 Context: Bucket flush, response code: 200 dump: 
2021/12/11 04:37:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestInterHandlerRecursion (113.55s)
=== RUN   TestInterBucketRecursion
2021/12/11 04:37:11 Deploying app: TestInterBucketRecursion_function1. Item count src bucket: 0 dst bucket: 0
2021/12/11 04:37:12 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterBucketRecursion_function1' in metakv",
  "warnings": null
 }
}
2021/12/11 04:37:12 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterBucketRecursion_function1' in metakv",
  "warnings": null
 }
} err 
2021/12/11 04:37:12 Deploying app: TestInterBucketRecursion_function2. Item count src bucket: 0 dst bucket: 0
2021/12/11 04:37:12 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_INTER_BUCKET_RECURSION",
 "code": 51,
 "description": "Inter bucket recursion error, deployment of current handler will cause inter bucket recursion",
 "attributes": null,
 "runtime_info": {
  "code": 51,
  "info": "Inter bucket recursion error; function: TestInterBucketRecursion_function2 causes a cycle involving functions: [TestInterBucketRecursion_function1], hence deployment is disallowed"
 }
}
2021/12/11 04:37:17 Waiting for app: TestInterBucketRecursion_function1 to get deployed
2021/12/11 04:37:22 Waiting for app: TestInterBucketRecursion_function1 to get deployed
2021/12/11 04:37:27 Waiting for app: TestInterBucketRecursion_function1 to get deployed
2021/12/11 04:37:32 Waiting for app: TestInterBucketRecursion_function1 to get deployed
2021/12/11 04:37:32 App: TestInterBucketRecursion_function1 got deployed
2021/12/11 04:37:32 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 04:37:37 Waiting for app: TestInterBucketRecursion_function1 to get un-deployed
2021/12/11 04:37:42 Waiting for app: TestInterBucketRecursion_function1 to get un-deployed
2021/12/11 04:37:47 Waiting for app: TestInterBucketRecursion_function1 to get un-deployed
2021/12/11 04:37:52 Waiting for app: TestInterBucketRecursion_function1 to get un-deployed
2021/12/11 04:37:52 App: TestInterBucketRecursion_function1 got un-deployed
2021/12/11 04:37:52 No eventing-con process running
2021/12/11 04:37:52 Function: TestInterBucketRecursion_function1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestInterBucketRecursion_function1 deleting in the background"
}
2021/12/11 04:37:53 Context: Bucket flush, response code: 200 dump: 
2021/12/11 04:37:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 04:38:00 Context: Bucket flush, response code: 200 dump: 
2021/12/11 04:38:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 04:38:10 Waiting for app: TestInterBucketRecursion_function2 to get un-deployed
2021/12/11 04:38:10 App: TestInterBucketRecursion_function2 got un-deployed
2021/12/11 04:38:10 No eventing-con process running
2021/12/11 04:38:10 Function: TestInterBucketRecursion_function2 delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestInterBucketRecursion_function2 not found"
 }
}
2021/12/11 04:38:12 Context: Bucket flush, response code: 200 dump: 
2021/12/11 04:38:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 04:38:18 Context: Bucket flush, response code: 200 dump: 
2021/12/11 04:38:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestInterBucketRecursion (77.47s)
=== RUN   TestLargeHandler
2021/12/11 04:38:23 Changed force_compress value to false
2021/12/11 04:38:23 Deploying app: TestLargeHandler_function. Item count src bucket: 0 dst bucket: 0
2021/12/11 04:38:24 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_APPCODE_SIZE",
 "code": 39,
 "description": "Function Code size is more than the configured limit",
 "attributes": null,
 "runtime_info": {
  "code": 39,
  "info": "Function: TestLargeHandler_function handler Code size is more than 131072. Code Size: 131768"
 }
}
2021/12/11 04:38:24 Changed force_compress value to true
2021/12/11 04:38:24 Deploying app: TestLargeHandler_function. Item count src bucket: 0 dst bucket: 0
2021/12/11 04:38:25 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestLargeHandler_function' in metakv",
  "warnings": null
 }
}
2021/12/11 04:38:30 Waiting for app: TestLargeHandler_function to get deployed
2021/12/11 04:38:35 Waiting for app: TestLargeHandler_function to get deployed
2021/12/11 04:38:40 Waiting for app: TestLargeHandler_function to get deployed
2021/12/11 04:38:45 Waiting for app: TestLargeHandler_function to get deployed
2021/12/11 04:38:45 App: TestLargeHandler_function got deployed
2021/12/11 04:38:46 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 04:38:51 Waiting for app: TestLargeHandler_function to get un-deployed
2021/12/11 04:38:56 Waiting for app: TestLargeHandler_function to get un-deployed
2021/12/11 04:39:01 Waiting for app: TestLargeHandler_function to get un-deployed
2021/12/11 04:39:06 Waiting for app: TestLargeHandler_function to get un-deployed
2021/12/11 04:39:06 App: TestLargeHandler_function got un-deployed
2021/12/11 04:39:06 No eventing-con process running
2021/12/11 04:39:06 Function: TestLargeHandler_function delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestLargeHandler_function deleting in the background"
}
2021/12/11 04:39:07 Context: Bucket flush, response code: 200 dump: 
2021/12/11 04:39:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 04:39:14 Context: Bucket flush, response code: 200 dump: 
2021/12/11 04:39:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestLargeHandler (55.32s)
=== RUN   TestAllowInterHandlerRecursion
2021/12/11 04:39:29 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/11 04:39:29 App: TestAllowInterHandlerRecursion_source_bucket_update_op got un-deployed
2021/12/11 04:39:29 No eventing-con process running
2021/12/11 04:39:29 Function: TestAllowInterHandlerRecursion_source_bucket_update_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestAllowInterHandlerRecursion_source_bucket_update_op not found"
 }
}
2021/12/11 04:39:30 Context: Bucket flush, response code: 200 dump: 
2021/12/11 04:39:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 04:39:37 Context: Bucket flush, response code: 200 dump: 
2021/12/11 04:39:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 04:39:47 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/11 04:39:47 App: TestAllowInterHandlerRecursion_src_bucket_op_on_delete got un-deployed
2021/12/11 04:39:47 No eventing-con process running
2021/12/11 04:39:47 Function: TestAllowInterHandlerRecursion_src_bucket_op_on_delete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestAllowInterHandlerRecursion_src_bucket_op_on_delete not found"
 }
}
2021/12/11 04:39:48 Context: Bucket flush, response code: 200 dump: 
2021/12/11 04:39:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 04:39:55 Context: Bucket flush, response code: 200 dump: 
2021/12/11 04:40:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 04:40:00 Deploying app: TestAllowInterHandlerRecursion_source_bucket_update_op. Item count src bucket: 0 dst bucket: 0
2021/12/11 04:40:01 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterHandlerRecursion_source_bucket_update_op' in metakv",
  "warnings": null
 }
}
2021/12/11 04:40:01 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterHandlerRecursion_source_bucket_update_op' in metakv",
  "warnings": null
 }
} err 
2021/12/11 04:40:06 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/11 04:40:11 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/11 04:40:16 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/11 04:40:21 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/11 04:40:21 App: TestAllowInterHandlerRecursion_source_bucket_update_op got deployed
2021/12/11 04:40:21 Deploying app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete. Item count src bucket: 0 dst bucket: 0
2021/12/11 04:40:22 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterHandlerRecursion_src_bucket_op_on_delete' in metakv",
  "warnings": null
 }
}
2021/12/11 04:40:22 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterHandlerRecursion_src_bucket_op_on_delete' in metakv",
  "warnings": null
 }
} err 
2021/12/11 04:40:27 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get deployed
2021/12/11 04:40:32 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get deployed
2021/12/11 04:40:37 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get deployed
2021/12/11 04:40:42 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get deployed
2021/12/11 04:40:42 App: TestAllowInterHandlerRecursion_src_bucket_op_on_delete got deployed
2021/12/11 04:40:42 TestAllowInterHandlerRecursion_src_bucket_op_on_delete is deployed
2021/12/11 04:40:42 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 04:40:47 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/11 04:40:52 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/11 04:40:57 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/11 04:41:02 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/11 04:41:02 App: TestAllowInterHandlerRecursion_source_bucket_update_op got un-deployed
2021/12/11 04:41:02 No eventing-con process running
2021/12/11 04:41:02 Function: TestAllowInterHandlerRecursion_source_bucket_update_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAllowInterHandlerRecursion_source_bucket_update_op deleting in the background"
}
2021/12/11 04:41:05 Context: Bucket flush, response code: 200 dump: 
2021/12/11 04:41:10 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 04:41:11 Context: Bucket flush, response code: 200 dump: 
2021/12/11 04:41:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 04:41:17 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 04:41:22 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/11 04:41:27 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/11 04:41:32 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/11 04:41:37 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/11 04:41:37 App: TestAllowInterHandlerRecursion_src_bucket_op_on_delete got un-deployed
2021/12/11 04:41:37 No eventing-con process running
2021/12/11 04:41:37 Function: TestAllowInterHandlerRecursion_src_bucket_op_on_delete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAllowInterHandlerRecursion_src_bucket_op_on_delete deleting in the background"
}
2021/12/11 04:41:38 Context: Bucket flush, response code: 200 dump: 
2021/12/11 04:41:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 04:41:45 Context: Bucket flush, response code: 200 dump: 
2021/12/11 04:41:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAllowInterHandlerRecursion (151.15s)
=== RUN   TestAllowInterBucketRecursion
2021/12/11 04:41:55 Deploying app: TestAllowInterBucketRecursion_function1. Item count src bucket: 0 dst bucket: 0
2021/12/11 04:41:56 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterBucketRecursion_function1' in metakv",
  "warnings": null
 }
}
2021/12/11 04:41:56 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterBucketRecursion_function1' in metakv",
  "warnings": null
 }
} err 
2021/12/11 04:42:01 Waiting for app: TestAllowInterBucketRecursion_function1 to get deployed
2021/12/11 04:42:06 Waiting for app: TestAllowInterBucketRecursion_function1 to get deployed
2021/12/11 04:42:11 Waiting for app: TestAllowInterBucketRecursion_function1 to get deployed
2021/12/11 04:42:16 Waiting for app: TestAllowInterBucketRecursion_function1 to get deployed
2021/12/11 04:42:16 App: TestAllowInterBucketRecursion_function1 got deployed
2021/12/11 04:42:16 Deploying app: TestAllowInterBucketRecursion_function2. Item count src bucket: 0 dst bucket: 0
2021/12/11 04:42:17 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterBucketRecursion_function2' in metakv",
  "warnings": null
 }
}
2021/12/11 04:42:17 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterBucketRecursion_function2' in metakv",
  "warnings": null
 }
} err 
2021/12/11 04:42:22 Waiting for app: TestAllowInterBucketRecursion_function2 to get deployed
2021/12/11 04:42:27 Waiting for app: TestAllowInterBucketRecursion_function2 to get deployed
2021/12/11 04:42:32 Waiting for app: TestAllowInterBucketRecursion_function2 to get deployed
2021/12/11 04:42:37 Waiting for app: TestAllowInterBucketRecursion_function2 to get deployed
2021/12/11 04:42:37 App: TestAllowInterBucketRecursion_function2 got deployed
2021/12/11 04:42:37 TestAllowInterBucketRecursion_function2 is deployed
2021/12/11 04:42:37 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 04:42:42 Waiting for app: TestAllowInterBucketRecursion_function1 to get un-deployed
2021/12/11 04:42:47 Waiting for app: TestAllowInterBucketRecursion_function1 to get un-deployed
2021/12/11 04:42:52 Waiting for app: TestAllowInterBucketRecursion_function1 to get un-deployed
2021/12/11 04:42:57 Waiting for app: TestAllowInterBucketRecursion_function1 to get un-deployed
2021/12/11 04:42:57 App: TestAllowInterBucketRecursion_function1 got un-deployed
2021/12/11 04:42:57 No eventing-con process running
2021/12/11 04:42:57 Function: TestAllowInterBucketRecursion_function1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAllowInterBucketRecursion_function1 deleting in the background"
}
2021/12/11 04:42:59 Context: Bucket flush, response code: 200 dump: 
2021/12/11 04:43:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 04:43:06 Context: Bucket flush, response code: 200 dump: 
2021/12/11 04:43:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 04:43:12 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 04:43:17 Waiting for app: TestAllowInterBucketRecursion_function2 to get un-deployed
2021/12/11 04:43:22 Waiting for app: TestAllowInterBucketRecursion_function2 to get un-deployed
2021/12/11 04:43:27 Waiting for app: TestAllowInterBucketRecursion_function2 to get un-deployed
2021/12/11 04:43:32 Waiting for app: TestAllowInterBucketRecursion_function2 to get un-deployed
2021/12/11 04:43:32 App: TestAllowInterBucketRecursion_function2 got un-deployed
2021/12/11 04:43:32 No eventing-con process running
2021/12/11 04:43:32 Function: TestAllowInterBucketRecursion_function2 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAllowInterBucketRecursion_function2 deleting in the background"
}
2021/12/11 04:43:33 Context: Bucket flush, response code: 200 dump: 
2021/12/11 04:43:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 04:43:40 Context: Bucket flush, response code: 200 dump: 
2021/12/11 04:43:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAllowInterBucketRecursion (114.93s)
=== RUN   TestInterBucketRecursion2
2021/12/11 04:43:51 Context: Bucket flush, response code: 200 dump: 
2021/12/11 04:43:57 Context: Bucket flush, response code: 200 dump: 
2021/12/11 04:44:03 Context: Bucket flush, response code: 200 dump: 
2021/12/11 04:44:08 Context: Bucket flush, response code: 200 dump: 
2021/12/11 04:44:13 Deploying app: TestInterBucketRecursion2_function1. Item count src bucket: 0 dst bucket: 0
2021/12/11 04:44:15 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterBucketRecursion2_function1' in metakv",
  "warnings": null
 }
}
2021/12/11 04:44:20 Waiting for app: TestInterBucketRecursion2_function1 to get deployed
2021/12/11 04:44:25 Waiting for app: TestInterBucketRecursion2_function1 to get deployed
2021/12/11 04:44:30 Waiting for app: TestInterBucketRecursion2_function1 to get deployed
2021/12/11 04:44:35 Waiting for app: TestInterBucketRecursion2_function1 to get deployed
2021/12/11 04:44:35 App: TestInterBucketRecursion2_function1 got deployed
2021/12/11 04:44:35 Deploying app: TestInterBucketRecursion2_function2. Item count src bucket: 0 dst bucket: 0
2021/12/11 04:44:36 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterBucketRecursion2_function2' in metakv",
  "warnings": null
 }
}
2021/12/11 04:44:41 Waiting for app: TestInterBucketRecursion2_function2 to get deployed
2021/12/11 04:44:46 Waiting for app: TestInterBucketRecursion2_function2 to get deployed
2021/12/11 04:44:51 Waiting for app: TestInterBucketRecursion2_function2 to get deployed
2021/12/11 04:44:56 Waiting for app: TestInterBucketRecursion2_function2 to get deployed
2021/12/11 04:44:56 App: TestInterBucketRecursion2_function2 got deployed
2021/12/11 04:44:56 Deploying app: TestInterBucketRecursion2_function3. Item count src bucket: 0 dst bucket: 0
2021/12/11 04:44:57 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterBucketRecursion2_function3' in metakv",
  "warnings": null
 }
}
2021/12/11 04:45:02 Waiting for app: TestInterBucketRecursion2_function3 to get deployed
2021/12/11 04:45:07 Waiting for app: TestInterBucketRecursion2_function3 to get deployed
2021/12/11 04:45:12 Waiting for app: TestInterBucketRecursion2_function3 to get deployed
2021/12/11 04:45:17 Waiting for app: TestInterBucketRecursion2_function3 to get deployed
2021/12/11 04:45:17 App: TestInterBucketRecursion2_function3 got deployed
2021/12/11 04:45:17 Deploying app: TestInterBucketRecursion2_function4. Item count src bucket: 0 dst bucket: 0
2021/12/11 04:45:17 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_INTER_BUCKET_RECURSION",
 "code": 51,
 "description": "Inter bucket recursion error, deployment of current handler will cause inter bucket recursion",
 "attributes": null,
 "runtime_info": {
  "code": 51,
  "info": "Inter bucket recursion error; function: TestInterBucketRecursion2_function4 causes a cycle involving functions: [TestInterBucketRecursion2_function1 TestInterBucketRecursion2_function2 TestInterBucketRecursion2_function3], hence deployment is disallowed"
 }
}
2021/12/11 04:45:17 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 04:45:22 Waiting for app: TestInterBucketRecursion2_function1 to get un-deployed
2021/12/11 04:45:27 Waiting for app: TestInterBucketRecursion2_function1 to get un-deployed
2021/12/11 04:45:32 Waiting for app: TestInterBucketRecursion2_function1 to get un-deployed
2021/12/11 04:45:37 Waiting for app: TestInterBucketRecursion2_function1 to get un-deployed
2021/12/11 04:45:37 App: TestInterBucketRecursion2_function1 got un-deployed
2021/12/11 04:45:37 No eventing-con process running
2021/12/11 04:45:37 Function: TestInterBucketRecursion2_function1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestInterBucketRecursion2_function1 deleting in the background"
}
2021/12/11 04:45:39 Context: Bucket flush, response code: 200 dump: 
2021/12/11 04:45:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 04:45:45 Context: Bucket flush, response code: 200 dump: 
2021/12/11 04:45:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 04:45:50 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 04:45:55 Waiting for app: TestInterBucketRecursion2_function2 to get un-deployed
2021/12/11 04:46:00 Waiting for app: TestInterBucketRecursion2_function2 to get un-deployed
2021/12/11 04:46:06 Waiting for app: TestInterBucketRecursion2_function2 to get un-deployed
2021/12/11 04:46:11 Waiting for app: TestInterBucketRecursion2_function2 to get un-deployed
2021/12/11 04:46:11 App: TestInterBucketRecursion2_function2 got un-deployed
2021/12/11 04:46:11 No eventing-con process running
2021/12/11 04:46:11 Function: TestInterBucketRecursion2_function2 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestInterBucketRecursion2_function2 deleting in the background"
}
2021/12/11 04:46:12 Context: Bucket flush, response code: 200 dump: 
2021/12/11 04:46:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 04:46:19 Context: Bucket flush, response code: 200 dump: 
2021/12/11 04:46:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 04:46:24 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 04:46:29 Waiting for app: TestInterBucketRecursion2_function3 to get un-deployed
2021/12/11 04:46:34 Waiting for app: TestInterBucketRecursion2_function3 to get un-deployed
2021/12/11 04:46:39 Waiting for app: TestInterBucketRecursion2_function3 to get un-deployed
2021/12/11 04:46:44 Waiting for app: TestInterBucketRecursion2_function3 to get un-deployed
2021/12/11 04:46:44 App: TestInterBucketRecursion2_function3 got un-deployed
2021/12/11 04:46:44 No eventing-con process running
2021/12/11 04:46:44 Function: TestInterBucketRecursion2_function3 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestInterBucketRecursion2_function3 deleting in the background"
}
2021/12/11 04:46:45 Context: Bucket flush, response code: 200 dump: 
2021/12/11 04:46:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 04:46:52 Context: Bucket flush, response code: 200 dump: 
2021/12/11 04:46:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 04:47:02 Waiting for app: TestInterBucketRecursion2_function4 to get un-deployed
2021/12/11 04:47:02 App: TestInterBucketRecursion2_function4 got un-deployed
2021/12/11 04:47:02 No eventing-con process running
2021/12/11 04:47:02 Function: TestInterBucketRecursion2_function4 delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestInterBucketRecursion2_function4 not found"
 }
}
2021/12/11 04:47:04 Context: Bucket flush, response code: 200 dump: 
2021/12/11 04:47:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 04:47:10 Context: Bucket flush, response code: 200 dump: 
2021/12/11 04:47:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestInterBucketRecursion2 (212.49s)
=== RUN   TestBucketDeleteWithRebOut
2021/12/11 04:47:22 Removing node(s): [127.0.0.1:19001 127.0.0.1:19002 127.0.0.1:19003] from the cluster
2021/12/11 04:47:27 Rebalance progress: 100

VB distribution across eventing nodes and their workers::

2021/12/11 04:47:30 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/11 04:47:45 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/11 04:48:00 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/11 04:48:15 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/11 04:48:30 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/11 04:48:45 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/11 04:49:00 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/11 04:49:15 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/11 04:49:30 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/11 04:49:45 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]

DCP Stream statuses:

vbucket curr owner:


2021/12/11 04:50:00 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 04:50:00 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/11 04:50:01 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/11 04:50:06 Rebalance progress: 100

VB distribution across eventing nodes and their workers::


DCP Stream statuses:

vbucket curr owner:


2021/12/11 04:50:06 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 04:50:06 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/11 04:50:06 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/11 04:50:11 Rebalance progress: 100

VB distribution across eventing nodes and their workers::


DCP Stream statuses:

vbucket curr owner:


2021/12/11 04:50:11 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 04:50:16 clusterSetup.go:397: Waiting for app: TestBucketDeleteWithRebOut to get un-deployed
2021/12/11 04:50:16 clusterSetup.go:405: App: TestBucketDeleteWithRebOut got un-deployed
2021/12/11 04:50:16 eventingSetup.go:46: No eventing-con process running
2021/12/11 04:50:16 eventingSetup.go:418: Function: TestBucketDeleteWithRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestBucketDeleteWithRebOut not found"
 }
}
2021/12/11 04:50:18 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/11 04:50:23 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 04:50:24 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/11 04:50:29 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 04:50:34 eventingSetup.go:125: Deploying app: TestBucketDeleteWithRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/11 04:50:36 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestBucketDeleteWithRebOut' in metakv",
  "warnings": null
 }
}
2021/12/11 04:50:41 clusterSetup.go:346: Waiting for app: TestBucketDeleteWithRebOut to get deployed
2021/12/11 04:50:46 clusterSetup.go:346: Waiting for app: TestBucketDeleteWithRebOut to get deployed
2021/12/11 04:50:51 clusterSetup.go:346: Waiting for app: TestBucketDeleteWithRebOut to get deployed
2021/12/11 04:50:56 clusterSetup.go:346: Waiting for app: TestBucketDeleteWithRebOut to get deployed
2021/12/11 04:50:56 clusterSetup.go:354: App: TestBucketDeleteWithRebOut got deployed

VB distribution across eventing nodes and their workers::


DCP Stream statuses:
	stream status: running
	len: 1024
	vb list dump: [0-1023]

vbucket curr owner:
Producer node: 127.0.0.1:9302	Node UUID: ff39ea7c3d7f9701d93e7c48208f6b74
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [0-113]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [114-227]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 114
	v: [228-341]
Producer node: 172.31.2.134:9300	Node UUID: 8cb6f622ccb07b97de85c9bedb4434c2
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 113
	v: [911-1023]
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [683-796]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [797-910]
Producer node: 127.0.0.1:9303	Node UUID: 46adfc38c81b9f50741786c7f6948222
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 113
	v: [570-682]
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [342-455]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [456-569]


2021/12/11 04:50:56 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 04:50:56 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/11 04:51:01 Rebalance progress: 29.16666666666666
2021/12/11 04:51:06 Rebalance progress: 29.16666666666666
2021/12/11 04:51:11 Rebalance progress: 100

VB distribution across eventing nodes and their workers::


DCP Stream statuses:
	stream status: running
	len: 1024
	vb list dump: [0-1023]

vbucket curr owner:
Producer node: 127.0.0.1:9302	Node UUID: ff39ea7c3d7f9701d93e7c48208f6b74
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [0-113]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [114-227]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 114
	v: [228-341]
Producer node: 172.31.2.134:9300	Node UUID: 8cb6f622ccb07b97de85c9bedb4434c2
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 113
	v: [911-1023]
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [683-796]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [797-910]
Producer node: 127.0.0.1:9303	Node UUID: 46adfc38c81b9f50741786c7f6948222
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [342-455]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [456-569]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 113
	v: [570-682]


2021/12/11 04:51:11 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 04:51:11 handler_test.go:1674: Deleting source bucket: default
2021/12/11 04:51:13 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/11 04:51:13 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/11 04:51:18 Rebalance progress: 29.16666666666666
2021/12/11 04:51:23 Rebalance progress: 29.16666666666666
2021/12/11 04:51:28 Rebalance progress: 100

VB distribution across eventing nodes and their workers::


DCP Stream statuses:
	stream status: running
	len: 904
	vb list dump: [0-17, 19-21, 23-32, 34-43, 45-53, 55-77, 79-81, 83-84, 86, 90, 92-109, 111-122, 124-130, 132-150, 152-153, 156-160, 162-163, 165-198, 200-203, 206-209, 211-215, 217-220, 222-226, 228-231, 233-237, 239-241, 243-248, 250-266, 268-277, 279-280, 282-297, 299, 301-303, 305-320, 322-330, 332-337, 339-368, 370-376, 378-382, 384, 387-389, 391-393, 395-404, 406-420, 422-425, 427-442, 444-450, 453-456, 459-462, 464-470, 472, 474-478, 483, 485-490, 493-497, 499-501, 503-504, 507-510, 512-576, 578-585, 587-592, 594-599, 601, 603, 605-626, 628-630, 632-649, 653-670, 672-696, 698, 700-703, 705, 707-715, 717-721, 723-738, 740-742, 744-753, 755-757, 759-773, 775-777, 779-795, 797-803, 806-807, 809-811, 813-817, 819-820, 822-824, 826-846, 848-853, 855-858, 860-866, 868-870, 872-904, 906-934, 936-938, 942-952, 955-971, 974-977, 979-988, 990-993, 995-997, 999-1023]
	stream status: stopped
	len: 120
	vb list dump: [18, 22, 33, 44, 54, 78, 82, 85, 87-89, 91, 110, 123, 131, 151, 154-155, 161, 164, 199, 204-205, 210, 216, 221, 227, 232, 238, 242, 249, 267, 278, 281, 298, 300, 304, 321, 331, 338, 369, 377, 383, 385-386, 390, 394, 405, 421, 426, 443, 451-452, 457-458, 463, 471, 473, 479-482, 484, 491-492, 498, 502, 505-506, 511, 577, 586, 593, 600, 602, 604, 627, 631, 650-652, 671, 697, 699, 704, 706, 716, 722, 739, 743, 754, 758, 774, 778, 796, 804-805, 808, 812, 818, 821, 825, 847, 854, 859, 867, 871, 905, 935, 939-941, 953-954, 972-973, 978, 989, 994, 998]

vbucket curr owner:
Producer node: 127.0.0.1:9302	Node UUID: ff39ea7c3d7f9701d93e7c48208f6b74
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 101
	v: [0-17, 19-21, 23-32, 34-43, 45-53, 55-77, 79-81, 83-84, 86, 90, 92-109, 111-113]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 100
	v: [114-122, 124-130, 132-150, 152-153, 156-160, 162-163, 165-198, 200-203, 206-209, 211-215, 217-220, 222-226]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 101
	v: [228-231, 233-237, 239-241, 243-248, 250-266, 268-277, 279-280, 282-297, 299, 301-303, 305-320, 322-330, 332-337, 339-341]
Producer node: 172.31.2.134:9300	Node UUID: 8cb6f622ccb07b97de85c9bedb4434c2
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 101
	v: [911-934, 936-938, 942-952, 955-971, 974-977, 979-988, 990-993, 995-997, 999-1023]
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 101
	v: [683-696, 698, 700-703, 705, 707-715, 717-721, 723-738, 740-742, 744-753, 755-757, 759-773, 775-777, 779-795]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 101
	v: [797-803, 806-807, 809-811, 813-817, 819-820, 822-824, 826-846, 848-853, 855-858, 860-866, 868-870, 872-904, 906-910]
Producer node: 127.0.0.1:9303	Node UUID: 46adfc38c81b9f50741786c7f6948222
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 101
	v: [570-576, 578-585, 587-592, 594-599, 601, 603, 605-626, 628-630, 632-649, 653-670, 672-682]
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 101
	v: [342-368, 370-376, 378-382, 384, 387-389, 391-393, 395-404, 406-420, 422-425, 427-442, 444-450, 453-455]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 97
	v: [456, 459-462, 464-470, 472, 474-478, 483, 485-490, 493-497, 499-501, 503-504, 507-510, 512-569]


2021/12/11 04:51:28 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 04:51:33 eventingSetup.go:719: Waiting for function: TestBucketDeleteWithRebOut status to change to undeployed
2021/12/11 04:51:33 eventingSetup.go:727: Function: TestBucketDeleteWithRebOut status changed to undeployed
2021/12/11 04:51:38 clusterSetup.go:397: Waiting for app: TestBucketDeleteWithRebOut to get un-deployed
2021/12/11 04:51:38 clusterSetup.go:405: App: TestBucketDeleteWithRebOut got un-deployed
2021/12/11 04:51:38 eventingSetup.go:46: No eventing-con process running
2021/12/11 04:51:38 eventingSetup.go:418: Function: TestBucketDeleteWithRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestBucketDeleteWithRebOut deleting in the background"
}
2021/12/11 04:51:40 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/11 04:51:45 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 04:51:46 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/11 04:51:51 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 04:51:51 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv,index,n1ql to the cluster
2021/12/11 04:51:52 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/11 04:51:57 Rebalance progress: 100
--- PASS: TestBucketDeleteWithRebOut (282.44s)
=== RUN   TestN1QLRecursion
2021/12/11 04:52:00 Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/11 04:52:00 Error in adding nodes : , response: ["Prepare join failed. Node is already part of cluster."]
2021/12/11 04:52:00 Removing node(s): [] from the cluster
2021/12/11 04:52:05 Rebalance progress: 100
2021/12/11 04:52:11 Context: Bucket flush, response code: 200 dump: 
2021/12/11 04:52:17 Context: Bucket flush, response code: 200 dump: 
2021/12/11 04:52:23 Context: Bucket flush, response code: 200 dump: 
2021/12/11 04:52:28 Deploying app: TestN1QLRecursion_function1. Item count src bucket: 0 dst bucket: 0
2021/12/11 04:52:29 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLRecursion_function1' in metakv",
  "warnings": null
 }
}
2021/12/11 04:52:29 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLRecursion_function1' in metakv",
  "warnings": null
 }
} err 
2021/12/11 04:52:34 Waiting for app: TestN1QLRecursion_function1 to get deployed
2021/12/11 04:52:39 Waiting for app: TestN1QLRecursion_function1 to get deployed
2021/12/11 04:52:44 Waiting for app: TestN1QLRecursion_function1 to get deployed
2021/12/11 04:52:49 Waiting for app: TestN1QLRecursion_function1 to get deployed
2021/12/11 04:52:49 App: TestN1QLRecursion_function1 got deployed
2021/12/11 04:52:49 TestN1QLRecursion_function1 is deployed
2021/12/11 04:52:49 Deploying app: TestN1QLRecursion_function3. Item count src bucket: 0 dst bucket: 0
2021/12/11 04:52:50 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLRecursion_function3' in metakv",
  "warnings": null
 }
}
2021/12/11 04:52:50 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLRecursion_function3' in metakv",
  "warnings": null
 }
} err 
2021/12/11 04:52:55 Waiting for app: TestN1QLRecursion_function3 to get deployed
2021/12/11 04:53:00 Waiting for app: TestN1QLRecursion_function3 to get deployed
2021/12/11 04:53:05 Waiting for app: TestN1QLRecursion_function3 to get deployed
2021/12/11 04:53:10 Waiting for app: TestN1QLRecursion_function3 to get deployed
2021/12/11 04:53:10 App: TestN1QLRecursion_function3 got deployed
2021/12/11 04:53:10 TestN1QLRecursion_function3 is deployed
2021/12/11 04:53:10 Deploying app: TestN1QLRecursion_function2. Item count src bucket: 0 dst bucket: 0
2021/12/11 04:53:10 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_INTER_BUCKET_RECURSION",
 "code": 51,
 "description": "Inter bucket recursion error, deployment of current handler will cause inter bucket recursion",
 "attributes": null,
 "runtime_info": {
  "code": 51,
  "info": "Inter bucket recursion error; function: TestN1QLRecursion_function2 causes a cycle involving functions: [TestN1QLRecursion_function3 TestN1QLRecursion_function1], hence deployment is disallowed"
 }
}
2021/12/11 04:53:10 response body {
 "name": "ERR_INTER_BUCKET_RECURSION",
 "code": 51,
 "description": "Inter bucket recursion error, deployment of current handler will cause inter bucket recursion",
 "attributes": null,
 "runtime_info": {
  "code": 51,
  "info": "Inter bucket recursion error; function: TestN1QLRecursion_function2 causes a cycle involving functions: [TestN1QLRecursion_function3 TestN1QLRecursion_function1], hence deployment is disallowed"
 }
} err 
2021/12/11 04:53:10 Success: TestN1QLRecursion_function2 is not deployed
2021/12/11 04:53:11 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 04:53:16 Waiting for app: TestN1QLRecursion_function1 to get un-deployed
2021/12/11 04:53:21 Waiting for app: TestN1QLRecursion_function1 to get un-deployed
2021/12/11 04:53:26 Waiting for app: TestN1QLRecursion_function1 to get un-deployed
2021/12/11 04:53:31 Waiting for app: TestN1QLRecursion_function1 to get un-deployed
2021/12/11 04:53:31 App: TestN1QLRecursion_function1 got un-deployed
2021/12/11 04:53:31 No eventing-con process running
2021/12/11 04:53:31 Function: TestN1QLRecursion_function1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLRecursion_function1 deleting in the background"
}
2021/12/11 04:53:32 Context: Bucket flush, response code: 200 dump: 
2021/12/11 04:53:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 04:53:39 Context: Bucket flush, response code: 200 dump: 
2021/12/11 04:53:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 04:53:44 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 04:53:49 Waiting for app: TestN1QLRecursion_function3 to get un-deployed
2021/12/11 04:53:55 Waiting for app: TestN1QLRecursion_function3 to get un-deployed
2021/12/11 04:54:00 Waiting for app: TestN1QLRecursion_function3 to get un-deployed
2021/12/11 04:54:05 Waiting for app: TestN1QLRecursion_function3 to get un-deployed
2021/12/11 04:54:05 App: TestN1QLRecursion_function3 got un-deployed
2021/12/11 04:54:05 No eventing-con process running
2021/12/11 04:54:05 Function: TestN1QLRecursion_function3 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLRecursion_function3 deleting in the background"
}
2021/12/11 04:54:06 Context: Bucket flush, response code: 200 dump: 
2021/12/11 04:54:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 04:54:13 Context: Bucket flush, response code: 200 dump: 
2021/12/11 04:54:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 04:54:20 Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/11 04:54:25 Rebalance progress: 100
--- PASS: TestN1QLRecursion (145.10s)
=== RUN   TestN1QLAllowRecursion
2021/12/11 04:54:25 Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/11 04:54:25 Error in adding nodes : , response: ["Prepare join failed. Node is already part of cluster."]
2021/12/11 04:54:25 Removing node(s): [] from the cluster
2021/12/11 04:54:30 Rebalance progress: 100
2021/12/11 04:54:36 Context: Bucket flush, response code: 200 dump: 
2021/12/11 04:54:42 Context: Bucket flush, response code: 200 dump: 
2021/12/11 04:54:48 Context: Bucket flush, response code: 200 dump: 
2021/12/11 04:54:53 Deploying app: TestN1QLAllowRecursion_function1. Item count src bucket: 0 dst bucket: 0
2021/12/11 04:54:54 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function1' in metakv",
  "warnings": null
 }
}
2021/12/11 04:54:54 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function1' in metakv",
  "warnings": null
 }
} err 
2021/12/11 04:54:59 Waiting for app: TestN1QLAllowRecursion_function1 to get deployed
2021/12/11 04:55:04 Waiting for app: TestN1QLAllowRecursion_function1 to get deployed
2021/12/11 04:55:09 Waiting for app: TestN1QLAllowRecursion_function1 to get deployed
2021/12/11 04:55:14 Waiting for app: TestN1QLAllowRecursion_function1 to get deployed
2021/12/11 04:55:14 App: TestN1QLAllowRecursion_function1 got deployed
2021/12/11 04:55:14 TestN1QLAllowRecursion_function1 is deployed
2021/12/11 04:55:14 Deploying app: TestN1QLAllowRecursion_function3. Item count src bucket: 0 dst bucket: 0
2021/12/11 04:55:15 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function3' in metakv",
  "warnings": null
 }
}
2021/12/11 04:55:15 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function3' in metakv",
  "warnings": null
 }
} err 
2021/12/11 04:55:20 Waiting for app: TestN1QLAllowRecursion_function3 to get deployed
2021/12/11 04:55:25 Waiting for app: TestN1QLAllowRecursion_function3 to get deployed
2021/12/11 04:55:30 Waiting for app: TestN1QLAllowRecursion_function3 to get deployed
2021/12/11 04:55:35 Waiting for app: TestN1QLAllowRecursion_function3 to get deployed
2021/12/11 04:55:35 App: TestN1QLAllowRecursion_function3 got deployed
2021/12/11 04:55:35 TestN1QLAllowRecursion_function3 is deployed
2021/12/11 04:55:35 Deploying app: TestN1QLAllowRecursion_function2. Item count src bucket: 0 dst bucket: 0
2021/12/11 04:55:36 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function2' in metakv",
  "warnings": null
 }
}
2021/12/11 04:55:36 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function2' in metakv",
  "warnings": null
 }
} err 
2021/12/11 04:55:41 Waiting for app: TestN1QLAllowRecursion_function2 to get deployed
2021/12/11 04:55:46 Waiting for app: TestN1QLAllowRecursion_function2 to get deployed
2021/12/11 04:55:51 Waiting for app: TestN1QLAllowRecursion_function2 to get deployed
2021/12/11 04:55:57 Waiting for app: TestN1QLAllowRecursion_function2 to get deployed
2021/12/11 04:55:57 App: TestN1QLAllowRecursion_function2 got deployed
2021/12/11 04:55:57 Success: TestN1QLAllowRecursion_function2 is deployed after allow_interbucket_recursion
2021/12/11 04:55:57 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 04:56:02 Waiting for app: TestN1QLAllowRecursion_function1 to get un-deployed
2021/12/11 04:56:07 Waiting for app: TestN1QLAllowRecursion_function1 to get un-deployed
2021/12/11 04:56:12 Waiting for app: TestN1QLAllowRecursion_function1 to get un-deployed
2021/12/11 04:56:17 Waiting for app: TestN1QLAllowRecursion_function1 to get un-deployed
2021/12/11 04:56:17 App: TestN1QLAllowRecursion_function1 got un-deployed
2021/12/11 04:56:17 No eventing-con process running
2021/12/11 04:56:17 Function: TestN1QLAllowRecursion_function1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLAllowRecursion_function1 deleting in the background"
}
2021/12/11 04:56:19 Context: Bucket flush, response code: 200 dump: 
2021/12/11 04:56:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 04:56:25 Context: Bucket flush, response code: 200 dump: 
2021/12/11 04:56:30 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 04:56:31 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 04:56:36 Waiting for app: TestN1QLAllowRecursion_function2 to get un-deployed
2021/12/11 04:56:41 Waiting for app: TestN1QLAllowRecursion_function2 to get un-deployed
2021/12/11 04:56:46 Waiting for app: TestN1QLAllowRecursion_function2 to get un-deployed
2021/12/11 04:56:51 Waiting for app: TestN1QLAllowRecursion_function2 to get un-deployed
2021/12/11 04:56:51 App: TestN1QLAllowRecursion_function2 got un-deployed
2021/12/11 04:56:51 No eventing-con process running
2021/12/11 04:56:51 Function: TestN1QLAllowRecursion_function2 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLAllowRecursion_function2 deleting in the background"
}
2021/12/11 04:56:53 Context: Bucket flush, response code: 200 dump: 
2021/12/11 04:56:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 04:57:00 Context: Bucket flush, response code: 200 dump: 
2021/12/11 04:57:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 04:57:05 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 04:57:10 Waiting for app: TestN1QLAllowRecursion_function3 to get un-deployed
2021/12/11 04:57:15 Waiting for app: TestN1QLAllowRecursion_function3 to get un-deployed
2021/12/11 04:57:20 Waiting for app: TestN1QLAllowRecursion_function3 to get un-deployed
2021/12/11 04:57:25 Waiting for app: TestN1QLAllowRecursion_function3 to get un-deployed
2021/12/11 04:57:25 App: TestN1QLAllowRecursion_function3 got un-deployed
2021/12/11 04:57:25 No eventing-con process running
2021/12/11 04:57:25 Function: TestN1QLAllowRecursion_function3 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLAllowRecursion_function3 deleting in the background"
}
2021/12/11 04:57:27 Context: Bucket flush, response code: 200 dump: 
2021/12/11 04:57:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 04:57:34 Context: Bucket flush, response code: 200 dump: 
2021/12/11 04:57:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 04:57:40 Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/11 04:57:45 Rebalance progress: 100
--- PASS: TestN1QLAllowRecursion (200.15s)
=== RUN   TestOnDeleteExpiryBucketOp
2021/12/11 04:57:45 Sleeping for some time
2021/12/11 04:57:55 Waiting for app: TestOnDeleteExpiryBucketOp to get un-deployed
2021/12/11 04:57:55 App: TestOnDeleteExpiryBucketOp got un-deployed
2021/12/11 04:57:55 No eventing-con process running
2021/12/11 04:57:55 Function: TestOnDeleteExpiryBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnDeleteExpiryBucketOp not found"
 }
}
2021/12/11 04:57:57 Context: Bucket flush, response code: 200 dump: 
2021/12/11 04:58:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 04:58:04 Context: Bucket flush, response code: 200 dump: 
2021/12/11 04:58:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 04:58:09 Deploying app: TestOnDeleteExpiryBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/11 04:58:10 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnDeleteExpiryBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/11 04:58:10 Starting bucket ops to source bucket
2021/12/11 04:58:59 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 04:59:04 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 04:59:04 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/11 04:59:04 Starting bucket ops to source bucket
2021/12/11 04:59:05 Starting bucket ops to source bucket
2021/12/11 05:00:11 Waiting for dst bucket item count to get to: 7000 curr count: 5000
2021/12/11 05:00:11 src & dst bucket item count matched up. src bucket count: 7000 dst bucket count: 7000
2021/12/11 05:00:11 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 05:00:11 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 05:00:16 Waiting for app: TestOnDeleteExpiryBucketOp to get un-deployed
2021/12/11 05:00:21 Waiting for app: TestOnDeleteExpiryBucketOp to get un-deployed
2021/12/11 05:00:26 Waiting for app: TestOnDeleteExpiryBucketOp to get un-deployed
2021/12/11 05:00:31 Waiting for app: TestOnDeleteExpiryBucketOp to get un-deployed
2021/12/11 05:00:31 App: TestOnDeleteExpiryBucketOp got un-deployed
2021/12/11 05:00:31 No eventing-con process running
2021/12/11 05:00:31 Function: TestOnDeleteExpiryBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnDeleteExpiryBucketOp deleting in the background"
}
2021/12/11 05:00:33 Context: Bucket flush, response code: 200 dump: 
2021/12/11 05:00:43 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/11 05:00:48 Waiting for dst bucket item count to get to: 0 curr count: 501
2021/12/11 05:00:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 05:00:50 Context: Bucket flush, response code: 200 dump: 
2021/12/11 05:01:00 Waiting for dst bucket item count to get to: 0 curr count: 4142
2021/12/11 05:01:05 Waiting for dst bucket item count to get to: 0 curr count: 4142
2021/12/11 05:01:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnDeleteExpiryBucketOp (200.02s)
=== RUN   TestTimerOverWriteSameReference
2021/12/11 05:01:10 Waiting for app: TestTimerOverWriteSameReference to get un-deployed
2021/12/11 05:01:10 App: TestTimerOverWriteSameReference got un-deployed
2021/12/11 05:01:10 No eventing-con process running
2021/12/11 05:01:10 Function: TestTimerOverWriteSameReference delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestTimerOverWriteSameReference not found"
 }
}
2021/12/11 05:01:12 Context: Bucket flush, response code: 200 dump: 
2021/12/11 05:01:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 05:01:19 Context: Bucket flush, response code: 200 dump: 
2021/12/11 05:01:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 05:01:24 Deploying app: TestTimerOverWriteSameReference. Item count src bucket: 0 dst bucket: 0
2021/12/11 05:01:25 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestTimerOverWriteSameReference' in metakv",
  "warnings": null
 }
}
2021/12/11 05:01:30 Waiting for app: TestTimerOverWriteSameReference to get deployed
2021/12/11 05:01:35 Waiting for app: TestTimerOverWriteSameReference to get deployed
2021/12/11 05:01:40 Waiting for app: TestTimerOverWriteSameReference to get deployed
2021/12/11 05:01:45 Waiting for app: TestTimerOverWriteSameReference to get deployed
2021/12/11 05:01:45 App: TestTimerOverWriteSameReference got deployed
2021/12/11 05:01:55 Starting bucket ops to source bucket
2021/12/11 05:02:56 src & dst bucket item count matched up. src bucket count: 1 dst bucket count: 1
2021/12/11 05:03:06 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 05:03:06 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 05:03:11 Waiting for app: TestTimerOverWriteSameReference to get un-deployed
2021/12/11 05:03:16 Waiting for app: TestTimerOverWriteSameReference to get un-deployed
2021/12/11 05:03:21 Waiting for app: TestTimerOverWriteSameReference to get un-deployed
2021/12/11 05:03:26 Waiting for app: TestTimerOverWriteSameReference to get un-deployed
2021/12/11 05:03:26 App: TestTimerOverWriteSameReference got un-deployed
2021/12/11 05:03:26 No eventing-con process running
2021/12/11 05:03:26 Function: TestTimerOverWriteSameReference delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestTimerOverWriteSameReference deleting in the background"
}
2021/12/11 05:03:28 Context: Bucket flush, response code: 200 dump: 
2021/12/11 05:03:38 Waiting for dst bucket item count to get to: 0 curr count: 614
2021/12/11 05:03:43 Waiting for dst bucket item count to get to: 0 curr count: 114
2021/12/11 05:03:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 05:03:45 Context: Bucket flush, response code: 200 dump: 
2021/12/11 05:03:55 Waiting for dst bucket item count to get to: 0 curr count: 1
2021/12/11 05:03:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestTimerOverWriteSameReference (169.80s)
=== RUN   TestCancelTimerBucketop
2021/12/11 05:04:00 Waiting for app: TestCancelTimerBucketop to get un-deployed
2021/12/11 05:04:00 App: TestCancelTimerBucketop got un-deployed
2021/12/11 05:04:00 No eventing-con process running
2021/12/11 05:04:00 Function: TestCancelTimerBucketop delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestCancelTimerBucketop not found"
 }
}
2021/12/11 05:04:02 Context: Bucket flush, response code: 200 dump: 
2021/12/11 05:04:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 05:04:08 Context: Bucket flush, response code: 200 dump: 
2021/12/11 05:04:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 05:04:13 Deploying app: TestCancelTimerBucketop. Item count src bucket: 0 dst bucket: 0
2021/12/11 05:04:14 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCancelTimerBucketop' in metakv",
  "warnings": null
 }
}
2021/12/11 05:04:19 Waiting for app: TestCancelTimerBucketop to get deployed
2021/12/11 05:04:24 Waiting for app: TestCancelTimerBucketop to get deployed
2021/12/11 05:04:29 Waiting for app: TestCancelTimerBucketop to get deployed
2021/12/11 05:04:34 Waiting for app: TestCancelTimerBucketop to get deployed
2021/12/11 05:04:34 App: TestCancelTimerBucketop got deployed
2021/12/11 05:04:34 Starting bucket ops to source bucket
2021/12/11 05:04:42 Starting bucket ops to source bucket
2021/12/11 05:05:44 src & dst bucket item count matched up. src bucket count: 1000 dst bucket count: 1000
2021/12/11 05:05:44 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 05:05:44 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 05:05:49 Waiting for app: TestCancelTimerBucketop to get un-deployed
2021/12/11 05:05:54 Waiting for app: TestCancelTimerBucketop to get un-deployed
2021/12/11 05:05:59 Waiting for app: TestCancelTimerBucketop to get un-deployed
2021/12/11 05:06:04 Waiting for app: TestCancelTimerBucketop to get un-deployed
2021/12/11 05:06:04 App: TestCancelTimerBucketop got un-deployed
2021/12/11 05:06:04 No eventing-con process running
2021/12/11 05:06:04 Function: TestCancelTimerBucketop delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCancelTimerBucketop deleting in the background"
}
2021/12/11 05:06:06 Context: Bucket flush, response code: 200 dump: 
2021/12/11 05:06:16 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/11 05:06:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 05:06:18 Context: Bucket flush, response code: 200 dump: 
2021/12/11 05:06:28 Waiting for dst bucket item count to get to: 0 curr count: 628
2021/12/11 05:06:33 Waiting for dst bucket item count to get to: 0 curr count: 128
2021/12/11 05:06:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCancelTimerBucketop (158.11s)
=== RUN   TestJSExpiryDate
2021/12/11 05:06:38 Starting bucket ops to source bucket
2021/12/11 05:06:56 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/11 05:06:56 Deploying app: TestJSExpiryDate. Item count src bucket: 5000 dst bucket: 0
2021/12/11 05:06:57 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestJSExpiryDate' in metakv",
  "warnings": null
 }
}
2021/12/11 05:07:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 05:07:27 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 05:07:27 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 05:07:32 Waiting for app: TestJSExpiryDate to get un-deployed
2021/12/11 05:07:37 Waiting for app: TestJSExpiryDate to get un-deployed
2021/12/11 05:07:42 Waiting for app: TestJSExpiryDate to get un-deployed
2021/12/11 05:07:47 Waiting for app: TestJSExpiryDate to get un-deployed
2021/12/11 05:07:47 App: TestJSExpiryDate got un-deployed
2021/12/11 05:07:47 No eventing-con process running
2021/12/11 05:07:47 Function: TestJSExpiryDate delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestJSExpiryDate deleting in the background"
}
2021/12/11 05:07:49 Context: Bucket flush, response code: 200 dump: 
2021/12/11 05:07:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 05:07:56 Context: Bucket flush, response code: 200 dump: 
2021/12/11 05:08:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestJSExpiryDate (88.64s)
=== RUN   TestBinaryDoc
2021/12/11 05:08:06 Starting bucket ops to source bucket
2021/12/11 05:08:07 Deploying app: TestBinaryDoc. Item count src bucket: 0 dst bucket: 0
2021/12/11 05:08:08 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestBinaryDoc' in metakv",
  "warnings": null
 }
}
2021/12/11 05:08:08 Starting bucket ops to source bucket
2021/12/11 05:08:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 05:08:08 Deleting source bucket docs now
2021/12/11 05:08:08 Starting bucket ops to source bucket
2021/12/11 05:08:45 Waiting for dst bucket item count to get to: 32 curr count: 20
2021/12/11 05:08:50 Waiting for dst bucket item count to get to: 32 curr count: 31
2021/12/11 05:08:55 Waiting for dst bucket item count to get to: 32 curr count: 31
2021/12/11 05:08:55 src & dst bucket item count matched up. src bucket count: 32 dst bucket count: 32
2021/12/11 05:08:55 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 05:08:55 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 05:09:00 Waiting for app: TestBinaryDoc to get un-deployed
2021/12/11 05:09:05 Waiting for app: TestBinaryDoc to get un-deployed
2021/12/11 05:09:10 Waiting for app: TestBinaryDoc to get un-deployed
2021/12/11 05:09:15 Waiting for app: TestBinaryDoc to get un-deployed
2021/12/11 05:09:15 App: TestBinaryDoc got un-deployed
2021/12/11 05:09:15 No eventing-con process running
2021/12/11 05:09:15 Function: TestBinaryDoc delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestBinaryDoc deleting in the background"
}
2021/12/11 05:09:17 Context: Bucket flush, response code: 200 dump: 
2021/12/11 05:09:27 Waiting for dst bucket item count to get to: 0 curr count: 32
2021/12/11 05:09:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 05:09:29 Context: Bucket flush, response code: 200 dump: 
2021/12/11 05:09:39 Waiting for dst bucket item count to get to: 0 curr count: 30
2021/12/11 05:09:44 Waiting for dst bucket item count to get to: 0 curr count: 14
2021/12/11 05:09:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestBinaryDoc (102.33s)
=== RUN   TestConstantBindings
2021/12/11 05:09:44 Deploying app: TestConstantBindings. Item count src bucket: 0 dst bucket: 0
2021/12/11 05:09:45 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestConstantBindings' in metakv",
  "warnings": null
 }
}
2021/12/11 05:09:45 Starting bucket ops to source bucket
2021/12/11 05:09:51 Waiting for dst bucket item count to get to: 6000 curr count: 0
2021/12/11 05:09:56 Waiting for dst bucket item count to get to: 6000 curr count: 0
2021/12/11 05:10:01 Waiting for dst bucket item count to get to: 6000 curr count: 0
2021/12/11 05:10:06 Waiting for dst bucket item count to get to: 6000 curr count: 904
2021/12/11 05:10:11 Waiting for dst bucket item count to get to: 6000 curr count: 3904
2021/12/11 05:10:11 src & dst bucket item count matched up. src bucket count: 6000 dst bucket count: 6000
2021/12/11 05:10:11 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 05:10:11 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 05:10:16 Waiting for app: TestConstantBindings to get un-deployed
2021/12/11 05:10:21 Waiting for app: TestConstantBindings to get un-deployed
2021/12/11 05:10:26 Waiting for app: TestConstantBindings to get un-deployed
2021/12/11 05:10:31 Waiting for app: TestConstantBindings to get un-deployed
2021/12/11 05:10:31 App: TestConstantBindings got un-deployed
2021/12/11 05:10:31 No eventing-con process running
2021/12/11 05:10:31 Function: TestConstantBindings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestConstantBindings deleting in the background"
}
2021/12/11 05:10:33 Context: Bucket flush, response code: 200 dump: 
2021/12/11 05:10:43 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/11 05:10:48 Waiting for dst bucket item count to get to: 0 curr count: 500
2021/12/11 05:10:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 05:10:49 Context: Bucket flush, response code: 200 dump: 
2021/12/11 05:10:59 Waiting for dst bucket item count to get to: 0 curr count: 3000
2021/12/11 05:11:04 Waiting for dst bucket item count to get to: 0 curr count: 3000
2021/12/11 05:11:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestConstantBindings (80.66s)
=== RUN   TestCodeInsight
2021/12/11 05:11:09 Waiting for app: TestCodeInsight to get un-deployed
2021/12/11 05:11:09 App: TestCodeInsight got un-deployed
2021/12/11 05:11:09 No eventing-con process running
2021/12/11 05:11:09 Function: TestCodeInsight delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestCodeInsight not found"
 }
}
2021/12/11 05:11:11 Context: Bucket flush, response code: 200 dump: 
2021/12/11 05:11:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 05:11:18 Context: Bucket flush, response code: 200 dump: 
2021/12/11 05:11:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 05:11:23 Starting bucket ops to source bucket
2021/12/11 05:11:24 Deploying app: TestCodeInsight. Item count src bucket: 0 dst bucket: 0
2021/12/11 05:11:25 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCodeInsight' in metakv",
  "warnings": null
 }
}
2021/12/11 05:11:30 Waiting for app: TestCodeInsight to get deployed
2021/12/11 05:11:35 Waiting for app: TestCodeInsight to get deployed
2021/12/11 05:11:40 Waiting for app: TestCodeInsight to get deployed
2021/12/11 05:11:45 Waiting for app: TestCodeInsight to get deployed
2021/12/11 05:11:45 App: TestCodeInsight got deployed
2021/12/11 05:11:50 Waiting for dst bucket item count to get to: 1000 curr count: 770
2021/12/11 05:11:50 src & dst bucket item count matched up. src bucket count: 1000 dst bucket count: 1000
2021/12/11 05:11:50 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 05:11:55 Waiting for app: TestCodeInsight to get un-deployed
2021/12/11 05:12:00 Waiting for app: TestCodeInsight to get un-deployed
2021/12/11 05:12:05 Waiting for app: TestCodeInsight to get un-deployed
2021/12/11 05:12:10 Waiting for app: TestCodeInsight to get un-deployed
2021/12/11 05:12:10 App: TestCodeInsight got un-deployed
2021/12/11 05:12:10 No eventing-con process running
2021/12/11 05:12:10 Function: TestCodeInsight delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCodeInsight deleting in the background"
}
2021/12/11 05:12:12 Context: Bucket flush, response code: 200 dump: 
2021/12/11 05:12:22 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/11 05:12:27 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/11 05:12:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 05:12:29 Context: Bucket flush, response code: 200 dump: 
2021/12/11 05:12:39 Waiting for dst bucket item count to get to: 0 curr count: 954
2021/12/11 05:12:44 Waiting for dst bucket item count to get to: 0 curr count: 454
2021/12/11 05:12:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCodeInsight (99.34s)
=== RUN   TestParserTransform
--- PASS: TestParserTransform (0.04s)
=== RUN   TestParserDetect
--- PASS: TestParserDetect (0.00s)
PASS
ok  	github.com/couchbase/eventing/tests/functional_tests	12858.157s

Functional tests took 3 hours, 34 minutes and 22 seconds.

N1ql tests

2021/12/11 05:12:55 Started n1ql
2021/12/11 05:13:04 Adding node: https://127.0.0.1:19001 with role: kv,index,n1ql to the cluster
2021/12/11 05:13:05 Error in adding nodes : , response: ["Prepare join failed. Node is already part of cluster."]
2021/12/11 05:13:05 Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/11 05:13:05 Removing node(s): [] from the cluster
2021/12/11 05:13:10 Rebalance progress: 100
=== RUN   TestRecursiveMutationN1QL
2021/12/11 05:13:15 Waiting for app: TestRecursiveMutationN1QL to get un-deployed
2021/12/11 05:13:15 App: TestRecursiveMutationN1QL got un-deployed
2021/12/11 05:13:15 No eventing-con process running
2021/12/11 05:13:15 Function: TestRecursiveMutationN1QL delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestRecursiveMutationN1QL not found"
 }
}
2021/12/11 05:13:17 Context: Bucket flush, response code: 200 dump: 
2021/12/11 05:13:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 05:13:24 Context: Bucket flush, response code: 200 dump: 
2021/12/11 05:13:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 05:13:29 Deploying app: TestRecursiveMutationN1QL. Item count src bucket: 0 dst bucket: 0
2021/12/11 05:13:29 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_INTER_BUCKET_RECURSION",
 "code": 51,
 "description": "Inter bucket recursion error, deployment of current handler will cause inter bucket recursion",
 "attributes": null,
 "runtime_info": {
  "code": 51,
  "info": "Function: TestRecursiveMutationN1QL N1QL DML to source keyspace default:default"
 }
}
--- PASS: TestRecursiveMutationN1QL (18.49s)
=== RUN   TestFlexReset1
2021/12/11 05:13:34 Waiting for app: TestFlexReset1 to get un-deployed
2021/12/11 05:13:34 App: TestFlexReset1 got un-deployed
2021/12/11 05:13:34 No eventing-con process running
2021/12/11 05:13:34 Function: TestFlexReset1 delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestFlexReset1 not found"
 }
}
2021/12/11 05:13:35 Context: Bucket flush, response code: 200 dump: 
2021/12/11 05:13:40 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 05:13:42 Context: Bucket flush, response code: 200 dump: 
2021/12/11 05:13:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 05:13:47 Starting bucket ops to source bucket
2021/12/11 05:13:48 Deploying app: TestFlexReset1. Item count src bucket: 0 dst bucket: 0
2021/12/11 05:13:49 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestFlexReset1' in metakv",
  "warnings": null
 }
}
2021/12/11 05:13:54 Waiting for app: TestFlexReset1 to get deployed
2021/12/11 05:13:59 Waiting for app: TestFlexReset1 to get deployed
2021/12/11 05:14:04 Waiting for app: TestFlexReset1 to get deployed
2021/12/11 05:14:09 Waiting for app: TestFlexReset1 to get deployed
2021/12/11 05:14:09 App: TestFlexReset1 got deployed
2021/12/11 05:14:14 Waiting for dst bucket item count to get to: 1000 curr count: 500
2021/12/11 05:14:14 src & dst bucket item count matched up. src bucket count: 1000 dst bucket count: 1000
2021/12/11 05:14:14 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 05:14:14 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 05:14:19 Waiting for app: TestFlexReset1 to get un-deployed
2021/12/11 05:14:24 Waiting for app: TestFlexReset1 to get un-deployed
2021/12/11 05:14:29 Waiting for app: TestFlexReset1 to get un-deployed
2021/12/11 05:14:34 Waiting for app: TestFlexReset1 to get un-deployed
2021/12/11 05:14:34 App: TestFlexReset1 got un-deployed
2021/12/11 05:14:34 No eventing-con process running
2021/12/11 05:14:34 Function: TestFlexReset1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestFlexReset1 deleting in the background"
}
2021/12/11 05:14:36 Context: Bucket flush, response code: 200 dump: 
2021/12/11 05:14:46 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/11 05:14:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 05:14:48 Context: Bucket flush, response code: 200 dump: 
2021/12/11 05:14:58 Waiting for dst bucket item count to get to: 0 curr count: 604
2021/12/11 05:15:03 Waiting for dst bucket item count to get to: 0 curr count: 104
2021/12/11 05:15:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestFlexReset1 (94.22s)
=== RUN   TestFlexReset2
2021/12/11 05:15:08 Waiting for app: TestFlexReset2 to get un-deployed
2021/12/11 05:15:08 App: TestFlexReset2 got un-deployed
2021/12/11 05:15:08 No eventing-con process running
2021/12/11 05:15:08 Function: TestFlexReset2 delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestFlexReset2 not found"
 }
}
2021/12/11 05:15:10 Context: Bucket flush, response code: 200 dump: 
2021/12/11 05:15:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 05:15:16 Context: Bucket flush, response code: 200 dump: 
2021/12/11 05:15:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 05:15:21 Starting bucket ops to source bucket
2021/12/11 05:15:22 Deploying app: TestFlexReset2. Item count src bucket: 0 dst bucket: 0
2021/12/11 05:15:23 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestFlexReset2' in metakv",
  "warnings": null
 }
}
2021/12/11 05:15:28 Waiting for app: TestFlexReset2 to get deployed
2021/12/11 05:15:33 Waiting for app: TestFlexReset2 to get deployed
2021/12/11 05:15:38 Waiting for app: TestFlexReset2 to get deployed
2021/12/11 05:15:43 Waiting for app: TestFlexReset2 to get deployed
2021/12/11 05:15:43 App: TestFlexReset2 got deployed
2021/12/11 05:15:48 Waiting for dst bucket item count to get to: 1000 curr count: 805
2021/12/11 05:15:53 Waiting for dst bucket item count to get to: 1000 curr count: 805
2021/12/11 05:15:53 src & dst bucket item count matched up. src bucket count: 1000 dst bucket count: 1000
2021/12/11 05:15:53 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 05:15:53 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 05:15:58 Waiting for app: TestFlexReset2 to get un-deployed
2021/12/11 05:16:03 Waiting for app: TestFlexReset2 to get un-deployed
2021/12/11 05:16:08 Waiting for app: TestFlexReset2 to get un-deployed
2021/12/11 05:16:13 Waiting for app: TestFlexReset2 to get un-deployed
2021/12/11 05:16:13 App: TestFlexReset2 got un-deployed
2021/12/11 05:16:13 No eventing-con process running
2021/12/11 05:16:14 Function: TestFlexReset2 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestFlexReset2 deleting in the background"
}
2021/12/11 05:16:15 Context: Bucket flush, response code: 200 dump: 
2021/12/11 05:16:25 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/11 05:16:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 05:16:27 Context: Bucket flush, response code: 200 dump: 
2021/12/11 05:16:37 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/11 05:16:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestFlexReset2 (94.26s)
=== RUN   TestN1QLLabelledBreak
2021/12/11 05:16:42 Waiting for app: TestN1QLLabelledBreak to get un-deployed
2021/12/11 05:16:42 App: TestN1QLLabelledBreak got un-deployed
2021/12/11 05:16:42 No eventing-con process running
2021/12/11 05:16:42 Function: TestN1QLLabelledBreak delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLLabelledBreak not found"
 }
}
2021/12/11 05:16:44 Context: Bucket flush, response code: 200 dump: 
2021/12/11 05:16:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 05:16:51 Context: Bucket flush, response code: 200 dump: 
2021/12/11 05:16:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 05:16:56 Deploying app: TestN1QLLabelledBreak. Item count src bucket: 0 dst bucket: 0
2021/12/11 05:16:57 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLLabelledBreak' in metakv",
  "warnings": null
 }
}
2021/12/11 05:17:02 Waiting for app: TestN1QLLabelledBreak to get deployed
2021/12/11 05:17:07 Waiting for app: TestN1QLLabelledBreak to get deployed
2021/12/11 05:17:12 Waiting for app: TestN1QLLabelledBreak to get deployed
2021/12/11 05:17:17 Waiting for app: TestN1QLLabelledBreak to get deployed
2021/12/11 05:17:17 App: TestN1QLLabelledBreak got deployed
2021/12/11 05:17:17 Starting bucket ops to source bucket
2021/12/11 05:17:26 Waiting for dst bucket item count to get to: 10000 curr count: 0
2021/12/11 05:17:31 Waiting for dst bucket item count to get to: 10000 curr count: 166
2021/12/11 05:17:36 Waiting for dst bucket item count to get to: 10000 curr count: 166
2021/12/11 05:17:41 Waiting for dst bucket item count to get to: 10000 curr count: 3408
2021/12/11 05:17:46 Waiting for dst bucket item count to get to: 10000 curr count: 3408
2021/12/11 05:17:51 Waiting for dst bucket item count to get to: 10000 curr count: 7489
2021/12/11 05:17:56 Waiting for dst bucket item count to get to: 10000 curr count: 7489
2021/12/11 05:17:56 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/11 05:17:56 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 05:17:56 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 05:18:01 Waiting for app: TestN1QLLabelledBreak to get un-deployed
2021/12/11 05:18:06 Waiting for app: TestN1QLLabelledBreak to get un-deployed
2021/12/11 05:18:11 Waiting for app: TestN1QLLabelledBreak to get un-deployed
2021/12/11 05:18:16 Waiting for app: TestN1QLLabelledBreak to get un-deployed
2021/12/11 05:18:16 App: TestN1QLLabelledBreak got un-deployed
2021/12/11 05:18:16 No eventing-con process running
2021/12/11 05:18:16 Function: TestN1QLLabelledBreak delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLLabelledBreak deleting in the background"
}
2021/12/11 05:18:18 Context: Bucket flush, response code: 200 dump: 
2021/12/11 05:18:28 Waiting for dst bucket item count to get to: 0 curr count: 3673
2021/12/11 05:18:33 Waiting for dst bucket item count to get to: 0 curr count: 1173
2021/12/11 05:18:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 05:18:35 Context: Bucket flush, response code: 200 dump: 
2021/12/11 05:18:45 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/11 05:18:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLLabelledBreak (128.05s)
=== RUN   TestN1QLUnlabelledBreak
2021/12/11 05:18:50 Waiting for app: TestN1QLUnlabelledBreak to get un-deployed
2021/12/11 05:18:50 App: TestN1QLUnlabelledBreak got un-deployed
2021/12/11 05:18:50 No eventing-con process running
2021/12/11 05:18:50 Function: TestN1QLUnlabelledBreak delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLUnlabelledBreak not found"
 }
}
2021/12/11 05:18:52 Context: Bucket flush, response code: 200 dump: 
2021/12/11 05:18:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 05:18:59 Context: Bucket flush, response code: 200 dump: 
2021/12/11 05:19:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 05:19:04 Deploying app: TestN1QLUnlabelledBreak. Item count src bucket: 0 dst bucket: 0
2021/12/11 05:19:05 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLUnlabelledBreak' in metakv",
  "warnings": null
 }
}
2021/12/11 05:19:10 Waiting for app: TestN1QLUnlabelledBreak to get deployed
2021/12/11 05:19:15 Waiting for app: TestN1QLUnlabelledBreak to get deployed
2021/12/11 05:19:20 Waiting for app: TestN1QLUnlabelledBreak to get deployed
2021/12/11 05:19:25 Waiting for app: TestN1QLUnlabelledBreak to get deployed
2021/12/11 05:19:25 App: TestN1QLUnlabelledBreak got deployed
2021/12/11 05:19:25 Starting bucket ops to source bucket
2021/12/11 05:19:34 Waiting for dst bucket item count to get to: 10000 curr count: 0
2021/12/11 05:19:39 Waiting for dst bucket item count to get to: 10000 curr count: 274
2021/12/11 05:19:44 Waiting for dst bucket item count to get to: 10000 curr count: 807
2021/12/11 05:19:49 Waiting for dst bucket item count to get to: 10000 curr count: 3357
2021/12/11 05:19:54 Waiting for dst bucket item count to get to: 10000 curr count: 6273
2021/12/11 05:19:54 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/11 05:19:54 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 05:19:54 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 05:19:59 Waiting for app: TestN1QLUnlabelledBreak to get un-deployed
2021/12/11 05:20:04 Waiting for app: TestN1QLUnlabelledBreak to get un-deployed
2021/12/11 05:20:09 Waiting for app: TestN1QLUnlabelledBreak to get un-deployed
2021/12/11 05:20:14 Waiting for app: TestN1QLUnlabelledBreak to get un-deployed
2021/12/11 05:20:14 App: TestN1QLUnlabelledBreak got un-deployed
2021/12/11 05:20:14 No eventing-con process running
2021/12/11 05:20:14 Function: TestN1QLUnlabelledBreak delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLUnlabelledBreak deleting in the background"
}
2021/12/11 05:20:16 Context: Bucket flush, response code: 200 dump: 
2021/12/11 05:20:26 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/11 05:20:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 05:20:28 Context: Bucket flush, response code: 200 dump: 
2021/12/11 05:20:38 Waiting for dst bucket item count to get to: 0 curr count: 6610
2021/12/11 05:20:43 Waiting for dst bucket item count to get to: 0 curr count: 1610
2021/12/11 05:20:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLUnlabelledBreak (117.85s)
=== RUN   TestN1QLThrowStatement
2021/12/11 05:20:48 Waiting for app: TestN1QLThrowStatement to get un-deployed
2021/12/11 05:20:48 App: TestN1QLThrowStatement got un-deployed
2021/12/11 05:20:48 No eventing-con process running
2021/12/11 05:20:48 Function: TestN1QLThrowStatement delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLThrowStatement not found"
 }
}
2021/12/11 05:20:50 Context: Bucket flush, response code: 200 dump: 
2021/12/11 05:20:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 05:20:56 Context: Bucket flush, response code: 200 dump: 
2021/12/11 05:21:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 05:21:01 Deploying app: TestN1QLThrowStatement. Item count src bucket: 0 dst bucket: 0
2021/12/11 05:21:03 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLThrowStatement' in metakv",
  "warnings": null
 }
}
2021/12/11 05:21:08 Waiting for app: TestN1QLThrowStatement to get deployed
2021/12/11 05:21:13 Waiting for app: TestN1QLThrowStatement to get deployed
2021/12/11 05:21:18 Waiting for app: TestN1QLThrowStatement to get deployed
2021/12/11 05:21:23 Waiting for app: TestN1QLThrowStatement to get deployed
2021/12/11 05:21:23 App: TestN1QLThrowStatement got deployed
2021/12/11 05:21:23 Starting bucket ops to source bucket
2021/12/11 05:21:32 Waiting for dst bucket item count to get to: 10000 curr count: 0
2021/12/11 05:21:37 Waiting for dst bucket item count to get to: 10000 curr count: 0
2021/12/11 05:21:42 Waiting for dst bucket item count to get to: 10000 curr count: 1128
2021/12/11 05:21:47 Waiting for dst bucket item count to get to: 10000 curr count: 1128
2021/12/11 05:21:52 Waiting for dst bucket item count to get to: 10000 curr count: 5249
2021/12/11 05:21:57 Waiting for dst bucket item count to get to: 10000 curr count: 7277
2021/12/11 05:22:02 Waiting for dst bucket item count to get to: 10000 curr count: 9265
2021/12/11 05:22:07 Waiting for dst bucket item count to get to: 10000 curr count: 9836
2021/12/11 05:22:07 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/11 05:22:07 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 05:22:07 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 05:22:12 Waiting for app: TestN1QLThrowStatement to get un-deployed
2021/12/11 05:22:17 Waiting for app: TestN1QLThrowStatement to get un-deployed
2021/12/11 05:22:22 Waiting for app: TestN1QLThrowStatement to get un-deployed
2021/12/11 05:22:27 Waiting for app: TestN1QLThrowStatement to get un-deployed
2021/12/11 05:22:27 App: TestN1QLThrowStatement got un-deployed
2021/12/11 05:22:27 No eventing-con process running
2021/12/11 05:22:27 Function: TestN1QLThrowStatement delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLThrowStatement deleting in the background"
}
2021/12/11 05:22:29 Context: Bucket flush, response code: 200 dump: 
2021/12/11 05:22:39 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/11 05:22:44 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/11 05:22:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 05:22:46 Context: Bucket flush, response code: 200 dump: 
2021/12/11 05:22:56 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/11 05:22:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLThrowStatement (133.00s)
=== RUN   TestN1QLNestedForLoop
2021/12/11 05:23:01 Waiting for app: TestN1QLNestedForLoop to get un-deployed
2021/12/11 05:23:01 App: TestN1QLNestedForLoop got un-deployed
2021/12/11 05:23:01 No eventing-con process running
2021/12/11 05:23:01 Function: TestN1QLNestedForLoop delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLNestedForLoop not found"
 }
}
2021/12/11 05:23:03 Context: Bucket flush, response code: 200 dump: 
2021/12/11 05:23:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 05:23:09 Context: Bucket flush, response code: 200 dump: 
2021/12/11 05:23:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 05:23:15 Deploying app: TestN1QLNestedForLoop. Item count src bucket: 0 dst bucket: 0
2021/12/11 05:23:16 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLNestedForLoop' in metakv",
  "warnings": null
 }
}
2021/12/11 05:23:21 Waiting for app: TestN1QLNestedForLoop to get deployed
2021/12/11 05:23:26 Waiting for app: TestN1QLNestedForLoop to get deployed
2021/12/11 05:23:31 Waiting for app: TestN1QLNestedForLoop to get deployed
2021/12/11 05:23:36 Waiting for app: TestN1QLNestedForLoop to get deployed
2021/12/11 05:23:36 App: TestN1QLNestedForLoop got deployed
2021/12/11 05:23:36 Starting bucket ops to source bucket
2021/12/11 05:23:45 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 05:23:50 Waiting for dst bucket item count to get to: 5000 curr count: 173
2021/12/11 05:23:55 Waiting for dst bucket item count to get to: 5000 curr count: 173
2021/12/11 05:24:00 Waiting for dst bucket item count to get to: 5000 curr count: 2260
2021/12/11 05:24:05 Waiting for dst bucket item count to get to: 5000 curr count: 2260
2021/12/11 05:24:10 Waiting for dst bucket item count to get to: 5000 curr count: 4669
2021/12/11 05:24:15 Waiting for dst bucket item count to get to: 5000 curr count: 4669
2021/12/11 05:24:15 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/11 05:24:15 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 05:24:15 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 05:24:20 Waiting for app: TestN1QLNestedForLoop to get un-deployed
2021/12/11 05:24:25 Waiting for app: TestN1QLNestedForLoop to get un-deployed
2021/12/11 05:24:30 Waiting for app: TestN1QLNestedForLoop to get un-deployed
2021/12/11 05:24:35 Waiting for app: TestN1QLNestedForLoop to get un-deployed
2021/12/11 05:24:35 App: TestN1QLNestedForLoop got un-deployed
2021/12/11 05:24:35 No eventing-con process running
2021/12/11 05:24:35 Function: TestN1QLNestedForLoop delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLNestedForLoop deleting in the background"
}
2021/12/11 05:24:37 Context: Bucket flush, response code: 200 dump: 
2021/12/11 05:24:47 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/11 05:24:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 05:24:49 Context: Bucket flush, response code: 200 dump: 
2021/12/11 05:24:59 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/11 05:25:04 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/11 05:25:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLNestedForLoop (127.98s)
=== RUN   TestN1QLAutoClose
2021/12/11 05:25:09 Waiting for app: TestN1QLAutoClose to get un-deployed
2021/12/11 05:25:09 App: TestN1QLAutoClose got un-deployed
2021/12/11 05:25:09 No eventing-con process running
2021/12/11 05:25:09 Function: TestN1QLAutoClose delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLAutoClose not found"
 }
}
2021/12/11 05:25:11 Context: Bucket flush, response code: 200 dump: 
2021/12/11 05:25:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 05:25:17 Context: Bucket flush, response code: 200 dump: 
2021/12/11 05:25:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 05:25:22 Deploying app: TestN1QLAutoClose. Item count src bucket: 0 dst bucket: 0
2021/12/11 05:25:24 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAutoClose' in metakv",
  "warnings": null
 }
}
2021/12/11 05:25:29 Waiting for app: TestN1QLAutoClose to get deployed
2021/12/11 05:25:34 Waiting for app: TestN1QLAutoClose to get deployed
2021/12/11 05:25:39 Waiting for app: TestN1QLAutoClose to get deployed
2021/12/11 05:25:39 App: TestN1QLAutoClose got deployed
2021/12/11 05:25:39 Starting bucket ops to source bucket
2021/12/11 05:25:44 Waiting for dst bucket item count to get to: 500 curr count: 0
2021/12/11 05:25:49 Waiting for dst bucket item count to get to: 500 curr count: 0
2021/12/11 05:25:54 Waiting for dst bucket item count to get to: 500 curr count: 9
2021/12/11 05:25:59 Waiting for dst bucket item count to get to: 500 curr count: 19
2021/12/11 05:26:04 Waiting for dst bucket item count to get to: 500 curr count: 29
2021/12/11 05:26:09 Waiting for dst bucket item count to get to: 500 curr count: 43
2021/12/11 05:26:14 Waiting for dst bucket item count to get to: 500 curr count: 53
2021/12/11 05:26:19 Waiting for dst bucket item count to get to: 500 curr count: 64
2021/12/11 05:26:24 Waiting for dst bucket item count to get to: 500 curr count: 75
2021/12/11 05:26:29 Waiting for dst bucket item count to get to: 500 curr count: 84
2021/12/11 05:26:34 Waiting for dst bucket item count to get to: 500 curr count: 95
2021/12/11 05:26:39 Waiting for dst bucket item count to get to: 500 curr count: 109
2021/12/11 05:26:44 Waiting for dst bucket item count to get to: 500 curr count: 116
2021/12/11 05:26:49 Waiting for dst bucket item count to get to: 500 curr count: 127
2021/12/11 05:26:54 Waiting for dst bucket item count to get to: 500 curr count: 127
2021/12/11 05:26:59 Waiting for dst bucket item count to get to: 500 curr count: 138
2021/12/11 05:27:04 Waiting for dst bucket item count to get to: 500 curr count: 151
2021/12/11 05:27:09 Waiting for dst bucket item count to get to: 500 curr count: 171
2021/12/11 05:27:14 Waiting for dst bucket item count to get to: 500 curr count: 171
2021/12/11 05:27:19 Waiting for dst bucket item count to get to: 500 curr count: 194
2021/12/11 05:27:24 Waiting for dst bucket item count to get to: 500 curr count: 194
2021/12/11 05:27:29 Waiting for dst bucket item count to get to: 500 curr count: 215
2021/12/11 05:27:34 Waiting for dst bucket item count to get to: 500 curr count: 215
2021/12/11 05:27:39 Waiting for dst bucket item count to get to: 500 curr count: 237
2021/12/11 05:27:44 Waiting for dst bucket item count to get to: 500 curr count: 237
2021/12/11 05:27:49 Waiting for dst bucket item count to get to: 500 curr count: 257
2021/12/11 05:27:54 Waiting for dst bucket item count to get to: 500 curr count: 257
2021/12/11 05:27:59 Waiting for dst bucket item count to get to: 500 curr count: 278
2021/12/11 05:28:04 Waiting for dst bucket item count to get to: 500 curr count: 278
2021/12/11 05:28:09 Waiting for dst bucket item count to get to: 500 curr count: 300
2021/12/11 05:28:14 Waiting for dst bucket item count to get to: 500 curr count: 300
2021/12/11 05:28:19 Waiting for dst bucket item count to get to: 500 curr count: 323
2021/12/11 05:28:24 Waiting for dst bucket item count to get to: 500 curr count: 323
2021/12/11 05:28:29 Waiting for dst bucket item count to get to: 500 curr count: 345
2021/12/11 05:28:34 Waiting for dst bucket item count to get to: 500 curr count: 345
2021/12/11 05:28:39 Waiting for dst bucket item count to get to: 500 curr count: 365
2021/12/11 05:28:44 Waiting for dst bucket item count to get to: 500 curr count: 365
2021/12/11 05:28:49 Waiting for dst bucket item count to get to: 500 curr count: 383
2021/12/11 05:28:55 Waiting for dst bucket item count to get to: 500 curr count: 383
2021/12/11 05:29:00 Waiting for dst bucket item count to get to: 500 curr count: 404
2021/12/11 05:29:05 Waiting for dst bucket item count to get to: 500 curr count: 404
2021/12/11 05:29:10 Waiting for dst bucket item count to get to: 500 curr count: 425
2021/12/11 05:29:15 Waiting for dst bucket item count to get to: 500 curr count: 425
2021/12/11 05:29:20 Waiting for dst bucket item count to get to: 500 curr count: 447
2021/12/11 05:29:25 Waiting for dst bucket item count to get to: 500 curr count: 447
2021/12/11 05:29:30 Waiting for dst bucket item count to get to: 500 curr count: 468
2021/12/11 05:29:35 Waiting for dst bucket item count to get to: 500 curr count: 477
2021/12/11 05:29:40 Waiting for dst bucket item count to get to: 500 curr count: 489
2021/12/11 05:29:45 Waiting for dst bucket item count to get to: 500 curr count: 497
2021/12/11 05:29:45 src & dst bucket item count matched up. src bucket count: 500 dst bucket count: 500
2021/12/11 05:29:45 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 05:29:45 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 05:29:50 Waiting for app: TestN1QLAutoClose to get un-deployed
2021/12/11 05:29:55 Waiting for app: TestN1QLAutoClose to get un-deployed
2021/12/11 05:30:00 Waiting for app: TestN1QLAutoClose to get un-deployed
2021/12/11 05:30:05 Waiting for app: TestN1QLAutoClose to get un-deployed
2021/12/11 05:30:05 App: TestN1QLAutoClose got un-deployed
2021/12/11 05:30:05 No eventing-con process running
2021/12/11 05:30:05 Function: TestN1QLAutoClose delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLAutoClose deleting in the background"
}
2021/12/11 05:30:07 Context: Bucket flush, response code: 200 dump: 
2021/12/11 05:30:17 Waiting for dst bucket item count to get to: 0 curr count: 500
2021/12/11 05:30:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 05:30:18 Context: Bucket flush, response code: 200 dump: 
2021/12/11 05:30:28 Waiting for dst bucket item count to get to: 0 curr count: 414
2021/12/11 05:30:33 Waiting for dst bucket item count to get to: 0 curr count: 164
2021/12/11 05:30:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLAutoClose (329.45s)
=== RUN   TestN1QLPosParams
2021/12/11 05:30:38 Waiting for app: TestN1QLPosParams to get un-deployed
2021/12/11 05:30:38 App: TestN1QLPosParams got un-deployed
2021/12/11 05:30:38 No eventing-con process running
2021/12/11 05:30:38 Function: TestN1QLPosParams delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLPosParams not found"
 }
}
2021/12/11 05:30:40 Context: Bucket flush, response code: 200 dump: 
2021/12/11 05:30:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 05:30:47 Context: Bucket flush, response code: 200 dump: 
2021/12/11 05:30:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 05:30:52 Deploying app: TestN1QLPosParams. Item count src bucket: 0 dst bucket: 0
2021/12/11 05:30:53 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLPosParams' in metakv",
  "warnings": null
 }
}
2021/12/11 05:30:58 Waiting for app: TestN1QLPosParams to get deployed
2021/12/11 05:31:03 Waiting for app: TestN1QLPosParams to get deployed
2021/12/11 05:31:08 Waiting for app: TestN1QLPosParams to get deployed
2021/12/11 05:31:13 Waiting for app: TestN1QLPosParams to get deployed
2021/12/11 05:31:13 App: TestN1QLPosParams got deployed
2021/12/11 05:31:13 Starting bucket ops to source bucket
2021/12/11 05:31:23 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 05:31:28 Waiting for dst bucket item count to get to: 5000 curr count: 644
2021/12/11 05:31:33 Waiting for dst bucket item count to get to: 5000 curr count: 2012
2021/12/11 05:31:38 Waiting for dst bucket item count to get to: 5000 curr count: 3868
2021/12/11 05:31:38 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/11 05:31:38 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 05:31:38 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 05:31:43 Waiting for app: TestN1QLPosParams to get un-deployed
2021/12/11 05:31:48 Waiting for app: TestN1QLPosParams to get un-deployed
2021/12/11 05:31:53 Waiting for app: TestN1QLPosParams to get un-deployed
2021/12/11 05:31:58 Waiting for app: TestN1QLPosParams to get un-deployed
2021/12/11 05:31:58 App: TestN1QLPosParams got un-deployed
2021/12/11 05:31:58 No eventing-con process running
2021/12/11 05:31:58 Function: TestN1QLPosParams delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLPosParams deleting in the background"
}
2021/12/11 05:32:00 Context: Bucket flush, response code: 200 dump: 
2021/12/11 05:32:10 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/11 05:32:15 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/11 05:32:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 05:32:16 Context: Bucket flush, response code: 200 dump: 
2021/12/11 05:32:26 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/11 05:32:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLPosParams (113.02s)
=== RUN   TestN1QLExhaustConnPool
2021/12/11 05:32:31 Waiting for app: TestN1QLExhaustConnPool to get un-deployed
2021/12/11 05:32:31 App: TestN1QLExhaustConnPool got un-deployed
2021/12/11 05:32:31 No eventing-con process running
2021/12/11 05:32:31 Function: TestN1QLExhaustConnPool delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLExhaustConnPool not found"
 }
}
2021/12/11 05:32:33 Context: Bucket flush, response code: 200 dump: 
2021/12/11 05:32:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 05:32:40 Context: Bucket flush, response code: 200 dump: 
2021/12/11 05:32:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 05:32:45 Deploying app: TestN1QLExhaustConnPool. Item count src bucket: 0 dst bucket: 0
2021/12/11 05:32:46 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLExhaustConnPool' in metakv",
  "warnings": null
 }
}
2021/12/11 05:32:51 Waiting for app: TestN1QLExhaustConnPool to get deployed
2021/12/11 05:32:56 Waiting for app: TestN1QLExhaustConnPool to get deployed
2021/12/11 05:33:01 Waiting for app: TestN1QLExhaustConnPool to get deployed
2021/12/11 05:33:06 Waiting for app: TestN1QLExhaustConnPool to get deployed
2021/12/11 05:33:06 App: TestN1QLExhaustConnPool got deployed
2021/12/11 05:33:06 Starting bucket ops to source bucket
2021/12/11 05:33:11 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/11 05:33:16 Waiting for dst bucket item count to get to: 100 curr count: 62
2021/12/11 05:33:21 Waiting for dst bucket item count to get to: 100 curr count: 62
2021/12/11 05:33:21 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/11 05:33:21 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 05:33:21 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 05:33:26 Waiting for app: TestN1QLExhaustConnPool to get un-deployed
2021/12/11 05:33:32 Waiting for app: TestN1QLExhaustConnPool to get un-deployed
2021/12/11 05:33:37 Waiting for app: TestN1QLExhaustConnPool to get un-deployed
2021/12/11 05:33:42 Waiting for app: TestN1QLExhaustConnPool to get un-deployed
2021/12/11 05:33:42 App: TestN1QLExhaustConnPool got un-deployed
2021/12/11 05:33:42 No eventing-con process running
2021/12/11 05:33:42 Function: TestN1QLExhaustConnPool delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLExhaustConnPool deleting in the background"
}
2021/12/11 05:33:43 Context: Bucket flush, response code: 200 dump: 
2021/12/11 05:33:53 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/11 05:33:58 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/11 05:33:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 05:34:00 Context: Bucket flush, response code: 200 dump: 
2021/12/11 05:34:10 Waiting for dst bucket item count to get to: 0 curr count: 65
2021/12/11 05:34:15 Waiting for dst bucket item count to get to: 0 curr count: 65
2021/12/11 05:34:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLExhaustConnPool (108.75s)
=== RUN   TestN1QLGraceSufficientTimeOut
2021/12/11 05:34:20 Waiting for app: TestN1QLGraceSufficientTimeOut to get un-deployed
2021/12/11 05:34:20 App: TestN1QLGraceSufficientTimeOut got un-deployed
2021/12/11 05:34:20 No eventing-con process running
2021/12/11 05:34:20 Function: TestN1QLGraceSufficientTimeOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLGraceSufficientTimeOut not found"
 }
}
2021/12/11 05:34:22 Context: Bucket flush, response code: 200 dump: 
2021/12/11 05:34:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 05:34:29 Context: Bucket flush, response code: 200 dump: 
2021/12/11 05:34:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 05:34:34 Starting bucket ops to source bucket
2021/12/11 05:34:51 Starting bucket ops to source bucket
2021/12/11 05:35:08 Deploying app: TestN1QLGraceSufficientTimeOut. Item count src bucket: 30000 dst bucket: 12757
2021/12/11 05:35:09 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLGraceSufficientTimeOut' in metakv",
  "warnings": null
 }
}
2021/12/11 05:35:14 Waiting for app: TestN1QLGraceSufficientTimeOut to get deployed
2021/12/11 05:35:19 Waiting for app: TestN1QLGraceSufficientTimeOut to get deployed
2021/12/11 05:35:24 Waiting for app: TestN1QLGraceSufficientTimeOut to get deployed
2021/12/11 05:35:29 Waiting for app: TestN1QLGraceSufficientTimeOut to get deployed
2021/12/11 05:35:29 App: TestN1QLGraceSufficientTimeOut got deployed
2021/12/11 05:35:29 Starting bucket ops to source bucket
2021/12/11 05:35:35 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/11 05:35:40 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/11 05:35:45 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/11 05:35:50 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/11 05:35:55 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/11 05:36:00 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/11 05:36:05 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/11 05:36:10 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/11 05:36:15 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/11 05:36:20 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/11 05:36:25 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/11 05:36:25 src & dst bucket item count matched up. src bucket count: 30001 dst bucket count: 30001
2021/12/11 05:36:25 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 05:36:25 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 05:36:30 Waiting for app: TestN1QLGraceSufficientTimeOut to get un-deployed
2021/12/11 05:36:35 Waiting for app: TestN1QLGraceSufficientTimeOut to get un-deployed
2021/12/11 05:36:40 Waiting for app: TestN1QLGraceSufficientTimeOut to get un-deployed
2021/12/11 05:36:45 Waiting for app: TestN1QLGraceSufficientTimeOut to get un-deployed
2021/12/11 05:36:45 App: TestN1QLGraceSufficientTimeOut got un-deployed
2021/12/11 05:36:45 No eventing-con process running
2021/12/11 05:36:45 Function: TestN1QLGraceSufficientTimeOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLGraceSufficientTimeOut deleting in the background"
}
2021/12/11 05:36:47 Context: Bucket flush, response code: 200 dump: 
2021/12/11 05:36:57 Waiting for dst bucket item count to get to: 0 curr count: 30000
2021/12/11 05:36:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 05:36:58 Context: Bucket flush, response code: 200 dump: 
2021/12/11 05:37:09 Waiting for dst bucket item count to get to: 0 curr count: 25950
2021/12/11 05:37:14 Waiting for dst bucket item count to get to: 0 curr count: 10950
2021/12/11 05:37:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLGraceSufficientTimeOut (178.37s)
=== RUN   TestN1QLQueryConsistency
2021/12/11 05:37:19 Waiting for app: n1qlquery_consistency to get un-deployed
2021/12/11 05:37:19 App: n1qlquery_consistency got un-deployed
2021/12/11 05:37:19 No eventing-con process running
2021/12/11 05:37:19 Function: n1qlquery_consistency delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: n1qlquery_consistency not found"
 }
}
2021/12/11 05:37:20 Context: Bucket flush, response code: 200 dump: 
2021/12/11 05:37:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 05:37:27 Context: Bucket flush, response code: 200 dump: 
2021/12/11 05:37:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 05:37:32 Starting bucket ops to source bucket
2021/12/11 05:37:35 Deploying app: n1qlquery_consistency. Item count src bucket: 0 dst bucket: 0
2021/12/11 05:37:36 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'n1qlquery_consistency' in metakv",
  "warnings": null
 }
}
2021/12/11 05:37:41 Waiting for app: n1qlquery_consistency to get deployed
2021/12/11 05:37:46 Waiting for app: n1qlquery_consistency to get deployed
2021/12/11 05:37:51 Waiting for app: n1qlquery_consistency to get deployed
2021/12/11 05:37:56 Waiting for app: n1qlquery_consistency to get deployed
2021/12/11 05:37:56 App: n1qlquery_consistency got deployed
2021/12/11 05:38:01 Waiting for dst bucket item count to get to: 5000 curr count: 205
2021/12/11 05:38:06 Waiting for dst bucket item count to get to: 5000 curr count: 2705
2021/12/11 05:38:06 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/11 05:38:06 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 05:38:06 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 05:38:11 Waiting for app: n1qlquery_consistency to get un-deployed
2021/12/11 05:38:16 Waiting for app: n1qlquery_consistency to get un-deployed
2021/12/11 05:38:21 Waiting for app: n1qlquery_consistency to get un-deployed
2021/12/11 05:38:26 Waiting for app: n1qlquery_consistency to get un-deployed
2021/12/11 05:38:26 App: n1qlquery_consistency got un-deployed
2021/12/11 05:38:26 No eventing-con process running
2021/12/11 05:38:26 Function: n1qlquery_consistency delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: n1qlquery_consistency deleting in the background"
}
2021/12/11 05:38:28 Context: Bucket flush, response code: 200 dump: 
2021/12/11 05:38:38 Waiting for dst bucket item count to get to: 0 curr count: 3840
2021/12/11 05:38:43 Waiting for dst bucket item count to get to: 0 curr count: 1340
2021/12/11 05:38:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 05:38:45 Context: Bucket flush, response code: 200 dump: 
2021/12/11 05:38:55 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/11 05:38:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLQueryConsistency (101.47s)
PASS
ok  	github.com/couchbase/eventing/tests/functional_tests	1556.175s

N1ql tests took 0 hours, 26 minutes and 0 seconds.

Starting server
Starting server: attempt 1

Rebalance tests

2021/12/11 05:40:30 Started rebalance
Post "http://127.0.0.1:9000/pools/default": read tcp 127.0.0.1:32862->127.0.0.1:9000: read: connection reset by peer
Quota setup Post "http://127.0.0.1:9000/pools/default": read tcp 127.0.0.1:32862->127.0.0.1:9000: read: connection reset by peer
=== RUN   TestDuplicateWithEventingRebIn
2021/12/11 05:40:49 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
running apps fetch error json: cannot unmarshal number into Go value of type string
2021/12/11 05:40:54 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
2021/12/11 05:40:54 App: TestDuplicateWithEventingRebIn got un-deployed
2021/12/11 05:40:54 No eventing-con process running
2021/12/11 05:40:54 Function: TestDuplicateWithEventingRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithEventingRebIn not found"
 }
}
2021/12/11 05:40:56 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 05:41:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 05:41:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 05:41:02 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 05:41:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 05:41:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 05:41:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 05:41:12 Deploying app: TestDuplicateWithEventingRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/11 05:41:13 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithEventingRebIn' in metakv",
  "warnings": null
 }
}
2021/12/11 05:41:18 Waiting for app: TestDuplicateWithEventingRebIn to get deployed
2021/12/11 05:41:23 Waiting for app: TestDuplicateWithEventingRebIn to get deployed
2021/12/11 05:41:28 Waiting for app: TestDuplicateWithEventingRebIn to get deployed
2021/12/11 05:41:33 Waiting for app: TestDuplicateWithEventingRebIn to get deployed
2021/12/11 05:41:33 App: TestDuplicateWithEventingRebIn got deployed
2021/12/11 05:41:33 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/11 05:41:33 Starting bucket ops to source bucket
2021/12/11 05:41:48 Removing node(s): [] from the cluster
2021/12/11 05:41:53 Rebalance progress: 0
2021/12/11 05:41:58 Rebalance progress: 0
2021/12/11 05:42:03 Rebalance progress: 8.333333333333332
2021/12/11 05:42:08 Rebalance progress: 25
2021/12/11 05:42:13 Rebalance progress: 25
2021/12/11 05:42:18 Rebalance progress: 25
2021/12/11 05:42:23 Rebalance progress: 25
2021/12/11 05:42:28 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 05:42:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 05:42:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 05:42:28 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9302/api/v1/stats": dial tcp 127.0.0.1:9302: connect: connection refused
2021/12/11 05:42:28 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/11 05:42:28 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/11 05:42:33 Rebalance progress: 25
2021/12/11 05:42:38 Rebalance progress: 25
2021/12/11 05:42:43 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 05:42:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 05:42:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 05:42:43 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9302/api/v1/stats": dial tcp 127.0.0.1:9302: connect: connection refused
2021/12/11 05:42:43 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/11 05:42:43 Waiting for src bucket item count to get to: 200000 curr count: 90481
2021/12/11 05:42:48 Waiting for src bucket item count to get to: 200000 curr count: 107484
2021/12/11 05:42:53 Waiting for src bucket item count to get to: 200000 curr count: 107484
2021/12/11 05:42:58 Waiting for src bucket item count to get to: 200000 curr count: 124363
2021/12/11 05:43:03 Waiting for src bucket item count to get to: 200000 curr count: 124363
2021/12/11 05:43:08 Waiting for src bucket item count to get to: 200000 curr count: 141422
2021/12/11 05:43:13 Waiting for src bucket item count to get to: 200000 curr count: 141422
2021/12/11 05:43:18 Waiting for src bucket item count to get to: 200000 curr count: 158375
2021/12/11 05:43:23 Waiting for src bucket item count to get to: 200000 curr count: 158375
2021/12/11 05:43:28 Waiting for src bucket item count to get to: 200000 curr count: 175178
2021/12/11 05:43:33 Waiting for src bucket item count to get to: 200000 curr count: 191695
2021/12/11 05:43:38 Waiting for src bucket item count to get to: 200000 curr count: 191695
2021/12/11 05:43:48 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/11 05:43:53 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/11 05:43:58 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/11 05:44:03 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/11 05:44:08 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/11 05:44:13 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/11 05:44:18 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/11 05:44:23 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/11 05:44:28 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/11 05:44:33 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/11 05:44:33 Post SET ops. Expected item count: 200000 got 200001
2021/12/11 05:44:33 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 05:44:38 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
2021/12/11 05:44:43 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
2021/12/11 05:44:48 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
2021/12/11 05:44:53 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
2021/12/11 05:44:53 App: TestDuplicateWithEventingRebIn got un-deployed
2021/12/11 05:44:53 No eventing-con process running
2021/12/11 05:44:53 Function: TestDuplicateWithEventingRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithEventingRebIn deleting in the background"
}
2021/12/11 05:44:55 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 05:45:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 05:45:05 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/11 05:45:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 05:45:06 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 05:45:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 05:45:16 Waiting for dst bucket item count to get to: 0 curr count: 200001
2021/12/11 05:45:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithEventingRebIn (277.55s)
=== RUN   TestDuplicateWithEventingRebOut
2021/12/11 05:45:21 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/11 05:45:22 Removing node(s): [] from the cluster
2021/12/11 05:45:27 Rebalance progress: 100
2021/12/11 05:45:32 Waiting for app: TestDuplicateWithEventingRebOut to get un-deployed
2021/12/11 05:45:32 App: TestDuplicateWithEventingRebOut got un-deployed
2021/12/11 05:45:32 No eventing-con process running
2021/12/11 05:45:32 Function: TestDuplicateWithEventingRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithEventingRebOut not found"
 }
}
2021/12/11 05:45:33 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 05:45:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 05:45:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 05:45:40 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 05:45:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 05:45:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 05:45:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 05:45:50 Deploying app: TestDuplicateWithEventingRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/11 05:45:51 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithEventingRebOut' in metakv",
  "warnings": null
 }
}
2021/12/11 05:45:56 Waiting for app: TestDuplicateWithEventingRebOut to get deployed
2021/12/11 05:46:01 Waiting for app: TestDuplicateWithEventingRebOut to get deployed
2021/12/11 05:46:06 Waiting for app: TestDuplicateWithEventingRebOut to get deployed
2021/12/11 05:46:11 Waiting for app: TestDuplicateWithEventingRebOut to get deployed
2021/12/11 05:46:11 App: TestDuplicateWithEventingRebOut got deployed
2021/12/11 05:46:11 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/11 05:46:11 Starting bucket ops to source bucket
2021/12/11 05:46:16 Rebalance progress: 25
2021/12/11 05:46:21 Rebalance progress: 25
2021/12/11 05:46:26 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 05:46:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 05:46:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 05:46:26 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9302/api/v1/stats": dial tcp 127.0.0.1:9302: connect: connection refused
2021/12/11 05:46:26 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/11 05:46:26 Waiting for src bucket item count to get to: 200000 curr count: 13024
2021/12/11 05:46:31 Waiting for src bucket item count to get to: 200000 curr count: 29626
2021/12/11 05:46:36 Waiting for src bucket item count to get to: 200000 curr count: 29626
2021/12/11 05:46:41 Waiting for src bucket item count to get to: 200000 curr count: 46594
2021/12/11 05:46:46 Waiting for src bucket item count to get to: 200000 curr count: 46594
2021/12/11 05:46:51 Waiting for src bucket item count to get to: 200000 curr count: 63469
2021/12/11 05:46:56 Waiting for src bucket item count to get to: 200000 curr count: 63469
2021/12/11 05:47:01 Waiting for src bucket item count to get to: 200000 curr count: 63469
2021/12/11 05:47:06 Waiting for src bucket item count to get to: 200000 curr count: 79835
2021/12/11 05:47:11 Waiting for src bucket item count to get to: 200000 curr count: 79835
2021/12/11 05:47:16 Waiting for src bucket item count to get to: 200000 curr count: 96127
2021/12/11 05:47:21 Waiting for src bucket item count to get to: 200000 curr count: 96127
2021/12/11 05:47:26 Waiting for src bucket item count to get to: 200000 curr count: 113089
2021/12/11 05:47:31 Waiting for src bucket item count to get to: 200000 curr count: 113089
2021/12/11 05:47:36 Waiting for src bucket item count to get to: 200000 curr count: 130023
2021/12/11 05:47:41 Waiting for src bucket item count to get to: 200000 curr count: 130023
2021/12/11 05:47:46 Waiting for src bucket item count to get to: 200000 curr count: 146994
2021/12/11 05:47:51 Waiting for src bucket item count to get to: 200000 curr count: 146994
2021/12/11 05:47:56 Waiting for src bucket item count to get to: 200000 curr count: 163563
2021/12/11 05:48:02 Waiting for src bucket item count to get to: 200000 curr count: 163563
2021/12/11 05:48:07 Waiting for src bucket item count to get to: 200000 curr count: 179895
2021/12/11 05:48:12 Waiting for src bucket item count to get to: 200000 curr count: 179895
2021/12/11 05:48:17 Waiting for src bucket item count to get to: 200000 curr count: 196565
2021/12/11 05:48:22 Waiting for src bucket item count to get to: 200000 curr count: 196565
2021/12/11 05:48:27 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/11 05:48:27 Post SET ops. Expected item count: 200000 got 200000
2021/12/11 05:48:27 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 05:48:32 Waiting for app: TestDuplicateWithEventingRebOut to get un-deployed
2021/12/11 05:48:37 Waiting for app: TestDuplicateWithEventingRebOut to get un-deployed
2021/12/11 05:48:42 Waiting for app: TestDuplicateWithEventingRebOut to get un-deployed
2021/12/11 05:48:47 Waiting for app: TestDuplicateWithEventingRebOut to get un-deployed
2021/12/11 05:48:47 App: TestDuplicateWithEventingRebOut got un-deployed
2021/12/11 05:48:47 No eventing-con process running
2021/12/11 05:48:47 Function: TestDuplicateWithEventingRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithEventingRebOut deleting in the background"
}
2021/12/11 05:48:48 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 05:48:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 05:48:58 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/11 05:48:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 05:49:00 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 05:49:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 05:49:10 Waiting for dst bucket item count to get to: 0 curr count: 46232
2021/12/11 05:49:15 Waiting for dst bucket item count to get to: 0 curr count: 46232
2021/12/11 05:49:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithEventingRebOut (238.65s)
=== RUN   TestDuplicateWithEventingSwapReb
2021/12/11 05:49:25 Waiting for app: TestDuplicateWithEventingSwapReb to get un-deployed
2021/12/11 05:49:25 App: TestDuplicateWithEventingSwapReb got un-deployed
2021/12/11 05:49:25 No eventing-con process running
2021/12/11 05:49:25 Function: TestDuplicateWithEventingSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithEventingSwapReb not found"
 }
}
2021/12/11 05:49:27 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 05:49:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 05:49:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 05:49:34 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 05:49:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 05:49:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 05:49:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 05:49:44 Deploying app: TestDuplicateWithEventingSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/11 05:49:45 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithEventingSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/11 05:49:50 Waiting for app: TestDuplicateWithEventingSwapReb to get deployed
2021/12/11 05:49:55 Waiting for app: TestDuplicateWithEventingSwapReb to get deployed
2021/12/11 05:50:00 Waiting for app: TestDuplicateWithEventingSwapReb to get deployed
2021/12/11 05:50:05 Waiting for app: TestDuplicateWithEventingSwapReb to get deployed
2021/12/11 05:50:05 App: TestDuplicateWithEventingSwapReb got deployed
2021/12/11 05:50:05 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/11 05:50:05 Starting bucket ops to source bucket
2021/12/11 05:50:05 Removing node(s): [] from the cluster
2021/12/11 05:50:10 Rebalance progress: 25
2021/12/11 05:50:15 Rebalance progress: 25
2021/12/11 05:50:20 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 05:50:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 05:50:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 05:50:20 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9302/api/v1/stats": dial tcp 127.0.0.1:9302: connect: connection refused
2021/12/11 05:50:20 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/11 05:50:20 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/11 05:50:27 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/11 05:50:32 Rebalance progress: 16.66666666666667
2021/12/11 05:50:37 Rebalance progress: 16.66666666666667
2021/12/11 05:50:42 Rebalance progress: 16.66666666666667
2021/12/11 05:50:47 Rebalance progress: 35.546875
2021/12/11 05:50:52 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 05:50:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 05:50:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 05:50:53 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/11 05:50:53 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/11 05:50:58 Rebalance progress: 16.66666666666667
2021/12/11 05:51:03 Rebalance progress: 16.66666666666667
2021/12/11 05:51:08 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 05:51:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 05:51:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 05:51:08 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/11 05:51:08 Waiting for src bucket item count to get to: 200000 curr count: 83635
2021/12/11 05:51:13 Waiting for src bucket item count to get to: 200000 curr count: 100059
2021/12/11 05:51:18 Waiting for src bucket item count to get to: 200000 curr count: 100059
2021/12/11 05:51:23 Waiting for src bucket item count to get to: 200000 curr count: 116928
2021/12/11 05:51:28 Waiting for src bucket item count to get to: 200000 curr count: 116928
2021/12/11 05:51:33 Waiting for src bucket item count to get to: 200000 curr count: 133689
2021/12/11 05:51:38 Waiting for src bucket item count to get to: 200000 curr count: 133689
2021/12/11 05:51:43 Waiting for src bucket item count to get to: 200000 curr count: 150217
2021/12/11 05:51:48 Waiting for src bucket item count to get to: 200000 curr count: 150217
2021/12/11 05:51:53 Waiting for src bucket item count to get to: 200000 curr count: 166893
2021/12/11 05:51:58 Waiting for src bucket item count to get to: 200000 curr count: 166893
2021/12/11 05:52:03 Waiting for src bucket item count to get to: 200000 curr count: 183060
2021/12/11 05:52:08 Waiting for src bucket item count to get to: 200000 curr count: 183060
2021/12/11 05:52:13 Waiting for src bucket item count to get to: 200000 curr count: 199827
2021/12/11 05:52:18 Waiting for src bucket item count to get to: 200000 curr count: 199827
2021/12/11 05:52:23 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/11 05:52:23 Post SET ops. Expected item count: 200000 got 200000
2021/12/11 05:52:23 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 05:52:28 Waiting for app: TestDuplicateWithEventingSwapReb to get un-deployed
2021/12/11 05:52:33 Waiting for app: TestDuplicateWithEventingSwapReb to get un-deployed
2021/12/11 05:52:38 Waiting for app: TestDuplicateWithEventingSwapReb to get un-deployed
2021/12/11 05:52:43 Waiting for app: TestDuplicateWithEventingSwapReb to get un-deployed
2021/12/11 05:52:43 App: TestDuplicateWithEventingSwapReb got un-deployed
2021/12/11 05:52:43 No eventing-con process running
2021/12/11 05:52:43 Function: TestDuplicateWithEventingSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithEventingSwapReb deleting in the background"
}
2021/12/11 05:52:45 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 05:52:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 05:52:55 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/11 05:52:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 05:52:56 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 05:53:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 05:53:06 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/11 05:53:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithEventingSwapReb (231.22s)
=== RUN   TestDuplicateWithKVRebIn
2021/12/11 05:53:16 Waiting for app: TestDuplicateWithKVRebIn to get un-deployed
2021/12/11 05:53:16 App: TestDuplicateWithKVRebIn got un-deployed
2021/12/11 05:53:16 No eventing-con process running
2021/12/11 05:53:16 Function: TestDuplicateWithKVRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithKVRebIn not found"
 }
}
2021/12/11 05:53:18 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 05:53:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 05:53:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 05:53:25 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 05:53:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 05:53:30 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 05:53:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 05:53:35 Deploying app: TestDuplicateWithKVRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/11 05:53:36 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithKVRebIn' in metakv",
  "warnings": null
 }
}
2021/12/11 05:53:41 Waiting for app: TestDuplicateWithKVRebIn to get deployed
2021/12/11 05:53:46 Waiting for app: TestDuplicateWithKVRebIn to get deployed
2021/12/11 05:53:51 Waiting for app: TestDuplicateWithKVRebIn to get deployed
2021/12/11 05:53:56 Waiting for app: TestDuplicateWithKVRebIn to get deployed
2021/12/11 05:53:56 App: TestDuplicateWithKVRebIn got deployed
2021/12/11 05:53:56 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/11 05:53:56 Starting bucket ops to source bucket
2021/12/11 05:53:56 Removing node(s): [] from the cluster
2021/12/11 05:54:01 Rebalance progress: 16.66666666666667
2021/12/11 05:54:06 Rebalance progress: 16.66666666666667
2021/12/11 05:54:11 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 05:54:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 05:54:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 05:54:11 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/11 05:54:11 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/11 05:54:17 Rebalance progress: 16.66666666666667
2021/12/11 05:54:22 Rebalance progress: 16.66666666666667
2021/12/11 05:54:27 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 05:54:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 05:54:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 05:54:27 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/11 05:54:27 Waiting for src bucket item count to get to: 200000 curr count: 36958
2021/12/11 05:54:32 Waiting for src bucket item count to get to: 200000 curr count: 53350
2021/12/11 05:54:37 Waiting for src bucket item count to get to: 200000 curr count: 53350
2021/12/11 05:54:42 Waiting for src bucket item count to get to: 200000 curr count: 70200
2021/12/11 05:54:47 Waiting for src bucket item count to get to: 200000 curr count: 70200
2021/12/11 05:54:52 Waiting for src bucket item count to get to: 200000 curr count: 85889
2021/12/11 05:54:57 Waiting for src bucket item count to get to: 200000 curr count: 85889
2021/12/11 05:55:02 Waiting for src bucket item count to get to: 200000 curr count: 102439
2021/12/11 05:55:07 Waiting for src bucket item count to get to: 200000 curr count: 102439
2021/12/11 05:55:12 Waiting for src bucket item count to get to: 200000 curr count: 118667
2021/12/11 05:55:17 Waiting for src bucket item count to get to: 200000 curr count: 118667
2021/12/11 05:55:22 Waiting for src bucket item count to get to: 200000 curr count: 118667
2021/12/11 05:55:27 Waiting for src bucket item count to get to: 200000 curr count: 135235
2021/12/11 05:55:32 Waiting for src bucket item count to get to: 200000 curr count: 135235
2021/12/11 05:55:37 Waiting for src bucket item count to get to: 200000 curr count: 151643
2021/12/11 05:55:42 Waiting for src bucket item count to get to: 200000 curr count: 151643
2021/12/11 05:55:47 Waiting for src bucket item count to get to: 200000 curr count: 168489
2021/12/11 05:55:52 Waiting for src bucket item count to get to: 200000 curr count: 168489
2021/12/11 05:55:57 Waiting for src bucket item count to get to: 200000 curr count: 184123
2021/12/11 05:56:02 Waiting for src bucket item count to get to: 200000 curr count: 184123
2021/12/11 05:56:07 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/11 05:56:07 Post SET ops. Expected item count: 200000 got 200000
2021/12/11 05:56:07 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 05:56:12 Waiting for app: TestDuplicateWithKVRebIn to get un-deployed
2021/12/11 05:56:17 Waiting for app: TestDuplicateWithKVRebIn to get un-deployed
2021/12/11 05:56:22 Waiting for app: TestDuplicateWithKVRebIn to get un-deployed
2021/12/11 05:56:27 Waiting for app: TestDuplicateWithKVRebIn to get un-deployed
2021/12/11 05:56:27 App: TestDuplicateWithKVRebIn got un-deployed
2021/12/11 05:56:27 No eventing-con process running
2021/12/11 05:56:27 Function: TestDuplicateWithKVRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithKVRebIn deleting in the background"
}
2021/12/11 05:56:29 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 05:56:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 05:56:39 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/11 05:56:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 05:56:40 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 05:56:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 05:56:50 Waiting for dst bucket item count to get to: 0 curr count: 86423
2021/12/11 05:56:55 Waiting for dst bucket item count to get to: 0 curr count: 86423
2021/12/11 05:56:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithKVRebIn (229.00s)
=== RUN   TestDuplicateWithKVRebOut
2021/12/11 05:57:00 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/11 05:57:01 Removing node(s): [] from the cluster
2021/12/11 05:57:06 Rebalance progress: 100
2021/12/11 05:57:11 Waiting for app: TestDuplicateWithKVRebOut to get un-deployed
2021/12/11 05:57:11 App: TestDuplicateWithKVRebOut got un-deployed
2021/12/11 05:57:11 No eventing-con process running
2021/12/11 05:57:11 Function: TestDuplicateWithKVRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithKVRebOut not found"
 }
}
2021/12/11 05:57:12 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 05:57:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 05:57:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 05:57:19 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 05:57:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 05:57:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 05:57:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 05:57:29 Deploying app: TestDuplicateWithKVRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/11 05:57:30 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithKVRebOut' in metakv",
  "warnings": null
 }
}
2021/12/11 05:57:35 Waiting for app: TestDuplicateWithKVRebOut to get deployed
2021/12/11 05:57:40 Waiting for app: TestDuplicateWithKVRebOut to get deployed
2021/12/11 05:57:45 Waiting for app: TestDuplicateWithKVRebOut to get deployed
2021/12/11 05:57:50 Waiting for app: TestDuplicateWithKVRebOut to get deployed
2021/12/11 05:57:50 App: TestDuplicateWithKVRebOut got deployed
2021/12/11 05:57:50 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/11 05:57:50 Starting bucket ops to source bucket
2021/12/11 05:57:55 Rebalance progress: 16.66666666666667
2021/12/11 05:58:00 Rebalance progress: 16.66666666666667
2021/12/11 05:58:05 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 05:58:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 05:58:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 05:58:05 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/11 05:58:05 Waiting for src bucket item count to get to: 200000 curr count: 14050
2021/12/11 05:58:10 Waiting for src bucket item count to get to: 200000 curr count: 30327
2021/12/11 05:58:15 Waiting for src bucket item count to get to: 200000 curr count: 30327
2021/12/11 05:58:20 Waiting for src bucket item count to get to: 200000 curr count: 46769
2021/12/11 05:58:25 Waiting for src bucket item count to get to: 200000 curr count: 46769
2021/12/11 05:58:30 Waiting for src bucket item count to get to: 200000 curr count: 63616
2021/12/11 05:58:35 Waiting for src bucket item count to get to: 200000 curr count: 63616
2021/12/11 05:58:41 Waiting for src bucket item count to get to: 200000 curr count: 63616
2021/12/11 05:58:46 Waiting for src bucket item count to get to: 200000 curr count: 79594
2021/12/11 05:58:51 Waiting for src bucket item count to get to: 200000 curr count: 79594
2021/12/11 05:58:56 Waiting for src bucket item count to get to: 200000 curr count: 95715
2021/12/11 05:59:01 Waiting for src bucket item count to get to: 200000 curr count: 95715
2021/12/11 05:59:06 Waiting for src bucket item count to get to: 200000 curr count: 112462
2021/12/11 05:59:11 Waiting for src bucket item count to get to: 200000 curr count: 112462
2021/12/11 05:59:16 Waiting for src bucket item count to get to: 200000 curr count: 129194
2021/12/11 05:59:21 Waiting for src bucket item count to get to: 200000 curr count: 129194
2021/12/11 05:59:26 Waiting for src bucket item count to get to: 200000 curr count: 145899
2021/12/11 05:59:31 Waiting for src bucket item count to get to: 200000 curr count: 145899
2021/12/11 05:59:36 Waiting for src bucket item count to get to: 200000 curr count: 162529
2021/12/11 05:59:41 Waiting for src bucket item count to get to: 200000 curr count: 162529
2021/12/11 05:59:46 Waiting for src bucket item count to get to: 200000 curr count: 178404
2021/12/11 05:59:51 Waiting for src bucket item count to get to: 200000 curr count: 178404
2021/12/11 05:59:56 Waiting for src bucket item count to get to: 200000 curr count: 194577
2021/12/11 06:00:01 Waiting for src bucket item count to get to: 200000 curr count: 194577
2021/12/11 06:00:06 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/11 06:00:06 Post SET ops. Expected item count: 200000 got 200000
2021/12/11 06:00:06 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 06:00:11 Waiting for app: TestDuplicateWithKVRebOut to get un-deployed
2021/12/11 06:00:16 Waiting for app: TestDuplicateWithKVRebOut to get un-deployed
2021/12/11 06:00:21 Waiting for app: TestDuplicateWithKVRebOut to get un-deployed
2021/12/11 06:00:26 Waiting for app: TestDuplicateWithKVRebOut to get un-deployed
2021/12/11 06:00:26 App: TestDuplicateWithKVRebOut got un-deployed
2021/12/11 06:00:26 No eventing-con process running
2021/12/11 06:00:26 Function: TestDuplicateWithKVRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithKVRebOut deleting in the background"
}
2021/12/11 06:00:28 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:00:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 06:00:38 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/11 06:00:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 06:00:39 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:00:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 06:00:49 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/11 06:00:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithKVRebOut (233.93s)
=== RUN   TestDuplicateWithKVSwapReb
2021/12/11 06:00:59 Waiting for app: TestDuplicateWithKVSwapReb to get un-deployed
2021/12/11 06:00:59 App: TestDuplicateWithKVSwapReb got un-deployed
2021/12/11 06:00:59 No eventing-con process running
2021/12/11 06:00:59 Function: TestDuplicateWithKVSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithKVSwapReb not found"
 }
}
2021/12/11 06:01:01 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:01:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 06:01:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 06:01:08 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:01:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 06:01:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:01:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 06:01:18 Deploying app: TestDuplicateWithKVSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/11 06:01:19 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithKVSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/11 06:01:24 Waiting for app: TestDuplicateWithKVSwapReb to get deployed
2021/12/11 06:01:29 Waiting for app: TestDuplicateWithKVSwapReb to get deployed
2021/12/11 06:01:34 Waiting for app: TestDuplicateWithKVSwapReb to get deployed
2021/12/11 06:01:39 Waiting for app: TestDuplicateWithKVSwapReb to get deployed
2021/12/11 06:01:39 App: TestDuplicateWithKVSwapReb got deployed
2021/12/11 06:01:39 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/11 06:01:39 Starting bucket ops to source bucket
2021/12/11 06:01:39 Removing node(s): [] from the cluster
2021/12/11 06:01:44 Rebalance progress: 16.66666666666667
2021/12/11 06:01:49 Rebalance progress: 16.66666666666667
2021/12/11 06:01:54 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:01:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:01:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 06:01:54 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/11 06:01:54 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/11 06:01:55 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/11 06:02:00 Rebalance progress: 16.66666666666667
2021/12/11 06:02:05 Rebalance progress: 16.66666666666667
2021/12/11 06:02:10 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:02:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:02:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 06:02:10 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/11 06:02:10 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/11 06:02:15 Rebalance progress: 16.66666666666667
2021/12/11 06:02:20 Rebalance progress: 16.66666666666667
2021/12/11 06:02:25 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:02:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:02:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 06:02:25 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/11 06:02:25 Waiting for src bucket item count to get to: 200000 curr count: 63690
2021/12/11 06:02:30 Waiting for src bucket item count to get to: 200000 curr count: 63690
2021/12/11 06:02:35 Waiting for src bucket item count to get to: 200000 curr count: 79120
2021/12/11 06:02:40 Waiting for src bucket item count to get to: 200000 curr count: 79120
2021/12/11 06:02:45 Waiting for src bucket item count to get to: 200000 curr count: 95430
2021/12/11 06:02:50 Waiting for src bucket item count to get to: 200000 curr count: 95430
2021/12/11 06:02:55 Waiting for src bucket item count to get to: 200000 curr count: 111947
2021/12/11 06:03:00 Waiting for src bucket item count to get to: 200000 curr count: 128641
2021/12/11 06:03:05 Waiting for src bucket item count to get to: 200000 curr count: 128641
2021/12/11 06:03:10 Waiting for src bucket item count to get to: 200000 curr count: 145467
2021/12/11 06:03:15 Waiting for src bucket item count to get to: 200000 curr count: 145467
2021/12/11 06:03:20 Waiting for src bucket item count to get to: 200000 curr count: 162046
2021/12/11 06:03:25 Waiting for src bucket item count to get to: 200000 curr count: 162046
2021/12/11 06:03:30 Waiting for src bucket item count to get to: 200000 curr count: 178181
2021/12/11 06:03:35 Waiting for src bucket item count to get to: 200000 curr count: 178181
2021/12/11 06:03:40 Waiting for src bucket item count to get to: 200000 curr count: 194557
2021/12/11 06:03:45 Waiting for src bucket item count to get to: 200000 curr count: 194557
2021/12/11 06:03:50 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/11 06:03:50 Post SET ops. Expected item count: 200000 got 200000
2021/12/11 06:03:51 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 06:03:56 Waiting for app: TestDuplicateWithKVSwapReb to get un-deployed
2021/12/11 06:04:01 Waiting for app: TestDuplicateWithKVSwapReb to get un-deployed
2021/12/11 06:04:06 Waiting for app: TestDuplicateWithKVSwapReb to get un-deployed
2021/12/11 06:04:11 Waiting for app: TestDuplicateWithKVSwapReb to get un-deployed
2021/12/11 06:04:11 App: TestDuplicateWithKVSwapReb got un-deployed
2021/12/11 06:04:11 No eventing-con process running
2021/12/11 06:04:11 Function: TestDuplicateWithKVSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithKVSwapReb deleting in the background"
}
2021/12/11 06:04:12 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:04:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 06:04:22 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/11 06:04:27 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/11 06:04:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 06:04:29 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:04:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 06:04:39 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/11 06:04:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithKVSwapReb (229.82s)
=== RUN   TestEventingRebKVOpsOnUpdateBucketOpOneByOne
2021/12/11 06:04:49 Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/11 06:04:49 App: TestEventingRebKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/11 06:04:49 No eventing-con process running
2021/12/11 06:04:49 Function: TestEventingRebKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpOneByOne not found"
 }
}
2021/12/11 06:04:51 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:04:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 06:04:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 06:04:58 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:05:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 06:05:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:05:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 06:05:08 Deploying app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne. Item count src bucket: 0 dst bucket: 0
2021/12/11 06:05:09 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebKVOpsOnUpdateBucketOpOneByOne' in metakv",
  "warnings": null
 }
}
2021/12/11 06:05:14 Starting bucket ops to source bucket
2021/12/11 06:05:19 Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/11 06:05:24 Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/11 06:05:29 Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/11 06:05:29 App: TestEventingRebKVOpsOnUpdateBucketOpOneByOne got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:05:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:05:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 06:05:29 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/11 06:05:29 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/11 06:05:29 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/11 06:05:34 Rebalance progress: 16.66666666666667
2021/12/11 06:05:39 Rebalance progress: 16.66666666666667
2021/12/11 06:05:44 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:05:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:05:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 06:05:44 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/11 06:05:44 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/11 06:05:44 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/11 06:05:49 Rebalance progress: 16.66666666666667
2021/12/11 06:05:54 Rebalance progress: 16.66666666666667
2021/12/11 06:05:59 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:05:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:05:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 06:05:59 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/11 06:05:59 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/11 06:06:05 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/11 06:06:10 Rebalance progress: 12.5
2021/12/11 06:06:15 Rebalance progress: 12.5
2021/12/11 06:06:20 Rebalance progress: 12.5
2021/12/11 06:06:25 Rebalance progress: 98.6328125
2021/12/11 06:06:30 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:06:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:06:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 06:06:30 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/11 06:06:35 Rebalance progress: 12.5
2021/12/11 06:06:40 Rebalance progress: 12.5
2021/12/11 06:06:45 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:06:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:06:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 06:06:45 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/11 06:06:50 Rebalance progress: 12.5
2021/12/11 06:06:55 Rebalance progress: 12.5
2021/12/11 06:07:00 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:07:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:07:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 06:07:00 clusterSetup.go:253: Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/11 06:07:05 Rebalance progress: 12.5
2021/12/11 06:07:10 Rebalance progress: 12.5
2021/12/11 06:07:15 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:07:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:07:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 06:07:15 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 06:07:20 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/11 06:07:26 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/11 06:07:31 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/11 06:07:36 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/11 06:07:36 clusterSetup.go:405: App: TestEventingRebKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/11 06:07:36 eventingSetup.go:46: No eventing-con process running
2021/12/11 06:07:36 eventingSetup.go:418: Function: TestEventingRebKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpOneByOne deleting in the background"
}
2021/12/11 06:07:37 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:07:42 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 06:07:47 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12159
2021/12/11 06:07:47 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 06:07:49 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:07:54 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 06:07:59 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12159
2021/12/11 06:07:59 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebKVOpsOnUpdateBucketOpOneByOne (200.02s)
=== RUN   TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce
2021/12/11 06:08:09 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get un-deployed
2021/12/11 06:08:09 clusterSetup.go:405: App: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce got un-deployed
2021/12/11 06:08:09 eventingSetup.go:46: No eventing-con process running
2021/12/11 06:08:09 eventingSetup.go:418: Function: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce not found"
 }
}
2021/12/11 06:08:11 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:08:16 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 06:08:16 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 06:08:18 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:08:23 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 06:08:23 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:08:28 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 06:08:28 eventingSetup.go:125: Deploying app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce. Item count src bucket: 0 dst bucket: 0
2021/12/11 06:08:29 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce' in metakv",
  "warnings": null
 }
}
2021/12/11 06:08:34 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/11 06:08:39 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get deployed
2021/12/11 06:08:44 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get deployed
2021/12/11 06:08:49 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get deployed
2021/12/11 06:08:49 clusterSetup.go:354: App: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:08:49 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:08:49 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 06:08:49 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/11 06:08:50 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/11 06:08:50 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/11 06:08:50 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/11 06:08:55 Rebalance progress: 12.5
2021/12/11 06:09:00 Rebalance progress: 12.5
2021/12/11 06:09:05 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:09:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:09:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 06:09:05 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002 127.0.0.1:19003] from the cluster
2021/12/11 06:09:10 Rebalance progress: 12.5
2021/12/11 06:09:15 Rebalance progress: 12.5
2021/12/11 06:09:20 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:09:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:09:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 06:09:20 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 06:09:25 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get un-deployed
2021/12/11 06:09:30 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get un-deployed
2021/12/11 06:09:35 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get un-deployed
2021/12/11 06:09:40 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get un-deployed
2021/12/11 06:09:40 clusterSetup.go:405: App: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce got un-deployed
2021/12/11 06:09:40 eventingSetup.go:46: No eventing-con process running
2021/12/11 06:09:40 eventingSetup.go:418: Function: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce deleting in the background"
}
2021/12/11 06:09:42 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:09:47 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 06:09:52 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4617
2021/12/11 06:09:57 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4617
2021/12/11 06:09:57 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 06:09:59 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:10:04 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 06:10:09 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4617
2021/12/11 06:10:09 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce (129.76s)
=== RUN   TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings
2021/12/11 06:10:19 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/11 06:10:19 clusterSetup.go:405: App: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings got un-deployed
2021/12/11 06:10:19 eventingSetup.go:46: No eventing-con process running
2021/12/11 06:10:19 eventingSetup.go:418: Function: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings not found"
 }
}
2021/12/11 06:10:21 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:10:26 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 06:10:26 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 06:10:27 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:10:32 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 06:10:32 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:10:37 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 06:10:37 eventingSetup.go:125: Deploying app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/11 06:10:39 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/11 06:10:44 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/11 06:10:49 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get deployed
2021/12/11 06:10:54 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get deployed
2021/12/11 06:10:59 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get deployed
2021/12/11 06:10:59 clusterSetup.go:354: App: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:10:59 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:10:59 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 06:10:59 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/11 06:10:59 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/11 06:11:04 Rebalance progress: 12.5
2021/12/11 06:11:09 Rebalance progress: 12.5
2021/12/11 06:11:14 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:11:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:11:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 06:11:14 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/11 06:11:14 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/11 06:11:20 Rebalance progress: 12.5
2021/12/11 06:11:25 Rebalance progress: 12.5
2021/12/11 06:11:30 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:11:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:11:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 06:11:30 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/11 06:11:30 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/11 06:11:35 Rebalance progress: 12.5
2021/12/11 06:11:40 Rebalance progress: 12.5
2021/12/11 06:11:45 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:11:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:11:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 06:11:45 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/11 06:11:50 Rebalance progress: 12.5
2021/12/11 06:11:55 Rebalance progress: 12.5
2021/12/11 06:12:00 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:12:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:12:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 06:12:00 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/11 06:12:05 Rebalance progress: 12.5
2021/12/11 06:12:10 Rebalance progress: 12.5
2021/12/11 06:12:15 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:12:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:12:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 06:12:15 clusterSetup.go:253: Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/11 06:12:20 Rebalance progress: 12.5
2021/12/11 06:12:25 Rebalance progress: 12.5
2021/12/11 06:12:30 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:12:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:12:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 06:12:30 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 06:12:35 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/11 06:12:40 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/11 06:12:45 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/11 06:12:50 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/11 06:12:50 clusterSetup.go:405: App: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings got un-deployed
2021/12/11 06:12:50 eventingSetup.go:46: No eventing-con process running
2021/12/11 06:12:50 eventingSetup.go:418: Function: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings deleting in the background"
}
2021/12/11 06:12:52 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:12:57 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 06:13:02 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 10658
2021/12/11 06:13:07 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 10658
2021/12/11 06:13:07 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 06:13:09 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:13:14 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 06:13:19 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 10655
2021/12/11 06:13:19 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings (190.09s)
=== RUN   TestEventingSwapRebOnUpdateBucketOp
2021/12/11 06:13:29 clusterSetup.go:397: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get un-deployed
2021/12/11 06:13:29 clusterSetup.go:405: App: TestEventingSwapRebOnUpdateBucketOp got un-deployed
2021/12/11 06:13:29 eventingSetup.go:46: No eventing-con process running
2021/12/11 06:13:29 eventingSetup.go:418: Function: TestEventingSwapRebOnUpdateBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingSwapRebOnUpdateBucketOp not found"
 }
}
2021/12/11 06:13:31 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:13:36 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 06:13:36 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 06:13:37 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:13:42 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 06:13:42 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:13:42 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 06:13:42 eventingSetup.go:125: Deploying app: TestEventingSwapRebOnUpdateBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/11 06:13:44 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingSwapRebOnUpdateBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/11 06:13:49 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/11 06:13:54 clusterSetup.go:346: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get deployed
2021/12/11 06:13:59 clusterSetup.go:346: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get deployed
2021/12/11 06:14:04 clusterSetup.go:346: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get deployed
2021/12/11 06:14:04 clusterSetup.go:354: App: TestEventingSwapRebOnUpdateBucketOp got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:14:04 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:14:04 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 06:14:04 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/11 06:14:04 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/11 06:14:09 Rebalance progress: 12.5
2021/12/11 06:14:14 Rebalance progress: 12.5
2021/12/11 06:14:19 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:14:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:14:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 06:14:19 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/11 06:14:19 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/11 06:14:24 Rebalance progress: 12.5
2021/12/11 06:14:29 Rebalance progress: 12.5
2021/12/11 06:14:34 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:14:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:14:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 06:14:34 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/11 06:14:39 Rebalance progress: 12.5
2021/12/11 06:14:44 Rebalance progress: 12.5
2021/12/11 06:14:49 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:14:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:14:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 06:14:50 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 06:14:55 clusterSetup.go:397: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get un-deployed
2021/12/11 06:15:00 clusterSetup.go:397: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get un-deployed
2021/12/11 06:15:05 clusterSetup.go:397: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get un-deployed
2021/12/11 06:15:10 clusterSetup.go:397: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get un-deployed
2021/12/11 06:15:10 clusterSetup.go:405: App: TestEventingSwapRebOnUpdateBucketOp got un-deployed
2021/12/11 06:15:10 eventingSetup.go:46: No eventing-con process running
2021/12/11 06:15:10 eventingSetup.go:418: Function: TestEventingSwapRebOnUpdateBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingSwapRebOnUpdateBucketOp deleting in the background"
}
2021/12/11 06:15:11 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:15:16 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 06:15:21 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6074
2021/12/11 06:15:26 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6074
2021/12/11 06:15:26 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 06:15:28 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:15:33 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 06:15:38 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6074
2021/12/11 06:15:38 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingSwapRebOnUpdateBucketOp (139.13s)
=== RUN   TestMetaRollbackWithEventingReb
2021/12/11 06:15:43 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/11 06:15:44 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/11 06:15:49 Rebalance progress: 100
2021/12/11 06:15:54 Waiting for app: TestMetaRollbackWithEventingReb to get un-deployed
2021/12/11 06:15:54 App: TestMetaRollbackWithEventingReb got un-deployed
2021/12/11 06:15:54 No eventing-con process running
2021/12/11 06:15:54 Function: TestMetaRollbackWithEventingReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMetaRollbackWithEventingReb not found"
 }
}
2021/12/11 06:15:55 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:16:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 06:16:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 06:16:02 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:16:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 06:16:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:16:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 06:16:07 Deploying app: TestMetaRollbackWithEventingReb. Item count src bucket: 0 dst bucket: 0
2021/12/11 06:16:08 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMetaRollbackWithEventingReb' in metakv",
  "warnings": null
 }
}
2021/12/11 06:16:13 Starting bucket ops to source bucket
2021/12/11 06:16:18 Waiting for app: TestMetaRollbackWithEventingReb to get deployed
2021/12/11 06:16:23 Waiting for app: TestMetaRollbackWithEventingReb to get deployed
2021/12/11 06:16:28 Waiting for app: TestMetaRollbackWithEventingReb to get deployed
2021/12/11 06:16:28 App: TestMetaRollbackWithEventingReb got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:16:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:16:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 06:16:28 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/11 06:16:28 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/11 06:16:33 Rebalance progress: 12.5
2021/12/11 06:16:38 Rebalance progress: 12.5
2021/12/11 06:16:43 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:16:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:16:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 06:16:44 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002] from the cluster
2021/12/11 06:16:44 bucketOps.go:134: Metakv lookup failed, err: Get "/_metakv/eventing/tempApps/TestMetaRollbackWithEventingReb/0": Unable to initialize cbauth's revrpc: cbauth environment variable CBAUTH_REVRPC_URL is not set
2021/12/11 06:16:49 Rebalance progress: 12.5
2021/12/11 06:16:54 Rebalance progress: 12.5
2021/12/11 06:16:59 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:16:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:16:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 06:16:59 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 06:17:04 clusterSetup.go:397: Waiting for app: TestMetaRollbackWithEventingReb to get un-deployed
2021/12/11 06:17:09 clusterSetup.go:397: Waiting for app: TestMetaRollbackWithEventingReb to get un-deployed
2021/12/11 06:17:14 clusterSetup.go:397: Waiting for app: TestMetaRollbackWithEventingReb to get un-deployed
2021/12/11 06:17:19 clusterSetup.go:397: Waiting for app: TestMetaRollbackWithEventingReb to get un-deployed
2021/12/11 06:17:19 clusterSetup.go:405: App: TestMetaRollbackWithEventingReb got un-deployed
2021/12/11 06:17:19 eventingSetup.go:46: No eventing-con process running
2021/12/11 06:17:19 eventingSetup.go:418: Function: TestMetaRollbackWithEventingReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMetaRollbackWithEventingReb deleting in the background"
}
2021/12/11 06:17:21 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:17:26 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 06:17:31 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3028
2021/12/11 06:17:36 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3028
2021/12/11 06:17:36 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 06:17:37 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:17:42 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 06:17:47 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4532
2021/12/11 06:17:47 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestMetaRollbackWithEventingReb (129.27s)
=== RUN   TestMetaPartialRollbackWithEventingReb
2021/12/11 06:17:52 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/11 06:17:53 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/11 06:17:58 Rebalance progress: 100
2021/12/11 06:18:03 Waiting for app: TestMetaPartialRollbackWithEventingReb to get un-deployed
2021/12/11 06:18:03 App: TestMetaPartialRollbackWithEventingReb got un-deployed
2021/12/11 06:18:03 No eventing-con process running
2021/12/11 06:18:03 Function: TestMetaPartialRollbackWithEventingReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMetaPartialRollbackWithEventingReb not found"
 }
}
2021/12/11 06:18:05 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:18:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 06:18:10 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 06:18:11 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:18:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 06:18:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:18:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 06:18:16 Deploying app: TestMetaPartialRollbackWithEventingReb. Item count src bucket: 0 dst bucket: 0
2021/12/11 06:18:17 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMetaPartialRollbackWithEventingReb' in metakv",
  "warnings": null
 }
}
2021/12/11 06:18:22 Starting bucket ops to source bucket
2021/12/11 06:18:27 Waiting for app: TestMetaPartialRollbackWithEventingReb to get deployed
2021/12/11 06:18:32 Waiting for app: TestMetaPartialRollbackWithEventingReb to get deployed
2021/12/11 06:18:37 Waiting for app: TestMetaPartialRollbackWithEventingReb to get deployed
2021/12/11 06:18:37 App: TestMetaPartialRollbackWithEventingReb got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:18:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:18:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 06:18:37 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/11 06:18:38 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/11 06:18:43 Rebalance progress: 12.5
2021/12/11 06:18:48 Rebalance progress: 12.5
2021/12/11 06:18:53 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:18:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:18:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 06:18:53 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002] from the cluster
2021/12/11 06:18:53 bucketOps.go:65: Metakv lookup failed, err: Get "/_metakv/eventing/tempApps/TestMetaPartialRollbackWithEventingReb/0": Unable to initialize cbauth's revrpc: cbauth environment variable CBAUTH_REVRPC_URL is not set
2021/12/11 06:18:58 Rebalance progress: 12.5
2021/12/11 06:19:03 Rebalance progress: 12.5
2021/12/11 06:19:08 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:19:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:19:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 06:19:08 bucketOps.go:65: Metakv lookup failed, err: Get "/_metakv/eventing/tempApps/TestMetaPartialRollbackWithEventingReb/0": Unable to initialize cbauth's revrpc: cbauth environment variable CBAUTH_REVRPC_URL is not set
2021/12/11 06:19:08 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 06:19:13 clusterSetup.go:397: Waiting for app: TestMetaPartialRollbackWithEventingReb to get un-deployed
2021/12/11 06:19:18 clusterSetup.go:397: Waiting for app: TestMetaPartialRollbackWithEventingReb to get un-deployed
2021/12/11 06:19:23 clusterSetup.go:397: Waiting for app: TestMetaPartialRollbackWithEventingReb to get un-deployed
2021/12/11 06:19:28 clusterSetup.go:397: Waiting for app: TestMetaPartialRollbackWithEventingReb to get un-deployed
2021/12/11 06:19:28 clusterSetup.go:405: App: TestMetaPartialRollbackWithEventingReb got un-deployed
2021/12/11 06:19:28 eventingSetup.go:46: No eventing-con process running
2021/12/11 06:19:28 eventingSetup.go:418: Function: TestMetaPartialRollbackWithEventingReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMetaPartialRollbackWithEventingReb deleting in the background"
}
2021/12/11 06:19:30 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:19:35 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 06:19:35 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 06:19:37 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:19:42 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 06:19:47 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 38037
2021/12/11 06:19:47 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestMetaPartialRollbackWithEventingReb (119.28s)
=== RUN   TestEventingRebMultipleHandlersOneByOne
2021/12/11 06:19:57 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get un-deployed
2021/12/11 06:19:57 clusterSetup.go:405: App: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op got un-deployed
2021/12/11 06:19:57 eventingSetup.go:46: No eventing-con process running
2021/12/11 06:19:57 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op not found"
 }
}
2021/12/11 06:19:59 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:20:04 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 06:20:04 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 06:20:05 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:20:10 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 06:20:10 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:20:15 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 06:20:15 eventingSetup.go:125: Deploying app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op. Item count src bucket: 0 dst bucket: 0
2021/12/11 06:20:16 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op' in metakv",
  "warnings": null
 }
}
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:20:16 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 06:20:16 eventingSetup.go:125: Deploying app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/11 06:20:18 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebMultipleHandlersOneByOnebucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/11 06:20:23 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/11 06:20:28 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get deployed
2021/12/11 06:20:33 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get deployed
2021/12/11 06:20:38 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get deployed
2021/12/11 06:20:38 clusterSetup.go:354: App: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op got deployed
2021/12/11 06:20:43 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get deployed
2021/12/11 06:20:48 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get deployed
2021/12/11 06:20:53 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get deployed
2021/12/11 06:20:53 clusterSetup.go:354: App: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:20:53 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:20:53 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 06:20:53 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/11 06:20:53 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/11 06:20:58 Rebalance progress: 12.5
2021/12/11 06:21:03 Rebalance progress: 12.5
2021/12/11 06:21:08 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:21:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:21:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 06:21:08 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/11 06:21:08 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/11 06:21:13 Rebalance progress: 12.5
2021/12/11 06:21:18 Rebalance progress: 12.5
2021/12/11 06:21:23 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:21:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:21:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 06:21:23 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/11 06:21:24 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/11 06:21:29 Rebalance progress: 12.5
2021/12/11 06:21:34 Rebalance progress: 12.5
2021/12/11 06:21:39 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:21:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:21:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 06:21:39 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/11 06:21:44 Rebalance progress: 12.5
2021/12/11 06:21:49 Rebalance progress: 12.5
2021/12/11 06:21:54 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:21:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:21:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 06:21:54 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/11 06:21:59 Rebalance progress: 12.5
2021/12/11 06:22:04 Rebalance progress: 12.5
2021/12/11 06:22:09 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:22:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:22:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 06:22:09 clusterSetup.go:253: Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/11 06:22:14 Rebalance progress: 12.5
2021/12/11 06:22:19 Rebalance progress: 12.5
2021/12/11 06:22:24 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:22:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:22:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 06:22:24 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 06:22:29 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get un-deployed
2021/12/11 06:22:34 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get un-deployed
2021/12/11 06:22:39 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get un-deployed
2021/12/11 06:22:44 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get un-deployed
2021/12/11 06:22:44 clusterSetup.go:405: App: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op got un-deployed
2021/12/11 06:22:44 eventingSetup.go:46: No eventing-con process running
2021/12/11 06:22:44 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op deleting in the background"
}
2021/12/11 06:22:48 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:22:53 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 06:22:58 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12152
2021/12/11 06:22:58 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 06:22:59 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:23:04 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 06:23:04 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 06:23:05 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 06:23:10 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get un-deployed
2021/12/11 06:23:15 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get un-deployed
2021/12/11 06:23:20 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get un-deployed
2021/12/11 06:23:25 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get un-deployed
2021/12/11 06:23:25 clusterSetup.go:405: App: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update got un-deployed
2021/12/11 06:23:25 eventingSetup.go:46: No eventing-con process running
2021/12/11 06:23:25 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update deleting in the background"
}
2021/12/11 06:23:26 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:23:31 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 06:23:31 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 06:23:33 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:23:38 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 06:23:38 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebMultipleHandlersOneByOne (231.51s)
=== RUN   TestEventingRebMultipleHandlersAllAtOnce
2021/12/11 06:23:48 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get un-deployed
2021/12/11 06:23:48 clusterSetup.go:405: App: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op got un-deployed
2021/12/11 06:23:48 eventingSetup.go:46: No eventing-con process running
2021/12/11 06:23:48 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op not found"
 }
}
2021/12/11 06:23:50 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:23:55 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 06:23:55 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 06:23:57 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:24:02 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 06:24:02 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:24:07 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 06:24:07 eventingSetup.go:125: Deploying app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op. Item count src bucket: 0 dst bucket: 0
2021/12/11 06:24:08 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op' in metakv",
  "warnings": null
 }
}
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:24:08 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 06:24:08 eventingSetup.go:125: Deploying app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/11 06:24:09 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/11 06:24:14 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/11 06:24:19 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get deployed
2021/12/11 06:24:24 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get deployed
2021/12/11 06:24:29 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get deployed
2021/12/11 06:24:29 clusterSetup.go:354: App: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op got deployed
2021/12/11 06:24:34 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get deployed
2021/12/11 06:24:39 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get deployed
2021/12/11 06:24:44 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get deployed
2021/12/11 06:24:44 clusterSetup.go:354: App: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:24:44 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:24:44 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 06:24:44 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/11 06:24:45 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/11 06:24:45 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/11 06:24:45 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/11 06:24:50 Rebalance progress: 12.5
2021/12/11 06:24:55 Rebalance progress: 12.5
2021/12/11 06:25:00 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:25:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:25:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 06:25:00 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002 127.0.0.1:19003] from the cluster
2021/12/11 06:25:05 Rebalance progress: 12.5
2021/12/11 06:25:10 Rebalance progress: 12.5
2021/12/11 06:25:15 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:25:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:25:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 06:25:16 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 06:25:21 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get un-deployed
2021/12/11 06:25:26 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get un-deployed
2021/12/11 06:25:31 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get un-deployed
2021/12/11 06:25:36 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get un-deployed
2021/12/11 06:25:36 clusterSetup.go:405: App: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op got un-deployed
2021/12/11 06:25:36 eventingSetup.go:46: No eventing-con process running
2021/12/11 06:25:36 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op deleting in the background"
}
2021/12/11 06:25:39 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:25:44 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 06:25:49 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6130
2021/12/11 06:25:49 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 06:25:50 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:25:55 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 06:26:00 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3258
2021/12/11 06:26:05 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3258
2021/12/11 06:26:05 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 06:26:06 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 06:26:11 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get un-deployed
2021/12/11 06:26:16 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get un-deployed
2021/12/11 06:26:21 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get un-deployed
2021/12/11 06:26:26 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get un-deployed
2021/12/11 06:26:26 clusterSetup.go:405: App: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update got un-deployed
2021/12/11 06:26:26 eventingSetup.go:46: No eventing-con process running
2021/12/11 06:26:26 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update deleting in the background"
}
2021/12/11 06:26:27 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:26:32 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 06:26:32 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 06:26:34 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:26:39 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 06:26:39 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebMultipleHandlersAllAtOnce (181.01s)
=== RUN   TestEventingSwapRebMultipleHandlers
2021/12/11 06:26:49 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get un-deployed
2021/12/11 06:26:49 clusterSetup.go:405: App: TestEventingSwapRebMultipleHandlersbucket_op_on_update got un-deployed
2021/12/11 06:26:49 eventingSetup.go:46: No eventing-con process running
2021/12/11 06:26:49 eventingSetup.go:418: Function: TestEventingSwapRebMultipleHandlersbucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingSwapRebMultipleHandlersbucket_op_on_update not found"
 }
}
2021/12/11 06:26:51 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:26:56 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 06:26:56 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 06:26:58 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:27:03 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 06:27:03 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 06:27:08 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get un-deployed
2021/12/11 06:27:08 clusterSetup.go:405: App: TestEventingSwapRebMultipleHandlerssys_test_bucket_op got un-deployed
2021/12/11 06:27:08 eventingSetup.go:46: No eventing-con process running
2021/12/11 06:27:08 eventingSetup.go:418: Function: TestEventingSwapRebMultipleHandlerssys_test_bucket_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingSwapRebMultipleHandlerssys_test_bucket_op not found"
 }
}
2021/12/11 06:27:09 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:27:14 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 06:27:14 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 06:27:16 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:27:21 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 06:27:21 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:27:21 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 06:27:21 eventingSetup.go:125: Deploying app: TestEventingSwapRebMultipleHandlersbucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/11 06:27:22 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingSwapRebMultipleHandlersbucket_op_on_update' in metakv",
  "warnings": null
 }
}
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:27:22 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 06:27:22 eventingSetup.go:125: Deploying app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op. Item count src bucket: 0 dst bucket: 0
2021/12/11 06:27:23 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingSwapRebMultipleHandlerssys_test_bucket_op' in metakv",
  "warnings": null
 }
}
2021/12/11 06:27:28 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/11 06:27:33 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get deployed
2021/12/11 06:27:39 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get deployed
2021/12/11 06:27:44 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get deployed
2021/12/11 06:27:44 clusterSetup.go:354: App: TestEventingSwapRebMultipleHandlersbucket_op_on_update got deployed
2021/12/11 06:27:49 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get deployed
2021/12/11 06:27:54 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get deployed
2021/12/11 06:27:59 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get deployed
2021/12/11 06:27:59 clusterSetup.go:354: App: TestEventingSwapRebMultipleHandlerssys_test_bucket_op got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:27:59 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:27:59 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 06:27:59 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/11 06:27:59 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/11 06:28:04 Rebalance progress: 12.5
2021/12/11 06:28:09 Rebalance progress: 12.5
2021/12/11 06:28:14 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:28:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:28:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 06:28:14 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/11 06:28:14 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/11 06:28:20 Rebalance progress: 12.5
2021/12/11 06:28:25 Rebalance progress: 12.5
2021/12/11 06:28:30 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:28:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:28:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 06:28:30 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/11 06:28:35 Rebalance progress: 12.5
2021/12/11 06:28:40 Rebalance progress: 12.5
2021/12/11 06:28:45 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:28:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:28:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 06:28:45 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 06:28:50 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get un-deployed
2021/12/11 06:28:55 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get un-deployed
2021/12/11 06:29:00 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get un-deployed
2021/12/11 06:29:05 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get un-deployed
2021/12/11 06:29:05 clusterSetup.go:405: App: TestEventingSwapRebMultipleHandlersbucket_op_on_update got un-deployed
2021/12/11 06:29:05 eventingSetup.go:46: No eventing-con process running
2021/12/11 06:29:05 eventingSetup.go:418: Function: TestEventingSwapRebMultipleHandlersbucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingSwapRebMultipleHandlersbucket_op_on_update deleting in the background"
}
2021/12/11 06:29:08 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:29:13 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 06:29:18 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7610
2021/12/11 06:29:18 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 06:29:20 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:29:25 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 06:29:30 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1880
2021/12/11 06:29:35 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1880
2021/12/11 06:29:35 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 06:29:35 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 06:29:40 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get un-deployed
2021/12/11 06:29:45 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get un-deployed
2021/12/11 06:29:50 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get un-deployed
2021/12/11 06:29:55 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get un-deployed
2021/12/11 06:29:55 clusterSetup.go:405: App: TestEventingSwapRebMultipleHandlerssys_test_bucket_op got un-deployed
2021/12/11 06:29:55 eventingSetup.go:46: No eventing-con process running
2021/12/11 06:29:55 eventingSetup.go:418: Function: TestEventingSwapRebMultipleHandlerssys_test_bucket_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingSwapRebMultipleHandlerssys_test_bucket_op deleting in the background"
}
2021/12/11 06:29:57 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:30:02 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 06:30:02 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 06:30:04 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:30:09 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 06:30:09 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingSwapRebMultipleHandlers (209.79s)
=== RUN   TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne
2021/12/11 06:30:19 clusterSetup.go:397: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/11 06:30:19 clusterSetup.go:405: App: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/11 06:30:19 eventingSetup.go:46: No eventing-con process running
2021/12/11 06:30:19 eventingSetup.go:418: Function: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne not found"
 }
}
2021/12/11 06:30:21 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:30:26 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 06:30:26 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 06:30:28 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:30:33 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 06:30:33 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:30:38 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 06:30:38 eventingSetup.go:125: Deploying app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne. Item count src bucket: 0 dst bucket: 0
2021/12/11 06:30:39 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne' in metakv",
  "warnings": null
 }
}
2021/12/11 06:30:44 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/11 06:30:49 clusterSetup.go:346: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/11 06:30:54 clusterSetup.go:346: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/11 06:30:59 clusterSetup.go:346: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/11 06:30:59 clusterSetup.go:354: App: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:30:59 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:30:59 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 06:30:59 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/11 06:30:59 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/11 06:31:19 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:31:20 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:31:20 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:31:50 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:31:50 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 06:31:50 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/11 06:31:55 Rebalance progress: 12.5
2021/12/11 06:32:00 Rebalance progress: 12.5
2021/12/11 06:32:05 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:32:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:32:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 06:32:05 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/11 06:32:05 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/11 06:32:10 Rebalance progress: 12.5
2021/12/11 06:32:15 Rebalance progress: 12.5
2021/12/11 06:32:20 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:32:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:32:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 06:32:20 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/11 06:32:25 Rebalance progress: 12.5
2021/12/11 06:32:30 Rebalance progress: 12.5
2021/12/11 06:32:35 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:32:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:32:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 06:32:35 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 06:32:40 clusterSetup.go:397: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/11 06:32:46 clusterSetup.go:397: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/11 06:32:51 clusterSetup.go:397: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/11 06:32:56 clusterSetup.go:397: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/11 06:32:56 clusterSetup.go:405: App: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/11 06:32:56 eventingSetup.go:46: No eventing-con process running
2021/12/11 06:32:56 eventingSetup.go:418: Function: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne deleting in the background"
}
2021/12/11 06:32:57 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:33:02 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 06:33:07 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 11155
2021/12/11 06:33:07 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 06:33:09 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:33:14 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 06:33:19 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 11155
2021/12/11 06:33:19 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne (190.12s)
=== RUN   TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne
2021/12/11 06:33:29 clusterSetup.go:397: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/11 06:33:29 clusterSetup.go:405: App: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/11 06:33:29 eventingSetup.go:46: No eventing-con process running
2021/12/11 06:33:29 eventingSetup.go:418: Function: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne not found"
 }
}
2021/12/11 06:33:31 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:33:36 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 06:33:36 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 06:33:38 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:33:43 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 06:33:43 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:33:48 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 06:33:48 eventingSetup.go:125: Deploying app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne. Item count src bucket: 0 dst bucket: 0
2021/12/11 06:33:49 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne' in metakv",
  "warnings": null
 }
}
2021/12/11 06:33:54 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/11 06:33:59 clusterSetup.go:346: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/11 06:34:04 clusterSetup.go:346: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/11 06:34:09 clusterSetup.go:346: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/11 06:34:09 clusterSetup.go:354: App: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:34:09 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:34:09 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 06:34:09 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/11 06:34:09 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/11 06:34:29 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:34:29 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:34:29 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:34:39 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:34:39 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 06:34:39 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/11 06:34:40 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/11 06:35:00 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:35:01 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:35:01 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:35:11 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:35:11 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 06:35:11 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/11 06:35:11 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/11 06:35:31 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:35:31 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:35:31 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:35:41 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:35:41 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 06:35:41 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/11 06:35:42 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/11 06:36:02 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:36:03 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:36:03 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:36:13 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:36:13 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 06:36:13 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/11 06:36:13 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/11 06:36:34 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:36:34 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:36:34 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:36:44 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:36:44 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 06:36:44 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/11 06:36:49 Rebalance progress: 12.5
2021/12/11 06:36:54 Rebalance progress: 12.5
2021/12/11 06:36:59 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:36:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:36:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 06:36:59 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/11 06:37:00 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/11 06:37:05 Rebalance progress: 12.5
2021/12/11 06:37:10 Rebalance progress: 12.5
2021/12/11 06:37:15 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:37:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:37:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 06:37:15 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/11 06:37:20 Rebalance progress: 12.5
2021/12/11 06:37:25 Rebalance progress: 12.5
2021/12/11 06:37:30 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:37:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:37:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 06:37:30 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 06:37:35 clusterSetup.go:397: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/11 06:37:40 clusterSetup.go:397: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/11 06:37:45 clusterSetup.go:397: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/11 06:37:50 clusterSetup.go:397: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/11 06:37:50 clusterSetup.go:405: App: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/11 06:37:50 eventingSetup.go:46: No eventing-con process running
2021/12/11 06:37:50 eventingSetup.go:418: Function: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne deleting in the background"
}
2021/12/11 06:37:52 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:37:57 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 06:38:02 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 21596
2021/12/11 06:38:07 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 21596
2021/12/11 06:38:07 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 06:38:09 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:38:14 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 06:38:19 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 21596
2021/12/11 06:38:19 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne (299.45s)
=== RUN   TestEventingFailoverOnUpdateBucketOp
2021/12/11 06:38:29 clusterSetup.go:397: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get un-deployed
2021/12/11 06:38:29 clusterSetup.go:405: App: TestEventingFailoverOnUpdateBucketOp got un-deployed
2021/12/11 06:38:29 eventingSetup.go:46: No eventing-con process running
2021/12/11 06:38:29 eventingSetup.go:418: Function: TestEventingFailoverOnUpdateBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingFailoverOnUpdateBucketOp not found"
 }
}
2021/12/11 06:38:30 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:38:35 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 06:38:35 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 06:38:37 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:38:42 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 06:38:42 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:38:47 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 06:38:47 eventingSetup.go:125: Deploying app: TestEventingFailoverOnUpdateBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/11 06:38:48 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingFailoverOnUpdateBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/11 06:38:53 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/11 06:38:58 clusterSetup.go:346: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get deployed
2021/12/11 06:39:03 clusterSetup.go:346: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get deployed
2021/12/11 06:39:08 clusterSetup.go:346: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get deployed
2021/12/11 06:39:08 clusterSetup.go:354: App: TestEventingFailoverOnUpdateBucketOp got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:39:08 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:39:08 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 06:39:08 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/11 06:39:08 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/11 06:39:13 Rebalance progress: 12.5
2021/12/11 06:39:18 Rebalance progress: 12.5
2021/12/11 06:39:23 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:39:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:39:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 06:39:24 clusterSetup.go:229: Failing over: [127.0.0.1:19001]
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:39:34 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:39:34 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 06:39:34 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/11 06:39:39 Rebalance progress: 12.5
2021/12/11 06:39:44 Rebalance progress: 12.5
2021/12/11 06:39:49 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:39:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:39:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 06:39:49 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/11 06:39:49 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/11 06:39:54 Rebalance progress: 12.5
2021/12/11 06:39:59 Rebalance progress: 12.5
2021/12/11 06:40:04 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:40:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:40:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 06:40:04 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/11 06:40:09 Rebalance progress: 12.5
2021/12/11 06:40:14 Rebalance progress: 12.5
2021/12/11 06:40:19 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:40:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:40:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 06:40:19 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 06:40:24 clusterSetup.go:397: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get un-deployed
2021/12/11 06:40:29 clusterSetup.go:397: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get un-deployed
2021/12/11 06:40:34 clusterSetup.go:397: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get un-deployed
2021/12/11 06:40:39 clusterSetup.go:397: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get un-deployed
2021/12/11 06:40:39 clusterSetup.go:405: App: TestEventingFailoverOnUpdateBucketOp got un-deployed
2021/12/11 06:40:39 eventingSetup.go:46: No eventing-con process running
2021/12/11 06:40:40 eventingSetup.go:418: Function: TestEventingFailoverOnUpdateBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingFailoverOnUpdateBucketOp deleting in the background"
}
2021/12/11 06:40:41 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:40:46 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 06:40:51 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 8604
2021/12/11 06:40:56 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 8604
2021/12/11 06:40:56 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 06:40:58 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:41:03 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 06:41:08 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 8604
2021/12/11 06:41:08 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingFailoverOnUpdateBucketOp (169.50s)
=== RUN   TestEventingKVRebalanceOnUpdateBucketOp
2021/12/11 06:41:18 clusterSetup.go:397: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get un-deployed
2021/12/11 06:41:18 clusterSetup.go:405: App: TestEventingKVRebalanceOnUpdateBucketOp got un-deployed
2021/12/11 06:41:18 eventingSetup.go:46: No eventing-con process running
2021/12/11 06:41:18 eventingSetup.go:418: Function: TestEventingKVRebalanceOnUpdateBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingKVRebalanceOnUpdateBucketOp not found"
 }
}
2021/12/11 06:41:20 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:41:25 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 06:41:25 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 06:41:27 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:41:32 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 06:41:32 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:41:37 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 06:41:37 eventingSetup.go:125: Deploying app: TestEventingKVRebalanceOnUpdateBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/11 06:41:38 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingKVRebalanceOnUpdateBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/11 06:41:43 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/11 06:41:48 clusterSetup.go:346: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get deployed
2021/12/11 06:41:53 clusterSetup.go:346: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get deployed
2021/12/11 06:41:58 clusterSetup.go:346: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get deployed
2021/12/11 06:41:58 clusterSetup.go:354: App: TestEventingKVRebalanceOnUpdateBucketOp got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:41:58 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:41:58 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 06:41:58 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing,kv to the cluster
2021/12/11 06:41:58 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/11 06:42:03 Rebalance progress: 12.5
2021/12/11 06:42:08 Rebalance progress: 12.5
2021/12/11 06:42:13 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:42:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:42:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 06:42:13 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing,kv to the cluster
2021/12/11 06:42:14 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/11 06:42:19 Rebalance progress: 12.5
2021/12/11 06:42:24 Rebalance progress: 12.5
2021/12/11 06:42:29 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:42:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:42:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 06:42:29 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/11 06:42:34 Rebalance progress: 12.5
2021/12/11 06:42:39 Rebalance progress: 12.5
2021/12/11 06:42:44 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:42:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:42:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 06:42:44 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 06:42:49 clusterSetup.go:397: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get un-deployed
2021/12/11 06:42:54 clusterSetup.go:397: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get un-deployed
2021/12/11 06:42:59 clusterSetup.go:397: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get un-deployed
2021/12/11 06:43:04 clusterSetup.go:397: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get un-deployed
2021/12/11 06:43:04 clusterSetup.go:405: App: TestEventingKVRebalanceOnUpdateBucketOp got un-deployed
2021/12/11 06:43:04 eventingSetup.go:46: No eventing-con process running
2021/12/11 06:43:04 eventingSetup.go:418: Function: TestEventingKVRebalanceOnUpdateBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingKVRebalanceOnUpdateBucketOp deleting in the background"
}
2021/12/11 06:43:06 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:43:11 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 06:43:16 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6089
2021/12/11 06:43:16 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 06:43:17 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:43:22 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 06:43:27 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6089
2021/12/11 06:43:27 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingKVRebalanceOnUpdateBucketOp (139.42s)
=== RUN   TestEventingRebInWithOps
2021/12/11 06:43:37 clusterSetup.go:397: Waiting for app: TestEventingRebInWithOps to get un-deployed
2021/12/11 06:43:37 clusterSetup.go:405: App: TestEventingRebInWithOps got un-deployed
2021/12/11 06:43:37 eventingSetup.go:46: No eventing-con process running
2021/12/11 06:43:37 eventingSetup.go:418: Function: TestEventingRebInWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebInWithOps not found"
 }
}
2021/12/11 06:43:39 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:43:44 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 06:43:44 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 06:43:46 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:43:51 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 06:43:51 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:43:56 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 06:43:56 eventingSetup.go:125: Deploying app: TestEventingRebInWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/11 06:43:57 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebInWithOps' in metakv",
  "warnings": null
 }
}
2021/12/11 06:44:02 clusterSetup.go:346: Waiting for app: TestEventingRebInWithOps to get deployed
2021/12/11 06:44:07 clusterSetup.go:346: Waiting for app: TestEventingRebInWithOps to get deployed
2021/12/11 06:44:12 clusterSetup.go:346: Waiting for app: TestEventingRebInWithOps to get deployed
2021/12/11 06:44:17 clusterSetup.go:346: Waiting for app: TestEventingRebInWithOps to get deployed
2021/12/11 06:44:17 clusterSetup.go:354: App: TestEventingRebInWithOps got deployed
2021/12/11 06:44:17 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/11 06:44:17 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/11 06:44:18 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/11 06:44:23 Rebalance progress: 12.5
2021/12/11 06:44:28 Rebalance progress: 12.5
2021/12/11 06:44:33 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:44:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:44:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 06:44:33 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/11 06:44:38 Rebalance progress: 12.5
2021/12/11 06:44:43 Rebalance progress: 12.5
2021/12/11 06:44:48 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:44:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:44:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 06:44:48 Waiting for src bucket item count to get to: 200000 curr count: 34250
2021/12/11 06:44:53 Waiting for src bucket item count to get to: 200000 curr count: 50207
2021/12/11 06:44:58 Waiting for src bucket item count to get to: 200000 curr count: 50207
2021/12/11 06:45:03 Waiting for src bucket item count to get to: 200000 curr count: 66684
2021/12/11 06:45:08 Waiting for src bucket item count to get to: 200000 curr count: 66684
2021/12/11 06:45:13 Waiting for src bucket item count to get to: 200000 curr count: 81175
2021/12/11 06:45:18 Waiting for src bucket item count to get to: 200000 curr count: 81175
2021/12/11 06:45:23 Waiting for src bucket item count to get to: 200000 curr count: 97406
2021/12/11 06:45:28 Waiting for src bucket item count to get to: 200000 curr count: 97406
2021/12/11 06:45:33 Waiting for src bucket item count to get to: 200000 curr count: 113632
2021/12/11 06:45:38 Waiting for src bucket item count to get to: 200000 curr count: 113632
2021/12/11 06:45:43 Waiting for src bucket item count to get to: 200000 curr count: 130001
2021/12/11 06:45:48 Waiting for src bucket item count to get to: 200000 curr count: 130001
2021/12/11 06:45:53 Waiting for src bucket item count to get to: 200000 curr count: 130001
2021/12/11 06:45:58 Waiting for src bucket item count to get to: 200000 curr count: 146279
2021/12/11 06:46:03 Waiting for src bucket item count to get to: 200000 curr count: 146279
2021/12/11 06:46:08 Waiting for src bucket item count to get to: 200000 curr count: 162285
2021/12/11 06:46:13 Waiting for src bucket item count to get to: 200000 curr count: 162285
2021/12/11 06:46:18 Waiting for src bucket item count to get to: 200000 curr count: 176893
2021/12/11 06:46:23 Waiting for src bucket item count to get to: 200000 curr count: 176893
2021/12/11 06:46:28 Waiting for src bucket item count to get to: 200000 curr count: 192945
2021/12/11 06:46:33 Waiting for src bucket item count to get to: 200000 curr count: 192945
2021/12/11 06:46:38 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/11 06:46:38 Post SET ops. Expected item count: 200000 got 200000
2021/12/11 06:46:38 Starting bucket ops to source bucket
2021/12/11 06:50:24 Waiting for dst bucket item count to get to: 0 curr count: 17283
2021/12/11 06:50:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 06:50:24 Post DELETE ops. Expected item count: 0 got 0
2021/12/11 06:50:24 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 06:50:29 Waiting for app: TestEventingRebInWithOps to get un-deployed
2021/12/11 06:50:34 Waiting for app: TestEventingRebInWithOps to get un-deployed
2021/12/11 06:50:39 Waiting for app: TestEventingRebInWithOps to get un-deployed
2021/12/11 06:50:44 Waiting for app: TestEventingRebInWithOps to get un-deployed
2021/12/11 06:50:44 App: TestEventingRebInWithOps got un-deployed
2021/12/11 06:50:44 No eventing-con process running
2021/12/11 06:50:44 Function: TestEventingRebInWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebInWithOps deleting in the background"
}
2021/12/11 06:50:46 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:50:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 06:50:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 06:50:52 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:50:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 06:50:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebInWithOps (449.89s)
=== RUN   TestEventingRebOutWithOps
2021/12/11 06:51:02 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/11 06:51:03 Removing node(s): [] from the cluster
2021/12/11 06:51:08 Rebalance progress: 100
2021/12/11 06:51:13 Waiting for app: TestEventingRebOutWithOps to get un-deployed
2021/12/11 06:51:13 App: TestEventingRebOutWithOps got un-deployed
2021/12/11 06:51:13 No eventing-con process running
2021/12/11 06:51:13 Function: TestEventingRebOutWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebOutWithOps not found"
 }
}
2021/12/11 06:51:14 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:51:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 06:51:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 06:51:21 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:51:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 06:51:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:51:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 06:51:31 Deploying app: TestEventingRebOutWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/11 06:51:32 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebOutWithOps' in metakv",
  "warnings": null
 }
}
2021/12/11 06:51:37 Waiting for app: TestEventingRebOutWithOps to get deployed
2021/12/11 06:51:42 Waiting for app: TestEventingRebOutWithOps to get deployed
2021/12/11 06:51:47 Waiting for app: TestEventingRebOutWithOps to get deployed
2021/12/11 06:51:52 Waiting for app: TestEventingRebOutWithOps to get deployed
2021/12/11 06:51:52 App: TestEventingRebOutWithOps got deployed
2021/12/11 06:51:52 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/11 06:51:52 Starting bucket ops to source bucket
2021/12/11 06:51:57 Rebalance progress: 12.5
2021/12/11 06:52:02 Rebalance progress: 12.5
2021/12/11 06:52:07 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:52:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:52:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 06:52:07 Waiting for src bucket item count to get to: 200000 curr count: 10475
2021/12/11 06:52:12 Waiting for src bucket item count to get to: 200000 curr count: 26625
2021/12/11 06:52:17 Waiting for src bucket item count to get to: 200000 curr count: 26625
2021/12/11 06:52:22 Waiting for src bucket item count to get to: 200000 curr count: 43098
2021/12/11 06:52:27 Waiting for src bucket item count to get to: 200000 curr count: 43098
2021/12/11 06:52:33 Waiting for src bucket item count to get to: 200000 curr count: 59793
2021/12/11 06:52:38 Waiting for src bucket item count to get to: 200000 curr count: 59793
2021/12/11 06:52:43 Waiting for src bucket item count to get to: 200000 curr count: 59793
2021/12/11 06:52:48 Waiting for src bucket item count to get to: 200000 curr count: 75439
2021/12/11 06:52:53 Waiting for src bucket item count to get to: 200000 curr count: 75439
2021/12/11 06:52:58 Waiting for src bucket item count to get to: 200000 curr count: 91694
2021/12/11 06:53:03 Waiting for src bucket item count to get to: 200000 curr count: 91694
2021/12/11 06:53:08 Waiting for src bucket item count to get to: 200000 curr count: 108164
2021/12/11 06:53:13 Waiting for src bucket item count to get to: 200000 curr count: 108164
2021/12/11 06:53:18 Waiting for src bucket item count to get to: 200000 curr count: 124258
2021/12/11 06:53:23 Waiting for src bucket item count to get to: 200000 curr count: 124258
2021/12/11 06:53:28 Waiting for src bucket item count to get to: 200000 curr count: 140868
2021/12/11 06:53:33 Waiting for src bucket item count to get to: 200000 curr count: 140868
2021/12/11 06:53:38 Waiting for src bucket item count to get to: 200000 curr count: 157467
2021/12/11 06:53:43 Waiting for src bucket item count to get to: 200000 curr count: 157467
2021/12/11 06:53:48 Waiting for src bucket item count to get to: 200000 curr count: 173247
2021/12/11 06:53:53 Waiting for src bucket item count to get to: 200000 curr count: 173247
2021/12/11 06:53:58 Waiting for src bucket item count to get to: 200000 curr count: 189402
2021/12/11 06:54:03 Waiting for src bucket item count to get to: 200000 curr count: 189402
2021/12/11 06:54:08 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/11 06:54:08 Post SET ops. Expected item count: 200000 got 200000
2021/12/11 06:54:08 Starting bucket ops to source bucket
2021/12/11 06:57:53 Waiting for dst bucket item count to get to: 0 curr count: 15426
2021/12/11 06:57:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 06:57:53 Post DELETE ops. Expected item count: 0 got 0
2021/12/11 06:57:53 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 06:57:58 Waiting for app: TestEventingRebOutWithOps to get un-deployed
2021/12/11 06:58:03 Waiting for app: TestEventingRebOutWithOps to get un-deployed
2021/12/11 06:58:08 Waiting for app: TestEventingRebOutWithOps to get un-deployed
2021/12/11 06:58:13 Waiting for app: TestEventingRebOutWithOps to get un-deployed
2021/12/11 06:58:13 App: TestEventingRebOutWithOps got un-deployed
2021/12/11 06:58:13 No eventing-con process running
2021/12/11 06:58:13 Function: TestEventingRebOutWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebOutWithOps deleting in the background"
}
2021/12/11 06:58:15 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:58:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 06:58:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 06:58:21 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:58:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 06:58:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebOutWithOps (449.14s)
=== RUN   TestEventingSwapRebWithOps
2021/12/11 06:58:36 Waiting for app: TestEventingSwapRebWithOps to get un-deployed
2021/12/11 06:58:36 App: TestEventingSwapRebWithOps got un-deployed
2021/12/11 06:58:37 No eventing-con process running
2021/12/11 06:58:37 Function: TestEventingSwapRebWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingSwapRebWithOps not found"
 }
}
2021/12/11 06:58:38 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:58:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 06:58:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 06:58:45 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:58:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 06:58:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:58:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 06:58:55 Deploying app: TestEventingSwapRebWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/11 06:58:56 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingSwapRebWithOps' in metakv",
  "warnings": null
 }
}
2021/12/11 06:59:01 Waiting for app: TestEventingSwapRebWithOps to get deployed
2021/12/11 06:59:06 Waiting for app: TestEventingSwapRebWithOps to get deployed
2021/12/11 06:59:11 Waiting for app: TestEventingSwapRebWithOps to get deployed
2021/12/11 06:59:16 Waiting for app: TestEventingSwapRebWithOps to get deployed
2021/12/11 06:59:16 App: TestEventingSwapRebWithOps got deployed
2021/12/11 06:59:16 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/11 06:59:16 Starting bucket ops to source bucket
2021/12/11 06:59:16 Removing node(s): [] from the cluster
2021/12/11 06:59:22 Rebalance progress: 12.5
2021/12/11 06:59:27 Rebalance progress: 12.5
2021/12/11 06:59:32 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:59:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:59:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 06:59:32 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/11 06:59:32 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/11 06:59:37 Rebalance progress: 12.5
2021/12/11 06:59:42 Rebalance progress: 12.5
2021/12/11 06:59:47 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:59:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 06:59:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 06:59:47 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/11 06:59:52 Rebalance progress: 12.5
2021/12/11 06:59:57 Rebalance progress: 12.5
2021/12/11 07:00:02 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 07:00:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 07:00:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 07:00:02 Waiting for src bucket item count to get to: 200000 curr count: 66768
2021/12/11 07:00:07 Waiting for src bucket item count to get to: 200000 curr count: 66768
2021/12/11 07:00:12 Waiting for src bucket item count to get to: 200000 curr count: 81465
2021/12/11 07:00:17 Waiting for src bucket item count to get to: 200000 curr count: 81465
2021/12/11 07:00:22 Waiting for src bucket item count to get to: 200000 curr count: 97990
2021/12/11 07:00:27 Waiting for src bucket item count to get to: 200000 curr count: 97990
2021/12/11 07:00:32 Waiting for src bucket item count to get to: 200000 curr count: 114564
2021/12/11 07:00:37 Waiting for src bucket item count to get to: 200000 curr count: 114564
2021/12/11 07:00:42 Waiting for src bucket item count to get to: 200000 curr count: 131169
2021/12/11 07:00:47 Waiting for src bucket item count to get to: 200000 curr count: 131169
2021/12/11 07:00:52 Waiting for src bucket item count to get to: 200000 curr count: 147637
2021/12/11 07:00:57 Waiting for src bucket item count to get to: 200000 curr count: 147637
2021/12/11 07:01:02 Waiting for src bucket item count to get to: 200000 curr count: 147637
2021/12/11 07:01:07 Waiting for src bucket item count to get to: 200000 curr count: 164008
2021/12/11 07:01:12 Waiting for src bucket item count to get to: 200000 curr count: 164008
2021/12/11 07:01:17 Waiting for src bucket item count to get to: 200000 curr count: 178853
2021/12/11 07:01:22 Waiting for src bucket item count to get to: 200000 curr count: 178853
2021/12/11 07:01:27 Waiting for src bucket item count to get to: 200000 curr count: 195221
2021/12/11 07:01:32 Waiting for src bucket item count to get to: 200000 curr count: 195221
2021/12/11 07:01:37 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/11 07:01:37 Post SET ops. Expected item count: 200000 got 200000
2021/12/11 07:01:37 Starting bucket ops to source bucket
2021/12/11 07:05:23 Waiting for dst bucket item count to get to: 0 curr count: 16528
2021/12/11 07:05:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 07:05:23 Post DELETE ops. Expected item count: 0 got 0
2021/12/11 07:05:23 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 07:05:28 Waiting for app: TestEventingSwapRebWithOps to get un-deployed
2021/12/11 07:05:33 Waiting for app: TestEventingSwapRebWithOps to get un-deployed
2021/12/11 07:05:38 Waiting for app: TestEventingSwapRebWithOps to get un-deployed
2021/12/11 07:05:43 Waiting for app: TestEventingSwapRebWithOps to get un-deployed
2021/12/11 07:05:43 App: TestEventingSwapRebWithOps got un-deployed
2021/12/11 07:05:43 No eventing-con process running
2021/12/11 07:05:43 Function: TestEventingSwapRebWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingSwapRebWithOps deleting in the background"
}
2021/12/11 07:05:45 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 07:05:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 07:05:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 07:05:52 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 07:05:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 07:05:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingSwapRebWithOps (450.42s)
=== RUN   TestKVRebInWithOps
2021/12/11 07:06:07 Waiting for app: TestKVRebInWithOps to get un-deployed
2021/12/11 07:06:07 App: TestKVRebInWithOps got un-deployed
2021/12/11 07:06:07 No eventing-con process running
2021/12/11 07:06:07 Function: TestKVRebInWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVRebInWithOps not found"
 }
}
2021/12/11 07:06:09 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 07:06:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 07:06:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 07:06:15 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 07:06:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 07:06:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 07:06:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 07:06:25 Deploying app: TestKVRebInWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/11 07:06:27 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVRebInWithOps' in metakv",
  "warnings": null
 }
}
2021/12/11 07:06:32 Waiting for app: TestKVRebInWithOps to get deployed
2021/12/11 07:06:37 Waiting for app: TestKVRebInWithOps to get deployed
2021/12/11 07:06:42 Waiting for app: TestKVRebInWithOps to get deployed
2021/12/11 07:06:47 Waiting for app: TestKVRebInWithOps to get deployed
2021/12/11 07:06:47 App: TestKVRebInWithOps got deployed
2021/12/11 07:06:47 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/11 07:06:47 Starting bucket ops to source bucket
2021/12/11 07:06:47 Removing node(s): [] from the cluster
2021/12/11 07:06:52 Rebalance progress: 12.5
2021/12/11 07:06:57 Rebalance progress: 12.5
2021/12/11 07:07:02 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 07:07:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 07:07:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 07:07:02 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/11 07:07:07 Rebalance progress: 12.5
2021/12/11 07:07:12 Rebalance progress: 12.5
2021/12/11 07:07:17 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 07:07:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 07:07:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 07:07:17 Waiting for src bucket item count to get to: 200000 curr count: 35099
2021/12/11 07:07:22 Waiting for src bucket item count to get to: 200000 curr count: 51241
2021/12/11 07:07:27 Waiting for src bucket item count to get to: 200000 curr count: 51241
2021/12/11 07:07:32 Waiting for src bucket item count to get to: 200000 curr count: 67750
2021/12/11 07:07:37 Waiting for src bucket item count to get to: 200000 curr count: 67750
2021/12/11 07:07:42 Waiting for src bucket item count to get to: 200000 curr count: 82916
2021/12/11 07:07:47 Waiting for src bucket item count to get to: 200000 curr count: 82916
2021/12/11 07:07:52 Waiting for src bucket item count to get to: 200000 curr count: 99370
2021/12/11 07:07:57 Waiting for src bucket item count to get to: 200000 curr count: 99370
2021/12/11 07:08:02 Waiting for src bucket item count to get to: 200000 curr count: 99370
2021/12/11 07:08:07 Waiting for src bucket item count to get to: 200000 curr count: 115949
2021/12/11 07:08:12 Waiting for src bucket item count to get to: 200000 curr count: 115949
2021/12/11 07:08:17 Waiting for src bucket item count to get to: 200000 curr count: 132127
2021/12/11 07:08:22 Waiting for src bucket item count to get to: 200000 curr count: 132127
2021/12/11 07:08:27 Waiting for src bucket item count to get to: 200000 curr count: 148485
2021/12/11 07:08:32 Waiting for src bucket item count to get to: 200000 curr count: 148485
2021/12/11 07:08:37 Waiting for src bucket item count to get to: 200000 curr count: 164765
2021/12/11 07:08:42 Waiting for src bucket item count to get to: 200000 curr count: 164765
2021/12/11 07:08:47 Waiting for src bucket item count to get to: 200000 curr count: 179618
2021/12/11 07:08:52 Waiting for src bucket item count to get to: 200000 curr count: 179618
2021/12/11 07:08:57 Waiting for src bucket item count to get to: 200000 curr count: 195791
2021/12/11 07:09:02 Waiting for src bucket item count to get to: 200000 curr count: 195791
2021/12/11 07:09:07 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/11 07:09:07 Post SET ops. Expected item count: 200000 got 200000
2021/12/11 07:09:07 Starting bucket ops to source bucket
2021/12/11 07:12:53 Waiting for dst bucket item count to get to: 0 curr count: 15511
2021/12/11 07:12:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 07:12:53 Post DELETE ops. Expected item count: 0 got 0
2021/12/11 07:12:53 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 07:12:58 Waiting for app: TestKVRebInWithOps to get un-deployed
2021/12/11 07:13:03 Waiting for app: TestKVRebInWithOps to get un-deployed
2021/12/11 07:13:08 Waiting for app: TestKVRebInWithOps to get un-deployed
2021/12/11 07:13:13 Waiting for app: TestKVRebInWithOps to get un-deployed
2021/12/11 07:13:13 App: TestKVRebInWithOps got un-deployed
2021/12/11 07:13:13 No eventing-con process running
2021/12/11 07:13:13 Function: TestKVRebInWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVRebInWithOps deleting in the background"
}
2021/12/11 07:13:15 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 07:13:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 07:13:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 07:13:21 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 07:13:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 07:13:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebInWithOps (449.37s)
=== RUN   TestKVRebOutWithOps
2021/12/11 07:13:31 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/11 07:13:32 Removing node(s): [] from the cluster
2021/12/11 07:13:37 Rebalance progress: 100
2021/12/11 07:13:42 Waiting for app: TestKVRebOutWithOps to get un-deployed
2021/12/11 07:13:42 App: TestKVRebOutWithOps got un-deployed
2021/12/11 07:13:42 No eventing-con process running
2021/12/11 07:13:42 Function: TestKVRebOutWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVRebOutWithOps not found"
 }
}
2021/12/11 07:13:43 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 07:13:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 07:13:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 07:13:50 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 07:13:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 07:13:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 07:14:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 07:14:00 Deploying app: TestKVRebOutWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/11 07:14:01 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVRebOutWithOps' in metakv",
  "warnings": null
 }
}
2021/12/11 07:14:06 Waiting for app: TestKVRebOutWithOps to get deployed
2021/12/11 07:14:11 Waiting for app: TestKVRebOutWithOps to get deployed
2021/12/11 07:14:16 Waiting for app: TestKVRebOutWithOps to get deployed
2021/12/11 07:14:21 Waiting for app: TestKVRebOutWithOps to get deployed
2021/12/11 07:14:21 App: TestKVRebOutWithOps got deployed
2021/12/11 07:14:21 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/11 07:14:21 Starting bucket ops to source bucket
2021/12/11 07:14:26 Rebalance progress: 12.5
2021/12/11 07:14:31 Rebalance progress: 12.5
2021/12/11 07:14:36 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 07:14:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 07:14:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 07:14:36 Waiting for src bucket item count to get to: 200000 curr count: 12150
2021/12/11 07:14:41 Waiting for src bucket item count to get to: 200000 curr count: 28399
2021/12/11 07:14:46 Waiting for src bucket item count to get to: 200000 curr count: 28399
2021/12/11 07:14:51 Waiting for src bucket item count to get to: 200000 curr count: 44882
2021/12/11 07:14:56 Waiting for src bucket item count to get to: 200000 curr count: 44882
2021/12/11 07:15:01 Waiting for src bucket item count to get to: 200000 curr count: 61404
2021/12/11 07:15:06 Waiting for src bucket item count to get to: 200000 curr count: 61404
2021/12/11 07:15:11 Waiting for src bucket item count to get to: 200000 curr count: 61404
2021/12/11 07:15:16 Waiting for src bucket item count to get to: 200000 curr count: 76741
2021/12/11 07:15:21 Waiting for src bucket item count to get to: 200000 curr count: 76741
2021/12/11 07:15:26 Waiting for src bucket item count to get to: 200000 curr count: 92893
2021/12/11 07:15:31 Waiting for src bucket item count to get to: 200000 curr count: 92893
2021/12/11 07:15:36 Waiting for src bucket item count to get to: 200000 curr count: 109113
2021/12/11 07:15:41 Waiting for src bucket item count to get to: 200000 curr count: 109113
2021/12/11 07:15:46 Waiting for src bucket item count to get to: 200000 curr count: 125745
2021/12/11 07:15:51 Waiting for src bucket item count to get to: 200000 curr count: 125745
2021/12/11 07:15:56 Waiting for src bucket item count to get to: 200000 curr count: 142134
2021/12/11 07:16:01 Waiting for src bucket item count to get to: 200000 curr count: 142134
2021/12/11 07:16:06 Waiting for src bucket item count to get to: 200000 curr count: 158783
2021/12/11 07:16:11 Waiting for src bucket item count to get to: 200000 curr count: 158783
2021/12/11 07:16:16 Waiting for src bucket item count to get to: 200000 curr count: 174201
2021/12/11 07:16:21 Waiting for src bucket item count to get to: 200000 curr count: 174201
2021/12/11 07:16:26 Waiting for src bucket item count to get to: 200000 curr count: 190550
2021/12/11 07:16:32 Waiting for src bucket item count to get to: 200000 curr count: 190550
2021/12/11 07:16:37 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/11 07:16:37 Post SET ops. Expected item count: 200000 got 200000
2021/12/11 07:16:37 Starting bucket ops to source bucket
2021/12/11 07:20:22 Waiting for dst bucket item count to get to: 0 curr count: 13447
2021/12/11 07:20:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 07:20:22 Post DELETE ops. Expected item count: 0 got 0
2021/12/11 07:20:22 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 07:20:27 Waiting for app: TestKVRebOutWithOps to get un-deployed
2021/12/11 07:20:32 Waiting for app: TestKVRebOutWithOps to get un-deployed
2021/12/11 07:20:37 Waiting for app: TestKVRebOutWithOps to get un-deployed
2021/12/11 07:20:42 Waiting for app: TestKVRebOutWithOps to get un-deployed
2021/12/11 07:20:42 App: TestKVRebOutWithOps got un-deployed
2021/12/11 07:20:42 No eventing-con process running
2021/12/11 07:20:42 Function: TestKVRebOutWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVRebOutWithOps deleting in the background"
}
2021/12/11 07:20:44 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 07:20:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 07:20:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 07:20:50 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 07:20:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 07:20:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebOutWithOps (449.06s)
=== RUN   TestKVSwapRebWithOps
2021/12/11 07:21:05 Waiting for app: TestKVSwapRebWithOps to get un-deployed
2021/12/11 07:21:05 App: TestKVSwapRebWithOps got un-deployed
2021/12/11 07:21:05 No eventing-con process running
2021/12/11 07:21:05 Function: TestKVSwapRebWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVSwapRebWithOps not found"
 }
}
2021/12/11 07:21:07 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 07:21:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 07:21:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 07:21:14 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 07:21:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 07:21:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 07:21:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 07:21:24 Deploying app: TestKVSwapRebWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/11 07:21:25 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVSwapRebWithOps' in metakv",
  "warnings": null
 }
}
2021/12/11 07:21:30 Waiting for app: TestKVSwapRebWithOps to get deployed
2021/12/11 07:21:35 Waiting for app: TestKVSwapRebWithOps to get deployed
2021/12/11 07:21:40 Waiting for app: TestKVSwapRebWithOps to get deployed
2021/12/11 07:21:45 Waiting for app: TestKVSwapRebWithOps to get deployed
2021/12/11 07:21:45 App: TestKVSwapRebWithOps got deployed
2021/12/11 07:21:45 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/11 07:21:45 Starting bucket ops to source bucket
2021/12/11 07:21:46 Removing node(s): [] from the cluster
2021/12/11 07:21:51 Rebalance progress: 12.5
2021/12/11 07:21:56 Rebalance progress: 12.5
2021/12/11 07:22:01 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 07:22:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 07:22:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 07:22:01 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/11 07:22:01 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/11 07:22:06 Rebalance progress: 12.5
2021/12/11 07:22:11 Rebalance progress: 12.5
2021/12/11 07:22:16 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 07:22:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 07:22:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 07:22:16 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/11 07:22:21 Rebalance progress: 12.5
2021/12/11 07:22:26 Rebalance progress: 12.5
2021/12/11 07:22:31 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 07:22:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 07:22:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 07:22:31 Waiting for src bucket item count to get to: 200000 curr count: 69129
2021/12/11 07:22:36 Waiting for src bucket item count to get to: 200000 curr count: 69129
2021/12/11 07:22:41 Waiting for src bucket item count to get to: 200000 curr count: 83861
2021/12/11 07:22:46 Waiting for src bucket item count to get to: 200000 curr count: 83861
2021/12/11 07:22:51 Waiting for src bucket item count to get to: 200000 curr count: 100001
2021/12/11 07:22:56 Waiting for src bucket item count to get to: 200000 curr count: 100001
2021/12/11 07:23:01 Waiting for src bucket item count to get to: 200000 curr count: 116339
2021/12/11 07:23:06 Waiting for src bucket item count to get to: 200000 curr count: 116339
2021/12/11 07:23:11 Waiting for src bucket item count to get to: 200000 curr count: 132795
2021/12/11 07:23:16 Waiting for src bucket item count to get to: 200000 curr count: 132795
2021/12/11 07:23:21 Waiting for src bucket item count to get to: 200000 curr count: 149413
2021/12/11 07:23:26 Waiting for src bucket item count to get to: 200000 curr count: 149413
2021/12/11 07:23:31 Waiting for src bucket item count to get to: 200000 curr count: 165997
2021/12/11 07:23:36 Waiting for src bucket item count to get to: 200000 curr count: 165997
2021/12/11 07:23:41 Waiting for src bucket item count to get to: 200000 curr count: 181091
2021/12/11 07:23:46 Waiting for src bucket item count to get to: 200000 curr count: 181091
2021/12/11 07:23:51 Waiting for src bucket item count to get to: 200000 curr count: 197377
2021/12/11 07:23:56 Waiting for src bucket item count to get to: 200000 curr count: 197377
2021/12/11 07:24:01 Waiting for src bucket item count to get to: 200000 curr count: 197377
2021/12/11 07:24:06 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/11 07:24:06 Post SET ops. Expected item count: 200000 got 200000
2021/12/11 07:24:06 Starting bucket ops to source bucket
2021/12/11 07:27:52 Waiting for dst bucket item count to get to: 0 curr count: 13801
2021/12/11 07:27:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 07:27:52 Post DELETE ops. Expected item count: 0 got 0
2021/12/11 07:27:52 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 07:27:57 Waiting for app: TestKVSwapRebWithOps to get un-deployed
2021/12/11 07:28:02 Waiting for app: TestKVSwapRebWithOps to get un-deployed
2021/12/11 07:28:07 Waiting for app: TestKVSwapRebWithOps to get un-deployed
2021/12/11 07:28:12 Waiting for app: TestKVSwapRebWithOps to get un-deployed
2021/12/11 07:28:12 App: TestKVSwapRebWithOps got un-deployed
2021/12/11 07:28:12 No eventing-con process running
2021/12/11 07:28:12 Function: TestKVSwapRebWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVSwapRebWithOps deleting in the background"
}
2021/12/11 07:28:14 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 07:28:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 07:28:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 07:28:20 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 07:28:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 07:28:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVSwapRebWithOps (450.10s)
=== RUN   TestEventingRebInWithTimers
2021/12/11 07:28:35 Waiting for app: TestEventingRebInWithTimers to get un-deployed
2021/12/11 07:28:35 App: TestEventingRebInWithTimers got un-deployed
2021/12/11 07:28:35 No eventing-con process running
2021/12/11 07:28:35 Function: TestEventingRebInWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebInWithTimers not found"
 }
}
2021/12/11 07:28:37 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 07:28:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 07:28:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 07:28:44 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 07:28:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 07:28:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 07:28:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 07:28:54 Deploying app: TestEventingRebInWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/11 07:28:55 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebInWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/11 07:29:00 Waiting for app: TestEventingRebInWithTimers to get deployed
2021/12/11 07:29:05 Waiting for app: TestEventingRebInWithTimers to get deployed
2021/12/11 07:29:10 Waiting for app: TestEventingRebInWithTimers to get deployed
2021/12/11 07:29:15 Waiting for app: TestEventingRebInWithTimers to get deployed
2021/12/11 07:29:15 App: TestEventingRebInWithTimers got deployed
2021/12/11 07:29:15 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/11 07:29:15 Starting bucket ops to source bucket
2021/12/11 07:29:16 Removing node(s): [] from the cluster
2021/12/11 07:29:21 Rebalance progress: 12.5
2021/12/11 07:29:26 Rebalance progress: 12.5
2021/12/11 07:29:31 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 07:29:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 07:29:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 07:29:31 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/11 07:29:36 Rebalance progress: 12.5
2021/12/11 07:29:41 Rebalance progress: 12.5
2021/12/11 07:29:46 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 07:29:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 07:29:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 07:29:46 Waiting for src bucket item count to get to: 50000 curr count: 30353
2021/12/11 07:29:51 Waiting for src bucket item count to get to: 50000 curr count: 41911
2021/12/11 07:29:56 Waiting for src bucket item count to get to: 50000 curr count: 41911
2021/12/11 07:30:06 Waiting for dst bucket item count to get to: 50000 curr count: 35048
2021/12/11 07:30:11 Waiting for dst bucket item count to get to: 50000 curr count: 35048
2021/12/11 07:30:16 Waiting for dst bucket item count to get to: 50000 curr count: 42144
2021/12/11 07:30:21 Waiting for dst bucket item count to get to: 50000 curr count: 42144
2021/12/11 07:30:21 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/11 07:30:21 Post SET ops. Expected item count: 50000 got 50000
2021/12/11 07:30:21 Starting bucket ops to source bucket
2021/12/11 07:31:30 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/11 07:31:35 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/11 07:31:40 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/11 07:31:45 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/11 07:31:50 Waiting for dst bucket item count to get to: 0 curr count: 40168
2021/12/11 07:31:55 Waiting for dst bucket item count to get to: 0 curr count: 40168
2021/12/11 07:32:00 Waiting for dst bucket item count to get to: 0 curr count: 27210
2021/12/11 07:32:05 Waiting for dst bucket item count to get to: 0 curr count: 27210
2021/12/11 07:32:10 Waiting for dst bucket item count to get to: 0 curr count: 8499
2021/12/11 07:32:15 Waiting for dst bucket item count to get to: 0 curr count: 8499
2021/12/11 07:32:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 07:32:15 Post DELETE ops. Expected item count: 0 got 0
2021/12/11 07:32:16 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 07:32:21 Waiting for app: TestEventingRebInWithTimers to get un-deployed
2021/12/11 07:32:26 Waiting for app: TestEventingRebInWithTimers to get un-deployed
2021/12/11 07:32:31 Waiting for app: TestEventingRebInWithTimers to get un-deployed
2021/12/11 07:32:36 Waiting for app: TestEventingRebInWithTimers to get un-deployed
2021/12/11 07:32:36 App: TestEventingRebInWithTimers got un-deployed
2021/12/11 07:32:36 No eventing-con process running
2021/12/11 07:32:36 Function: TestEventingRebInWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebInWithTimers deleting in the background"
}
2021/12/11 07:32:37 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 07:32:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 07:32:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 07:32:44 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 07:32:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 07:32:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebInWithTimers (263.73s)
=== RUN   TestEventingRebOutWithTimers
2021/12/11 07:32:54 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/11 07:32:54 Removing node(s): [] from the cluster
2021/12/11 07:32:59 Rebalance progress: 100
2021/12/11 07:33:04 Waiting for app: TestEventingRebOutWithTimers to get un-deployed
2021/12/11 07:33:04 App: TestEventingRebOutWithTimers got un-deployed
2021/12/11 07:33:04 No eventing-con process running
2021/12/11 07:33:04 Function: TestEventingRebOutWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebOutWithTimers not found"
 }
}
2021/12/11 07:33:06 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 07:33:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 07:33:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 07:33:13 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 07:33:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 07:33:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 07:33:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 07:33:23 Deploying app: TestEventingRebOutWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/11 07:33:24 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebOutWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/11 07:33:29 Waiting for app: TestEventingRebOutWithTimers to get deployed
2021/12/11 07:33:34 Waiting for app: TestEventingRebOutWithTimers to get deployed
2021/12/11 07:33:39 Waiting for app: TestEventingRebOutWithTimers to get deployed
2021/12/11 07:33:44 Waiting for app: TestEventingRebOutWithTimers to get deployed
2021/12/11 07:33:44 App: TestEventingRebOutWithTimers got deployed
2021/12/11 07:33:44 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/11 07:33:44 Starting bucket ops to source bucket
2021/12/11 07:33:49 Rebalance progress: 12.5
2021/12/11 07:33:54 Rebalance progress: 12.5
2021/12/11 07:33:59 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 07:33:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 07:33:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 07:33:59 Waiting for src bucket item count to get to: 50000 curr count: 6959
2021/12/11 07:34:04 Waiting for src bucket item count to get to: 50000 curr count: 20506
2021/12/11 07:34:09 Waiting for src bucket item count to get to: 50000 curr count: 20506
2021/12/11 07:34:14 Waiting for src bucket item count to get to: 50000 curr count: 31337
2021/12/11 07:34:19 Waiting for src bucket item count to get to: 50000 curr count: 31337
2021/12/11 07:34:24 Waiting for src bucket item count to get to: 50000 curr count: 31337
2021/12/11 07:34:29 Waiting for src bucket item count to get to: 50000 curr count: 42925
2021/12/11 07:34:34 Waiting for src bucket item count to get to: 50000 curr count: 42925
2021/12/11 07:34:44 Waiting for dst bucket item count to get to: 50000 curr count: 41095
2021/12/11 07:34:49 Waiting for dst bucket item count to get to: 50000 curr count: 41095
2021/12/11 07:34:54 Waiting for dst bucket item count to get to: 50000 curr count: 49716
2021/12/11 07:34:59 Waiting for dst bucket item count to get to: 50000 curr count: 49716
2021/12/11 07:34:59 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/11 07:34:59 Post SET ops. Expected item count: 50000 got 50000
2021/12/11 07:34:59 Starting bucket ops to source bucket
2021/12/11 07:36:08 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/11 07:36:13 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/11 07:36:18 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/11 07:36:23 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/11 07:36:28 Waiting for dst bucket item count to get to: 0 curr count: 34247
2021/12/11 07:36:33 Waiting for dst bucket item count to get to: 0 curr count: 34247
2021/12/11 07:36:38 Waiting for dst bucket item count to get to: 0 curr count: 17007
2021/12/11 07:36:43 Waiting for dst bucket item count to get to: 0 curr count: 17007
2021/12/11 07:36:48 Waiting for dst bucket item count to get to: 0 curr count: 501
2021/12/11 07:36:53 Waiting for dst bucket item count to get to: 0 curr count: 501
2021/12/11 07:36:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 07:36:53 Post DELETE ops. Expected item count: 0 got 0
2021/12/11 07:36:53 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 07:36:58 Waiting for app: TestEventingRebOutWithTimers to get un-deployed
2021/12/11 07:37:03 Waiting for app: TestEventingRebOutWithTimers to get un-deployed
2021/12/11 07:37:09 Waiting for app: TestEventingRebOutWithTimers to get un-deployed
2021/12/11 07:37:14 Waiting for app: TestEventingRebOutWithTimers to get un-deployed
2021/12/11 07:37:14 App: TestEventingRebOutWithTimers got un-deployed
2021/12/11 07:37:14 No eventing-con process running
2021/12/11 07:37:14 Function: TestEventingRebOutWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebOutWithTimers deleting in the background"
}
2021/12/11 07:37:15 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 07:37:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 07:37:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 07:37:22 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 07:37:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 07:37:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebOutWithTimers (277.87s)
=== RUN   TestEventingSwapRebWithTimers
2021/12/11 07:37:37 Waiting for app: TestEventingSwapRebWithTimers to get un-deployed
2021/12/11 07:37:37 App: TestEventingSwapRebWithTimers got un-deployed
2021/12/11 07:37:37 No eventing-con process running
2021/12/11 07:37:37 Function: TestEventingSwapRebWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingSwapRebWithTimers not found"
 }
}
2021/12/11 07:37:39 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 07:37:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 07:37:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 07:37:46 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 07:37:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 07:37:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 07:37:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 07:37:56 Deploying app: TestEventingSwapRebWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/11 07:37:57 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingSwapRebWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/11 07:38:02 Waiting for app: TestEventingSwapRebWithTimers to get deployed
2021/12/11 07:38:07 Waiting for app: TestEventingSwapRebWithTimers to get deployed
2021/12/11 07:38:12 Waiting for app: TestEventingSwapRebWithTimers to get deployed
2021/12/11 07:38:17 Waiting for app: TestEventingSwapRebWithTimers to get deployed
2021/12/11 07:38:17 App: TestEventingSwapRebWithTimers got deployed
2021/12/11 07:38:17 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/11 07:38:17 Starting bucket ops to source bucket
2021/12/11 07:38:17 Removing node(s): [] from the cluster
2021/12/11 07:38:22 Rebalance progress: 12.5
2021/12/11 07:38:27 Rebalance progress: 12.5
2021/12/11 07:38:32 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 07:38:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 07:38:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 07:38:32 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/11 07:38:33 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/11 07:38:38 Rebalance progress: 12.5
2021/12/11 07:38:43 Rebalance progress: 12.5
2021/12/11 07:38:48 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 07:38:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 07:38:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 07:38:48 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/11 07:38:53 Rebalance progress: 12.5
2021/12/11 07:38:58 Rebalance progress: 12.5
2021/12/11 07:39:03 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 07:39:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 07:39:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 07:39:03 Waiting for src bucket item count to get to: 50000 curr count: 48804
2021/12/11 07:39:08 Waiting for src bucket item count to get to: 50000 curr count: 48804
2021/12/11 07:39:18 Waiting for dst bucket item count to get to: 50000 curr count: 47213
2021/12/11 07:39:23 Waiting for dst bucket item count to get to: 50000 curr count: 47213
2021/12/11 07:39:23 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/11 07:39:23 Post SET ops. Expected item count: 50000 got 50000
2021/12/11 07:39:23 Starting bucket ops to source bucket
2021/12/11 07:40:32 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/11 07:40:37 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/11 07:40:42 Waiting for dst bucket item count to get to: 0 curr count: 48902
2021/12/11 07:40:47 Waiting for dst bucket item count to get to: 0 curr count: 48902
2021/12/11 07:40:52 Waiting for dst bucket item count to get to: 0 curr count: 40257
2021/12/11 07:40:57 Waiting for dst bucket item count to get to: 0 curr count: 40257
2021/12/11 07:41:02 Waiting for dst bucket item count to get to: 0 curr count: 25538
2021/12/11 07:41:07 Waiting for dst bucket item count to get to: 0 curr count: 25538
2021/12/11 07:41:12 Waiting for dst bucket item count to get to: 0 curr count: 4134
2021/12/11 07:41:17 Waiting for dst bucket item count to get to: 0 curr count: 4134
2021/12/11 07:41:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 07:41:17 Post DELETE ops. Expected item count: 0 got 0
2021/12/11 07:41:17 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 07:41:22 Waiting for app: TestEventingSwapRebWithTimers to get un-deployed
2021/12/11 07:41:27 Waiting for app: TestEventingSwapRebWithTimers to get un-deployed
2021/12/11 07:41:32 Waiting for app: TestEventingSwapRebWithTimers to get un-deployed
2021/12/11 07:41:37 Waiting for app: TestEventingSwapRebWithTimers to get un-deployed
2021/12/11 07:41:37 App: TestEventingSwapRebWithTimers got un-deployed
2021/12/11 07:41:37 No eventing-con process running
2021/12/11 07:41:37 Function: TestEventingSwapRebWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingSwapRebWithTimers deleting in the background"
}
2021/12/11 07:41:39 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 07:41:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 07:41:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 07:41:46 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 07:41:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 07:41:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingSwapRebWithTimers (263.78s)
=== RUN   TestKVRebInWithTimers
2021/12/11 07:42:01 Waiting for app: TestKVRebInWithTimers to get un-deployed
2021/12/11 07:42:01 App: TestKVRebInWithTimers got un-deployed
2021/12/11 07:42:01 No eventing-con process running
2021/12/11 07:42:01 Function: TestKVRebInWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVRebInWithTimers not found"
 }
}
2021/12/11 07:42:03 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 07:42:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 07:42:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 07:42:09 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 07:42:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 07:42:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 07:42:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 07:42:19 Deploying app: TestKVRebInWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/11 07:42:20 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVRebInWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/11 07:42:25 Waiting for app: TestKVRebInWithTimers to get deployed
2021/12/11 07:42:30 Waiting for app: TestKVRebInWithTimers to get deployed
2021/12/11 07:42:35 Waiting for app: TestKVRebInWithTimers to get deployed
2021/12/11 07:42:40 Waiting for app: TestKVRebInWithTimers to get deployed
2021/12/11 07:42:40 App: TestKVRebInWithTimers got deployed
2021/12/11 07:42:40 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/11 07:42:40 Starting bucket ops to source bucket
2021/12/11 07:42:41 Removing node(s): [] from the cluster
2021/12/11 07:42:46 Rebalance progress: 12.5
2021/12/11 07:42:51 Rebalance progress: 12.5
2021/12/11 07:42:56 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 07:42:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 07:42:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 07:42:56 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/11 07:43:01 Rebalance progress: 12.5
2021/12/11 07:43:06 Rebalance progress: 12.5
2021/12/11 07:43:11 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 07:43:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 07:43:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 07:43:12 Waiting for src bucket item count to get to: 50000 curr count: 36288
2021/12/11 07:43:17 Waiting for src bucket item count to get to: 50000 curr count: 36288
2021/12/11 07:43:22 Waiting for src bucket item count to get to: 50000 curr count: 47009
2021/12/11 07:43:27 Waiting for src bucket item count to get to: 50000 curr count: 47009
2021/12/11 07:43:37 Waiting for dst bucket item count to get to: 50000 curr count: 37395
2021/12/11 07:43:42 Waiting for dst bucket item count to get to: 50000 curr count: 37395
2021/12/11 07:43:42 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/11 07:43:42 Post SET ops. Expected item count: 50000 got 50000
2021/12/11 07:43:42 Starting bucket ops to source bucket
2021/12/11 07:44:51 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/11 07:44:56 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/11 07:45:01 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/11 07:45:06 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/11 07:45:11 Waiting for dst bucket item count to get to: 0 curr count: 43886
2021/12/11 07:45:16 Waiting for dst bucket item count to get to: 0 curr count: 43886
2021/12/11 07:45:21 Waiting for dst bucket item count to get to: 0 curr count: 24088
2021/12/11 07:45:26 Waiting for dst bucket item count to get to: 0 curr count: 24088
2021/12/11 07:45:31 Waiting for dst bucket item count to get to: 0 curr count: 12975
2021/12/11 07:45:36 Waiting for dst bucket item count to get to: 0 curr count: 12975
2021/12/11 07:45:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 07:45:36 Post DELETE ops. Expected item count: 0 got 0
2021/12/11 07:45:36 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 07:45:41 Waiting for app: TestKVRebInWithTimers to get un-deployed
2021/12/11 07:45:46 Waiting for app: TestKVRebInWithTimers to get un-deployed
2021/12/11 07:45:51 Waiting for app: TestKVRebInWithTimers to get un-deployed
2021/12/11 07:45:56 Waiting for app: TestKVRebInWithTimers to get un-deployed
2021/12/11 07:45:56 App: TestKVRebInWithTimers got un-deployed
2021/12/11 07:45:56 No eventing-con process running
2021/12/11 07:45:56 Function: TestKVRebInWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVRebInWithTimers deleting in the background"
}
2021/12/11 07:45:58 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 07:46:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 07:46:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 07:46:05 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 07:46:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 07:46:10 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebInWithTimers (259.05s)
=== RUN   TestKVRebOutWithTimers
2021/12/11 07:46:15 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/11 07:46:15 Removing node(s): [] from the cluster
2021/12/11 07:46:20 Rebalance progress: 100
2021/12/11 07:46:25 Waiting for app: TestKVRebOutWithTimers to get un-deployed
2021/12/11 07:46:25 App: TestKVRebOutWithTimers got un-deployed
2021/12/11 07:46:25 No eventing-con process running
2021/12/11 07:46:25 Function: TestKVRebOutWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVRebOutWithTimers not found"
 }
}
2021/12/11 07:46:27 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 07:46:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 07:46:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 07:46:34 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 07:46:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 07:46:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 07:46:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 07:46:44 Deploying app: TestKVRebOutWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/11 07:46:45 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVRebOutWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/11 07:46:50 Waiting for app: TestKVRebOutWithTimers to get deployed
2021/12/11 07:46:55 Waiting for app: TestKVRebOutWithTimers to get deployed
2021/12/11 07:47:00 Waiting for app: TestKVRebOutWithTimers to get deployed
2021/12/11 07:47:05 Waiting for app: TestKVRebOutWithTimers to get deployed
2021/12/11 07:47:05 App: TestKVRebOutWithTimers got deployed
2021/12/11 07:47:05 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/11 07:47:05 Starting bucket ops to source bucket
2021/12/11 07:47:10 Rebalance progress: 12.5
2021/12/11 07:47:15 Rebalance progress: 12.5
2021/12/11 07:47:20 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 07:47:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 07:47:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 07:47:20 Waiting for src bucket item count to get to: 50000 curr count: 5675
2021/12/11 07:47:25 Waiting for src bucket item count to get to: 50000 curr count: 19809
2021/12/11 07:47:30 Waiting for src bucket item count to get to: 50000 curr count: 31295
2021/12/11 07:47:35 Waiting for src bucket item count to get to: 50000 curr count: 31295
2021/12/11 07:47:40 Waiting for src bucket item count to get to: 50000 curr count: 43082
2021/12/11 07:47:45 Waiting for src bucket item count to get to: 50000 curr count: 43082
2021/12/11 07:47:55 Waiting for dst bucket item count to get to: 50000 curr count: 37066
2021/12/11 07:48:00 Waiting for dst bucket item count to get to: 50000 curr count: 37066
2021/12/11 07:48:05 Waiting for dst bucket item count to get to: 50000 curr count: 37066
2021/12/11 07:48:10 Waiting for dst bucket item count to get to: 50000 curr count: 46149
2021/12/11 07:48:15 Waiting for dst bucket item count to get to: 50000 curr count: 46149
2021/12/11 07:48:15 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/11 07:48:15 Post SET ops. Expected item count: 50000 got 50000
2021/12/11 07:48:15 Starting bucket ops to source bucket
2021/12/11 07:49:24 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/11 07:49:29 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/11 07:49:34 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/11 07:49:39 Waiting for dst bucket item count to get to: 0 curr count: 47072
2021/12/11 07:49:44 Waiting for dst bucket item count to get to: 0 curr count: 47072
2021/12/11 07:49:49 Waiting for dst bucket item count to get to: 0 curr count: 31943
2021/12/11 07:49:54 Waiting for dst bucket item count to get to: 0 curr count: 31943
2021/12/11 07:49:59 Waiting for dst bucket item count to get to: 0 curr count: 13404
2021/12/11 07:50:04 Waiting for dst bucket item count to get to: 0 curr count: 13404
2021/12/11 07:50:09 Waiting for dst bucket item count to get to: 0 curr count: 511
2021/12/11 07:50:14 Waiting for dst bucket item count to get to: 0 curr count: 511
2021/12/11 07:50:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 07:50:14 Post DELETE ops. Expected item count: 0 got 0
2021/12/11 07:50:14 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 07:50:19 Waiting for app: TestKVRebOutWithTimers to get un-deployed
2021/12/11 07:50:25 Waiting for app: TestKVRebOutWithTimers to get un-deployed
2021/12/11 07:50:30 Waiting for app: TestKVRebOutWithTimers to get un-deployed
2021/12/11 07:50:35 Waiting for app: TestKVRebOutWithTimers to get un-deployed
2021/12/11 07:50:35 App: TestKVRebOutWithTimers got un-deployed
2021/12/11 07:50:35 No eventing-con process running
2021/12/11 07:50:35 Function: TestKVRebOutWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVRebOutWithTimers deleting in the background"
}
2021/12/11 07:50:36 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 07:50:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 07:50:41 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 07:50:43 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 07:50:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 07:50:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebOutWithTimers (278.30s)
=== RUN   TestKVSwapRebWithTimers
2021/12/11 07:50:58 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/11 07:50:58 App: TestKVSwapRebWithTimers got un-deployed
2021/12/11 07:50:58 No eventing-con process running
2021/12/11 07:50:58 Function: TestKVSwapRebWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVSwapRebWithTimers not found"
 }
}
2021/12/11 07:51:00 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 07:51:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 07:51:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 07:51:07 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 07:51:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 07:51:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 07:51:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 07:51:17 Deploying app: TestKVSwapRebWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/11 07:51:18 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVSwapRebWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/11 07:51:23 Waiting for app: TestKVSwapRebWithTimers to get deployed
2021/12/11 07:51:28 Waiting for app: TestKVSwapRebWithTimers to get deployed
2021/12/11 07:51:33 Waiting for app: TestKVSwapRebWithTimers to get deployed
2021/12/11 07:51:38 Waiting for app: TestKVSwapRebWithTimers to get deployed
2021/12/11 07:51:38 App: TestKVSwapRebWithTimers got deployed
2021/12/11 07:51:38 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/11 07:51:38 Starting bucket ops to source bucket
2021/12/11 07:51:38 Removing node(s): [] from the cluster
2021/12/11 07:51:43 Rebalance progress: 12.5
2021/12/11 07:51:48 Rebalance progress: 12.5
2021/12/11 07:51:53 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 07:51:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 07:51:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 07:51:53 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/11 07:51:54 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/11 07:51:59 Rebalance progress: 12.5
2021/12/11 07:52:04 Rebalance progress: 12.5
2021/12/11 07:52:09 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 07:52:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 07:52:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 07:52:10 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/11 07:52:15 Rebalance progress: 12.5
2021/12/11 07:52:20 Rebalance progress: 12.5
2021/12/11 07:52:25 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 07:52:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 07:52:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 07:52:30 Waiting for dst bucket item count to get to: 50000 curr count: 27889
2021/12/11 07:52:35 Waiting for dst bucket item count to get to: 50000 curr count: 27889
2021/12/11 07:52:40 Waiting for dst bucket item count to get to: 50000 curr count: 44391
2021/12/11 07:52:45 Waiting for dst bucket item count to get to: 50000 curr count: 44391
2021/12/11 07:52:45 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/11 07:52:45 Post SET ops. Expected item count: 50000 got 50000
2021/12/11 07:52:45 Starting bucket ops to source bucket
2021/12/11 07:53:54 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/11 07:53:59 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/11 07:54:04 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/11 07:54:09 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/11 07:54:14 Waiting for dst bucket item count to get to: 0 curr count: 41897
2021/12/11 07:54:19 Waiting for dst bucket item count to get to: 0 curr count: 32343
2021/12/11 07:54:24 Waiting for dst bucket item count to get to: 0 curr count: 32343
2021/12/11 07:54:29 Waiting for dst bucket item count to get to: 0 curr count: 15564
2021/12/11 07:54:34 Waiting for dst bucket item count to get to: 0 curr count: 15564
2021/12/11 07:54:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 07:54:34 Post DELETE ops. Expected item count: 0 got 0
2021/12/11 07:54:34 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 07:54:39 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/11 07:54:44 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/11 07:54:49 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/11 07:54:55 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/11 07:55:00 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/11 07:55:05 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/11 07:55:05 App: TestKVSwapRebWithTimers got un-deployed
2021/12/11 07:55:05 No eventing-con process running
2021/12/11 07:55:05 Function: TestKVSwapRebWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVSwapRebWithTimers deleting in the background"
}
2021/12/11 07:55:06 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 07:55:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 07:55:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 07:55:13 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 07:55:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 07:55:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVSwapRebWithTimers (269.93s)
=== RUN   TestDuplicateTimerCreationEventingRebIn
2021/12/11 07:55:28 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/11 07:55:28 App: TestDuplicateTimerCreationEventingRebIn got un-deployed
2021/12/11 07:55:28 No eventing-con process running
2021/12/11 07:55:28 Function: TestDuplicateTimerCreationEventingRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationEventingRebIn not found"
 }
}
2021/12/11 07:55:30 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 07:55:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 07:55:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 07:55:37 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 07:55:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 07:55:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 07:55:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 07:55:47 Deploying app: TestDuplicateTimerCreationEventingRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/11 07:55:48 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationEventingRebIn' in metakv",
  "warnings": null
 }
}
2021/12/11 07:55:53 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get deployed
2021/12/11 07:55:58 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get deployed
2021/12/11 07:56:03 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get deployed
2021/12/11 07:56:08 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get deployed
2021/12/11 07:56:08 App: TestDuplicateTimerCreationEventingRebIn got deployed
2021/12/11 07:56:08 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/11 07:56:08 Starting bucket ops to source bucket
2021/12/11 07:56:08 Removing node(s): [] from the cluster
2021/12/11 07:56:13 Rebalance progress: 12.5
2021/12/11 07:56:18 Rebalance progress: 12.5
2021/12/11 07:56:23 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 07:56:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 07:56:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 07:56:23 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/11 07:56:28 Rebalance progress: 12.5
2021/12/11 07:56:33 Rebalance progress: 12.5
2021/12/11 07:56:38 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 07:56:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 07:56:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 07:56:38 Waiting for src bucket item count to get to: 50000 curr count: 27230
2021/12/11 07:56:43 Waiting for src bucket item count to get to: 50000 curr count: 37688
2021/12/11 07:56:48 Waiting for src bucket item count to get to: 50000 curr count: 37688
2021/12/11 07:56:53 Waiting for src bucket item count to get to: 50000 curr count: 49229
2021/12/11 07:56:58 Waiting for src bucket item count to get to: 50000 curr count: 49229
2021/12/11 07:57:08 Waiting for dst bucket item count to get to: 50000 curr count: 39421
2021/12/11 07:57:13 Waiting for dst bucket item count to get to: 50000 curr count: 39421
2021/12/11 07:57:18 Waiting for dst bucket item count to get to: 50000 curr count: 46773
2021/12/11 07:57:23 Waiting for dst bucket item count to get to: 50000 curr count: 46773
2021/12/11 07:57:23 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/11 07:57:23 Post SET ops. Expected item count: 50000 got 50000
2021/12/11 07:57:24 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 07:57:29 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/11 07:57:34 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/11 07:57:39 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/11 07:57:44 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/11 07:57:49 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/11 07:57:49 App: TestDuplicateTimerCreationEventingRebIn got un-deployed
2021/12/11 07:57:49 No eventing-con process running
2021/12/11 07:57:49 Function: TestDuplicateTimerCreationEventingRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationEventingRebIn deleting in the background"
}
2021/12/11 07:57:50 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 07:57:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 07:58:01 Waiting for dst bucket item count to get to: 0 curr count: 29012
2021/12/11 07:58:06 Waiting for dst bucket item count to get to: 0 curr count: 29012
2021/12/11 07:58:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 07:58:07 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 07:58:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 07:58:17 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/11 07:58:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationEventingRebIn (179.18s)
=== RUN   TestDuplicateTimerCreationEventingRebOut
2021/12/11 07:58:22 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/11 07:58:22 Removing node(s): [] from the cluster
2021/12/11 07:58:27 Rebalance progress: 100
2021/12/11 07:58:32 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/11 07:58:32 App: TestDuplicateTimerCreationEventingRebOut got un-deployed
2021/12/11 07:58:32 No eventing-con process running
2021/12/11 07:58:32 Function: TestDuplicateTimerCreationEventingRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationEventingRebOut not found"
 }
}
2021/12/11 07:58:34 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 07:58:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 07:58:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 07:58:41 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 07:58:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 07:58:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 07:58:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 07:58:51 Deploying app: TestDuplicateTimerCreationEventingRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/11 07:58:52 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationEventingRebOut' in metakv",
  "warnings": null
 }
}
2021/12/11 07:58:57 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get deployed
2021/12/11 07:59:02 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get deployed
2021/12/11 07:59:07 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get deployed
2021/12/11 07:59:12 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get deployed
2021/12/11 07:59:12 App: TestDuplicateTimerCreationEventingRebOut got deployed
2021/12/11 07:59:12 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/11 07:59:12 Starting bucket ops to source bucket
2021/12/11 07:59:17 Rebalance progress: 12.5
2021/12/11 07:59:22 Rebalance progress: 12.5
2021/12/11 07:59:27 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 07:59:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 07:59:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 07:59:27 Waiting for src bucket item count to get to: 50000 curr count: 9159
2021/12/11 07:59:32 Waiting for src bucket item count to get to: 50000 curr count: 23273
2021/12/11 07:59:37 Waiting for src bucket item count to get to: 50000 curr count: 23273
2021/12/11 07:59:42 Waiting for src bucket item count to get to: 50000 curr count: 36616
2021/12/11 07:59:47 Waiting for src bucket item count to get to: 50000 curr count: 36616
2021/12/11 07:59:52 Waiting for src bucket item count to get to: 50000 curr count: 36616
2021/12/11 07:59:57 Waiting for src bucket item count to get to: 50000 curr count: 45519
2021/12/11 08:00:02 Waiting for src bucket item count to get to: 50000 curr count: 45519
2021/12/11 08:00:12 Waiting for dst bucket item count to get to: 50000 curr count: 31292
2021/12/11 08:00:17 Waiting for dst bucket item count to get to: 50000 curr count: 31292
2021/12/11 08:00:22 Waiting for dst bucket item count to get to: 50000 curr count: 43942
2021/12/11 08:00:27 Waiting for dst bucket item count to get to: 50000 curr count: 43942
2021/12/11 08:00:27 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/11 08:00:27 Post SET ops. Expected item count: 50000 got 50000
2021/12/11 08:00:27 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 08:00:32 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/11 08:00:37 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/11 08:00:43 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/11 08:00:48 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/11 08:00:53 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/11 08:00:58 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/11 08:00:58 App: TestDuplicateTimerCreationEventingRebOut got un-deployed
2021/12/11 08:00:58 No eventing-con process running
2021/12/11 08:00:58 Function: TestDuplicateTimerCreationEventingRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationEventingRebOut deleting in the background"
}
2021/12/11 08:00:59 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:01:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 08:01:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 08:01:06 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:01:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 08:01:16 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/11 08:01:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationEventingRebOut (178.94s)
=== RUN   TestDuplicateTimerCreationEventingSwapReb
2021/12/11 08:01:26 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/11 08:01:26 App: TestDuplicateTimerCreationEventingSwapReb got un-deployed
2021/12/11 08:01:26 No eventing-con process running
2021/12/11 08:01:26 Function: TestDuplicateTimerCreationEventingSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationEventingSwapReb not found"
 }
}
2021/12/11 08:01:28 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:01:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 08:01:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 08:01:35 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:01:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 08:01:40 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:01:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 08:01:45 Deploying app: TestDuplicateTimerCreationEventingSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/11 08:01:46 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationEventingSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/11 08:01:51 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get deployed
2021/12/11 08:01:56 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get deployed
2021/12/11 08:02:01 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get deployed
2021/12/11 08:02:06 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get deployed
2021/12/11 08:02:06 App: TestDuplicateTimerCreationEventingSwapReb got deployed
2021/12/11 08:02:06 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/11 08:02:06 Starting bucket ops to source bucket
2021/12/11 08:02:06 Removing node(s): [] from the cluster
2021/12/11 08:02:11 Rebalance progress: 12.5
2021/12/11 08:02:16 Rebalance progress: 12.5
2021/12/11 08:02:21 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:02:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:02:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 08:02:21 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/11 08:02:23 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/11 08:02:28 Rebalance progress: 12.5
2021/12/11 08:02:33 Rebalance progress: 12.5
2021/12/11 08:02:38 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:02:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:02:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 08:02:38 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/11 08:02:43 Rebalance progress: 12.5
2021/12/11 08:02:48 Rebalance progress: 12.5
2021/12/11 08:02:53 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:02:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:02:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 08:02:53 Waiting for src bucket item count to get to: 50000 curr count: 49751
2021/12/11 08:02:58 Waiting for src bucket item count to get to: 50000 curr count: 49751
2021/12/11 08:03:08 Waiting for dst bucket item count to get to: 50000 curr count: 38277
2021/12/11 08:03:13 Waiting for dst bucket item count to get to: 50000 curr count: 38277
2021/12/11 08:03:18 Waiting for dst bucket item count to get to: 50000 curr count: 49176
2021/12/11 08:03:23 Waiting for dst bucket item count to get to: 50000 curr count: 49176
2021/12/11 08:03:23 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/11 08:03:23 Post SET ops. Expected item count: 50000 got 50000
2021/12/11 08:03:23 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 08:03:28 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/11 08:03:33 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/11 08:03:38 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/11 08:03:43 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/11 08:03:48 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/11 08:03:53 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/11 08:03:53 App: TestDuplicateTimerCreationEventingSwapReb got un-deployed
2021/12/11 08:03:53 No eventing-con process running
2021/12/11 08:03:53 Function: TestDuplicateTimerCreationEventingSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationEventingSwapReb deleting in the background"
}
2021/12/11 08:03:55 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:04:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 08:04:05 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/11 08:04:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 08:04:07 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:04:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 08:04:17 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/11 08:04:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationEventingSwapReb (180.50s)
=== RUN   TestDuplicateTimerCreationKVRebIn
2021/12/11 08:04:27 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/11 08:04:27 App: TestDuplicateTimerCreationKVRebIn got un-deployed
2021/12/11 08:04:27 No eventing-con process running
2021/12/11 08:04:27 Function: TestDuplicateTimerCreationKVRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationKVRebIn not found"
 }
}
2021/12/11 08:04:28 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:04:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 08:04:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 08:04:35 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:04:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 08:04:40 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:04:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 08:04:45 Deploying app: TestDuplicateTimerCreationKVRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/11 08:04:46 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationKVRebIn' in metakv",
  "warnings": null
 }
}
2021/12/11 08:04:51 Waiting for app: TestDuplicateTimerCreationKVRebIn to get deployed
2021/12/11 08:04:56 Waiting for app: TestDuplicateTimerCreationKVRebIn to get deployed
2021/12/11 08:05:01 Waiting for app: TestDuplicateTimerCreationKVRebIn to get deployed
2021/12/11 08:05:06 Waiting for app: TestDuplicateTimerCreationKVRebIn to get deployed
2021/12/11 08:05:06 App: TestDuplicateTimerCreationKVRebIn got deployed
2021/12/11 08:05:06 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/11 08:05:06 Starting bucket ops to source bucket
2021/12/11 08:05:07 Removing node(s): [] from the cluster
2021/12/11 08:05:12 Rebalance progress: 12.5
2021/12/11 08:05:17 Rebalance progress: 12.5
2021/12/11 08:05:22 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:05:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:05:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 08:05:22 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/11 08:05:27 Rebalance progress: 12.5
2021/12/11 08:05:32 Rebalance progress: 12.5
2021/12/11 08:05:37 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:05:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:05:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 08:05:37 Waiting for src bucket item count to get to: 50000 curr count: 28681
2021/12/11 08:05:42 Waiting for src bucket item count to get to: 50000 curr count: 39787
2021/12/11 08:05:47 Waiting for src bucket item count to get to: 50000 curr count: 39787
2021/12/11 08:05:57 Waiting for dst bucket item count to get to: 50000 curr count: 30246
2021/12/11 08:06:02 Waiting for dst bucket item count to get to: 50000 curr count: 30246
2021/12/11 08:06:07 Waiting for dst bucket item count to get to: 50000 curr count: 40060
2021/12/11 08:06:12 Waiting for dst bucket item count to get to: 50000 curr count: 40060
2021/12/11 08:06:17 Waiting for dst bucket item count to get to: 50000 curr count: 49688
2021/12/11 08:06:22 Waiting for dst bucket item count to get to: 50000 curr count: 49688
2021/12/11 08:06:22 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/11 08:06:22 Post SET ops. Expected item count: 50000 got 50000
2021/12/11 08:06:22 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 08:06:27 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/11 08:06:32 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/11 08:06:37 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/11 08:06:42 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/11 08:06:47 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/11 08:06:52 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/11 08:06:52 App: TestDuplicateTimerCreationKVRebIn got un-deployed
2021/12/11 08:06:52 No eventing-con process running
2021/12/11 08:06:52 Function: TestDuplicateTimerCreationKVRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationKVRebIn deleting in the background"
}
2021/12/11 08:06:54 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:06:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 08:07:04 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/11 08:07:09 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/11 08:07:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 08:07:11 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:07:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 08:07:21 Waiting for dst bucket item count to get to: 0 curr count: 45246
2021/12/11 08:07:26 Waiting for dst bucket item count to get to: 0 curr count: 45246
2021/12/11 08:07:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationKVRebIn (189.17s)
=== RUN   TestDuplicateTimerCreationKVRebOut
2021/12/11 08:07:31 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/11 08:07:31 Removing node(s): [] from the cluster
2021/12/11 08:07:36 Rebalance progress: 100
2021/12/11 08:07:41 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/11 08:07:41 App: TestDuplicateTimerCreationKVRebOut got un-deployed
2021/12/11 08:07:41 No eventing-con process running
2021/12/11 08:07:41 Function: TestDuplicateTimerCreationKVRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationKVRebOut not found"
 }
}
2021/12/11 08:07:43 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:07:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 08:07:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 08:07:50 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:07:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 08:07:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:08:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 08:08:00 Deploying app: TestDuplicateTimerCreationKVRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/11 08:08:01 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationKVRebOut' in metakv",
  "warnings": null
 }
}
2021/12/11 08:08:06 Waiting for app: TestDuplicateTimerCreationKVRebOut to get deployed
2021/12/11 08:08:11 Waiting for app: TestDuplicateTimerCreationKVRebOut to get deployed
2021/12/11 08:08:16 Waiting for app: TestDuplicateTimerCreationKVRebOut to get deployed
2021/12/11 08:08:21 Waiting for app: TestDuplicateTimerCreationKVRebOut to get deployed
2021/12/11 08:08:21 App: TestDuplicateTimerCreationKVRebOut got deployed
2021/12/11 08:08:21 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/11 08:08:21 Starting bucket ops to source bucket
2021/12/11 08:08:26 Rebalance progress: 12.5
2021/12/11 08:08:31 Rebalance progress: 12.5
2021/12/11 08:08:36 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:08:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:08:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 08:08:36 Waiting for src bucket item count to get to: 50000 curr count: 11183
2021/12/11 08:08:41 Waiting for src bucket item count to get to: 50000 curr count: 25373
2021/12/11 08:08:46 Waiting for src bucket item count to get to: 50000 curr count: 25373
2021/12/11 08:08:51 Waiting for src bucket item count to get to: 50000 curr count: 38746
2021/12/11 08:08:56 Waiting for src bucket item count to get to: 50000 curr count: 38746
2021/12/11 08:09:01 Waiting for src bucket item count to get to: 50000 curr count: 48492
2021/12/11 08:09:06 Waiting for src bucket item count to get to: 50000 curr count: 48492
2021/12/11 08:09:11 Waiting for src bucket item count to get to: 50000 curr count: 48492
2021/12/11 08:09:21 Waiting for dst bucket item count to get to: 50000 curr count: 29143
2021/12/11 08:09:26 Waiting for dst bucket item count to get to: 50000 curr count: 29143
2021/12/11 08:09:31 Waiting for dst bucket item count to get to: 50000 curr count: 45695
2021/12/11 08:09:36 Waiting for dst bucket item count to get to: 50000 curr count: 45695
2021/12/11 08:09:36 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/11 08:09:36 Post SET ops. Expected item count: 50000 got 50000
2021/12/11 08:09:36 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 08:09:41 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/11 08:09:46 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/11 08:09:51 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/11 08:09:56 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/11 08:10:01 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/11 08:10:06 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/11 08:10:06 App: TestDuplicateTimerCreationKVRebOut got un-deployed
2021/12/11 08:10:06 No eventing-con process running
2021/12/11 08:10:06 Function: TestDuplicateTimerCreationKVRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationKVRebOut deleting in the background"
}
2021/12/11 08:10:08 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:10:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 08:10:18 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/11 08:10:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 08:10:20 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:10:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 08:10:30 Waiting for dst bucket item count to get to: 0 curr count: 2697
2021/12/11 08:10:35 Waiting for dst bucket item count to get to: 0 curr count: 2697
2021/12/11 08:10:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationKVRebOut (189.09s)
=== RUN   TestDuplicateTimerCreationKVSwapReb
2021/12/11 08:10:45 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/11 08:10:45 App: TestDuplicateTimerCreationKVSwapReb got un-deployed
2021/12/11 08:10:45 No eventing-con process running
2021/12/11 08:10:45 Function: TestDuplicateTimerCreationKVSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationKVSwapReb not found"
 }
}
2021/12/11 08:10:47 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:10:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 08:10:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 08:10:53 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:10:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 08:10:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:11:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 08:11:03 Deploying app: TestDuplicateTimerCreationKVSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/11 08:11:05 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationKVSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/11 08:11:10 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get deployed
2021/12/11 08:11:15 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get deployed
2021/12/11 08:11:20 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get deployed
2021/12/11 08:11:25 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get deployed
2021/12/11 08:11:25 App: TestDuplicateTimerCreationKVSwapReb got deployed
2021/12/11 08:11:25 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/11 08:11:25 Starting bucket ops to source bucket
2021/12/11 08:11:25 Removing node(s): [] from the cluster
2021/12/11 08:11:30 Rebalance progress: 12.5
2021/12/11 08:11:35 Rebalance progress: 12.5
2021/12/11 08:11:40 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:11:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:11:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 08:11:40 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/11 08:11:41 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/11 08:11:46 Rebalance progress: 12.5
2021/12/11 08:11:51 Rebalance progress: 12.5
2021/12/11 08:11:56 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:11:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:11:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 08:11:56 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/11 08:12:01 Rebalance progress: 12.5
2021/12/11 08:12:06 Rebalance progress: 12.5
2021/12/11 08:12:11 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:12:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:12:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 08:12:16 Waiting for dst bucket item count to get to: 50000 curr count: 32649
2021/12/11 08:12:21 Waiting for dst bucket item count to get to: 50000 curr count: 32649
2021/12/11 08:12:26 Waiting for dst bucket item count to get to: 50000 curr count: 38795
2021/12/11 08:12:31 Waiting for dst bucket item count to get to: 50000 curr count: 38795
2021/12/11 08:12:31 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/11 08:12:31 Post SET ops. Expected item count: 50000 got 50000
2021/12/11 08:12:31 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 08:12:36 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/11 08:12:41 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/11 08:12:46 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/11 08:12:51 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/11 08:12:56 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/11 08:13:01 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/11 08:13:01 App: TestDuplicateTimerCreationKVSwapReb got un-deployed
2021/12/11 08:13:01 No eventing-con process running
2021/12/11 08:13:01 Function: TestDuplicateTimerCreationKVSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationKVSwapReb deleting in the background"
}
2021/12/11 08:13:03 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:13:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 08:13:13 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/11 08:13:18 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/11 08:13:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 08:13:20 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:13:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 08:13:30 Waiting for dst bucket item count to get to: 0 curr count: 2879
2021/12/11 08:13:35 Waiting for dst bucket item count to get to: 0 curr count: 2879
2021/12/11 08:13:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationKVSwapReb (179.99s)
=== RUN   TestDuplicateTimerFiringEventingRebIn
2021/12/11 08:13:45 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/11 08:13:45 App: TestDuplicateTimerFiringEventingRebIn got un-deployed
2021/12/11 08:13:45 No eventing-con process running
2021/12/11 08:13:45 Function: TestDuplicateTimerFiringEventingRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringEventingRebIn not found"
 }
}
2021/12/11 08:13:47 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:13:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 08:13:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 08:13:53 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:13:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 08:13:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:14:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 08:14:03 Deploying app: TestDuplicateTimerFiringEventingRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/11 08:14:05 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringEventingRebIn' in metakv",
  "warnings": null
 }
}
2021/12/11 08:14:10 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get deployed
2021/12/11 08:14:15 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get deployed
2021/12/11 08:14:20 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get deployed
2021/12/11 08:14:25 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get deployed
2021/12/11 08:14:25 App: TestDuplicateTimerFiringEventingRebIn got deployed
2021/12/11 08:14:25 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/11 08:14:25 Starting bucket ops to source bucket
2021/12/11 08:14:25 Removing node(s): [] from the cluster
2021/12/11 08:14:30 Rebalance progress: 12.5
2021/12/11 08:14:35 Rebalance progress: 12.5
2021/12/11 08:14:40 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:14:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:14:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 08:14:40 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/11 08:14:45 Rebalance progress: 12.5
2021/12/11 08:14:50 Rebalance progress: 12.5
2021/12/11 08:14:55 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:14:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:14:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 08:14:55 Waiting for src bucket item count to get to: 50000 curr count: 31050
2021/12/11 08:15:00 Waiting for src bucket item count to get to: 50000 curr count: 31050
2021/12/11 08:15:05 Waiting for src bucket item count to get to: 50000 curr count: 42253
2021/12/11 08:15:15 Waiting for dst bucket item count to get to: 50000 curr count: 34886
2021/12/11 08:15:20 Waiting for dst bucket item count to get to: 50000 curr count: 34886
2021/12/11 08:15:25 Waiting for dst bucket item count to get to: 50000 curr count: 41970
2021/12/11 08:15:30 Waiting for dst bucket item count to get to: 50000 curr count: 41970
2021/12/11 08:15:30 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/11 08:15:30 Post SET ops. Expected item count: 50000 got 50000
2021/12/11 08:15:30 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 08:15:35 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/11 08:15:40 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/11 08:15:45 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/11 08:15:50 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/11 08:15:55 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/11 08:16:00 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/11 08:16:00 App: TestDuplicateTimerFiringEventingRebIn got un-deployed
2021/12/11 08:16:00 No eventing-con process running
2021/12/11 08:16:00 Function: TestDuplicateTimerFiringEventingRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringEventingRebIn deleting in the background"
}
2021/12/11 08:16:02 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:16:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 08:16:12 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/11 08:16:17 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/11 08:16:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 08:16:19 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:16:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 08:16:29 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/11 08:16:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringEventingRebIn (174.07s)
=== RUN   TestDuplicateTimerFiringEventingRebOut
2021/12/11 08:16:34 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/11 08:16:34 Removing node(s): [] from the cluster
2021/12/11 08:16:39 Rebalance progress: 100
2021/12/11 08:16:44 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/11 08:16:44 App: TestDuplicateTimerFiringEventingRebOut got un-deployed
2021/12/11 08:16:44 No eventing-con process running
2021/12/11 08:16:44 Function: TestDuplicateTimerFiringEventingRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringEventingRebOut not found"
 }
}
2021/12/11 08:16:46 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:16:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 08:16:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 08:16:53 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:16:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 08:16:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:17:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 08:17:03 Deploying app: TestDuplicateTimerFiringEventingRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/11 08:17:04 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringEventingRebOut' in metakv",
  "warnings": null
 }
}
2021/12/11 08:17:09 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get deployed
2021/12/11 08:17:14 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get deployed
2021/12/11 08:17:19 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get deployed
2021/12/11 08:17:24 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get deployed
2021/12/11 08:17:24 App: TestDuplicateTimerFiringEventingRebOut got deployed
2021/12/11 08:17:24 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/11 08:17:24 Starting bucket ops to source bucket
2021/12/11 08:17:29 Rebalance progress: 12.5
2021/12/11 08:17:34 Rebalance progress: 12.5
2021/12/11 08:17:39 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:17:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:17:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 08:17:39 Waiting for src bucket item count to get to: 50000 curr count: 6800
2021/12/11 08:17:44 Waiting for src bucket item count to get to: 50000 curr count: 20150
2021/12/11 08:17:49 Waiting for src bucket item count to get to: 50000 curr count: 20150
2021/12/11 08:17:54 Waiting for src bucket item count to get to: 50000 curr count: 31659
2021/12/11 08:17:59 Waiting for src bucket item count to get to: 50000 curr count: 31659
2021/12/11 08:18:04 Waiting for src bucket item count to get to: 50000 curr count: 42985
2021/12/11 08:18:09 Waiting for src bucket item count to get to: 50000 curr count: 42985
2021/12/11 08:18:14 Waiting for src bucket item count to get to: 50000 curr count: 42985
2021/12/11 08:18:24 Waiting for dst bucket item count to get to: 50000 curr count: 38122
2021/12/11 08:18:29 Waiting for dst bucket item count to get to: 50000 curr count: 38122
2021/12/11 08:18:34 Waiting for dst bucket item count to get to: 50000 curr count: 46813
2021/12/11 08:18:39 Waiting for dst bucket item count to get to: 50000 curr count: 46813
2021/12/11 08:18:39 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/11 08:18:39 Post SET ops. Expected item count: 50000 got 50000
2021/12/11 08:18:39 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 08:18:44 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/11 08:18:49 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/11 08:18:54 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/11 08:18:59 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/11 08:19:04 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/11 08:19:09 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/11 08:19:10 App: TestDuplicateTimerFiringEventingRebOut got un-deployed
2021/12/11 08:19:10 No eventing-con process running
2021/12/11 08:19:10 Function: TestDuplicateTimerFiringEventingRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringEventingRebOut deleting in the background"
}
2021/12/11 08:19:11 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:19:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 08:19:21 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/11 08:19:26 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/11 08:19:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 08:19:28 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:19:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 08:19:38 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/11 08:19:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringEventingRebOut (189.06s)
=== RUN   TestDuplicateTimerFiringEventingSwapReb
2021/12/11 08:19:48 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/11 08:19:48 App: TestDuplicateTimerFiringEventingSwapReb got un-deployed
2021/12/11 08:19:48 No eventing-con process running
2021/12/11 08:19:48 Function: TestDuplicateTimerFiringEventingSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringEventingSwapReb not found"
 }
}
2021/12/11 08:19:50 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:19:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 08:19:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 08:19:57 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:20:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 08:20:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:20:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 08:20:07 Deploying app: TestDuplicateTimerFiringEventingSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/11 08:20:08 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringEventingSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/11 08:20:13 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get deployed
2021/12/11 08:20:18 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get deployed
2021/12/11 08:20:23 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get deployed
2021/12/11 08:20:28 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get deployed
2021/12/11 08:20:28 App: TestDuplicateTimerFiringEventingSwapReb got deployed
2021/12/11 08:20:28 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/11 08:20:28 Starting bucket ops to source bucket
2021/12/11 08:20:28 Removing node(s): [] from the cluster
2021/12/11 08:20:33 Rebalance progress: 12.5
2021/12/11 08:20:38 Rebalance progress: 12.5
2021/12/11 08:20:43 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:20:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:20:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 08:20:43 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/11 08:20:43 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/11 08:20:48 Rebalance progress: 12.5
2021/12/11 08:20:53 Rebalance progress: 12.5
2021/12/11 08:20:58 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:20:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:20:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 08:20:58 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/11 08:21:03 Rebalance progress: 12.5
2021/12/11 08:21:08 Rebalance progress: 12.5
2021/12/11 08:21:13 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:21:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:21:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 08:21:14 Waiting for src bucket item count to get to: 50000 curr count: 49273
2021/12/11 08:21:19 Waiting for src bucket item count to get to: 50000 curr count: 49273
2021/12/11 08:21:29 Waiting for dst bucket item count to get to: 50000 curr count: 42100
2021/12/11 08:21:34 Waiting for dst bucket item count to get to: 50000 curr count: 42100
2021/12/11 08:21:34 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/11 08:21:34 Post SET ops. Expected item count: 50000 got 50000
2021/12/11 08:21:34 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 08:21:39 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/11 08:21:44 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/11 08:21:49 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/11 08:21:54 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/11 08:21:59 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/11 08:22:04 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/11 08:22:04 App: TestDuplicateTimerFiringEventingSwapReb got un-deployed
2021/12/11 08:22:04 No eventing-con process running
2021/12/11 08:22:04 Function: TestDuplicateTimerFiringEventingSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringEventingSwapReb deleting in the background"
}
2021/12/11 08:22:06 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:22:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 08:22:16 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/11 08:22:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 08:22:17 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:22:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 08:22:27 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/11 08:22:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringEventingSwapReb (169.38s)
=== RUN   TestDuplicateTimerFiringKVRebIn
2021/12/11 08:22:37 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/11 08:22:37 App: TestDuplicateTimerFiringKVRebIn got un-deployed
2021/12/11 08:22:37 No eventing-con process running
2021/12/11 08:22:37 Function: TestDuplicateTimerFiringKVRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringKVRebIn not found"
 }
}
2021/12/11 08:22:39 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:22:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 08:22:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 08:22:46 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:22:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 08:22:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:22:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 08:22:56 Deploying app: TestDuplicateTimerFiringKVRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/11 08:22:57 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringKVRebIn' in metakv",
  "warnings": null
 }
}
2021/12/11 08:23:02 Waiting for app: TestDuplicateTimerFiringKVRebIn to get deployed
2021/12/11 08:23:07 Waiting for app: TestDuplicateTimerFiringKVRebIn to get deployed
2021/12/11 08:23:12 Waiting for app: TestDuplicateTimerFiringKVRebIn to get deployed
2021/12/11 08:23:17 Waiting for app: TestDuplicateTimerFiringKVRebIn to get deployed
2021/12/11 08:23:17 App: TestDuplicateTimerFiringKVRebIn got deployed
2021/12/11 08:23:17 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/11 08:23:17 Starting bucket ops to source bucket
2021/12/11 08:23:18 Removing node(s): [] from the cluster
2021/12/11 08:23:23 Rebalance progress: 12.5
2021/12/11 08:23:28 Rebalance progress: 12.5
2021/12/11 08:23:33 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:23:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:23:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 08:23:33 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/11 08:23:38 Rebalance progress: 12.5
2021/12/11 08:23:43 Rebalance progress: 12.5
2021/12/11 08:23:48 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:23:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:23:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 08:23:48 Waiting for src bucket item count to get to: 50000 curr count: 27938
2021/12/11 08:23:53 Waiting for src bucket item count to get to: 50000 curr count: 38183
2021/12/11 08:23:58 Waiting for src bucket item count to get to: 50000 curr count: 38183
2021/12/11 08:24:08 Waiting for dst bucket item count to get to: 50000 curr count: 32872
2021/12/11 08:24:13 Waiting for dst bucket item count to get to: 50000 curr count: 32872
2021/12/11 08:24:18 Waiting for dst bucket item count to get to: 50000 curr count: 44636
2021/12/11 08:24:23 Waiting for dst bucket item count to get to: 50000 curr count: 44636
2021/12/11 08:24:28 Waiting for dst bucket item count to get to: 50000 curr count: 49688
2021/12/11 08:24:33 Waiting for dst bucket item count to get to: 50000 curr count: 49688
2021/12/11 08:24:33 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/11 08:24:33 Post SET ops. Expected item count: 50000 got 50000
2021/12/11 08:24:33 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 08:24:38 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/11 08:24:43 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/11 08:24:48 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/11 08:24:53 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/11 08:24:58 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/11 08:24:58 App: TestDuplicateTimerFiringKVRebIn got un-deployed
2021/12/11 08:24:58 No eventing-con process running
2021/12/11 08:24:58 Function: TestDuplicateTimerFiringKVRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringKVRebIn deleting in the background"
}
2021/12/11 08:25:00 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:25:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 08:25:10 Waiting for dst bucket item count to get to: 0 curr count: 13774
2021/12/11 08:25:15 Waiting for dst bucket item count to get to: 0 curr count: 13774
2021/12/11 08:25:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 08:25:17 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:25:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 08:25:27 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/11 08:25:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringKVRebIn (179.54s)
=== RUN   TestDuplicateTimerFiringKVRebOut
2021/12/11 08:25:32 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/11 08:25:32 Removing node(s): [] from the cluster
2021/12/11 08:25:37 Rebalance progress: 100
2021/12/11 08:25:42 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/11 08:25:42 App: TestDuplicateTimerFiringKVRebOut got un-deployed
2021/12/11 08:25:42 No eventing-con process running
2021/12/11 08:25:42 Function: TestDuplicateTimerFiringKVRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringKVRebOut not found"
 }
}
2021/12/11 08:25:44 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:25:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 08:25:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 08:25:51 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:25:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 08:25:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:26:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 08:26:01 Deploying app: TestDuplicateTimerFiringKVRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/11 08:26:02 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringKVRebOut' in metakv",
  "warnings": null
 }
}
2021/12/11 08:26:07 Waiting for app: TestDuplicateTimerFiringKVRebOut to get deployed
2021/12/11 08:26:12 Waiting for app: TestDuplicateTimerFiringKVRebOut to get deployed
2021/12/11 08:26:17 Waiting for app: TestDuplicateTimerFiringKVRebOut to get deployed
2021/12/11 08:26:22 Waiting for app: TestDuplicateTimerFiringKVRebOut to get deployed
2021/12/11 08:26:22 App: TestDuplicateTimerFiringKVRebOut got deployed
2021/12/11 08:26:22 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/11 08:26:22 Starting bucket ops to source bucket
2021/12/11 08:26:27 Rebalance progress: 12.5
2021/12/11 08:26:32 Rebalance progress: 12.5
2021/12/11 08:26:37 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:26:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:26:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 08:26:37 Waiting for src bucket item count to get to: 50000 curr count: 9852
2021/12/11 08:26:42 Waiting for src bucket item count to get to: 50000 curr count: 22417
2021/12/11 08:26:47 Waiting for src bucket item count to get to: 50000 curr count: 22417
2021/12/11 08:26:52 Waiting for src bucket item count to get to: 50000 curr count: 33727
2021/12/11 08:26:57 Waiting for src bucket item count to get to: 50000 curr count: 33727
2021/12/11 08:27:02 Waiting for src bucket item count to get to: 50000 curr count: 33727
2021/12/11 08:27:07 Waiting for src bucket item count to get to: 50000 curr count: 43924
2021/12/11 08:27:12 Waiting for src bucket item count to get to: 50000 curr count: 43924
2021/12/11 08:27:22 Waiting for dst bucket item count to get to: 50000 curr count: 38999
2021/12/11 08:27:27 Waiting for dst bucket item count to get to: 50000 curr count: 38999
2021/12/11 08:27:27 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/11 08:27:27 Post SET ops. Expected item count: 50000 got 50000
2021/12/11 08:27:27 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 08:27:32 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/11 08:27:37 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/11 08:27:42 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/11 08:27:47 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/11 08:27:53 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/11 08:27:58 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/11 08:27:58 App: TestDuplicateTimerFiringKVRebOut got un-deployed
2021/12/11 08:27:58 No eventing-con process running
2021/12/11 08:27:58 Function: TestDuplicateTimerFiringKVRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringKVRebOut deleting in the background"
}
2021/12/11 08:27:59 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:28:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 08:28:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 08:28:06 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:28:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 08:28:16 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/11 08:28:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringKVRebOut (169.16s)
=== RUN   TestDuplicateTimerFiringKVSwapReb
2021/12/11 08:28:26 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/11 08:28:26 App: TestDuplicateTimerFiringKVSwapReb got un-deployed
2021/12/11 08:28:26 No eventing-con process running
2021/12/11 08:28:26 Function: TestDuplicateTimerFiringKVSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringKVSwapReb not found"
 }
}
2021/12/11 08:28:28 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:28:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 08:28:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 08:28:35 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:28:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 08:28:40 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:28:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 08:28:45 Deploying app: TestDuplicateTimerFiringKVSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/11 08:28:46 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringKVSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/11 08:28:51 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get deployed
2021/12/11 08:28:56 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get deployed
2021/12/11 08:29:01 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get deployed
2021/12/11 08:29:06 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get deployed
2021/12/11 08:29:06 App: TestDuplicateTimerFiringKVSwapReb got deployed
2021/12/11 08:29:06 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/11 08:29:06 Starting bucket ops to source bucket
2021/12/11 08:29:06 Removing node(s): [] from the cluster
2021/12/11 08:29:11 Rebalance progress: 12.5
2021/12/11 08:29:16 Rebalance progress: 12.5
2021/12/11 08:29:21 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:29:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:29:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 08:29:21 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/11 08:29:21 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/11 08:29:27 Rebalance progress: 12.5
2021/12/11 08:29:32 Rebalance progress: 12.5
2021/12/11 08:29:37 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:29:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:29:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 08:29:37 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/11 08:29:42 Rebalance progress: 12.5
2021/12/11 08:29:47 Rebalance progress: 12.5
2021/12/11 08:29:52 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:29:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:29:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 08:29:57 Waiting for dst bucket item count to get to: 50000 curr count: 34589
2021/12/11 08:30:02 Waiting for dst bucket item count to get to: 50000 curr count: 34589
2021/12/11 08:30:07 Waiting for dst bucket item count to get to: 50000 curr count: 42393
2021/12/11 08:30:12 Waiting for dst bucket item count to get to: 50000 curr count: 42393
2021/12/11 08:30:12 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/11 08:30:12 Post SET ops. Expected item count: 50000 got 50000
2021/12/11 08:30:12 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 08:30:17 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/11 08:30:22 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/11 08:30:27 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/11 08:30:32 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/11 08:30:37 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/11 08:30:42 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/11 08:30:42 App: TestDuplicateTimerFiringKVSwapReb got un-deployed
2021/12/11 08:30:42 No eventing-con process running
2021/12/11 08:30:42 Function: TestDuplicateTimerFiringKVSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringKVSwapReb deleting in the background"
}
2021/12/11 08:30:44 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:30:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 08:30:54 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/11 08:30:59 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/11 08:30:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 08:31:01 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:31:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 08:31:11 Waiting for dst bucket item count to get to: 0 curr count: 32357
2021/12/11 08:31:16 Waiting for dst bucket item count to get to: 0 curr count: 32357
2021/12/11 08:31:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringKVSwapReb (179.41s)
=== RUN   TestKVRebOnUpdateBucketOpOneByOne
2021/12/11 08:31:26 Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 08:31:26 App: bucket_op_on_update got un-deployed
2021/12/11 08:31:26 No eventing-con process running
2021/12/11 08:31:26 Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/11 08:31:27 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:31:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 08:31:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 08:31:34 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:31:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 08:31:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:31:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 08:31:44 Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/11 08:31:45 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/11 08:31:50 Starting bucket ops to source bucket
2021/12/11 08:31:55 Waiting for app: bucket_op_on_update to get deployed
2021/12/11 08:32:00 Waiting for app: bucket_op_on_update to get deployed
2021/12/11 08:32:05 Waiting for app: bucket_op_on_update to get deployed
2021/12/11 08:32:05 App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:32:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:32:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 08:32:05 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/11 08:32:06 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/11 08:32:11 Rebalance progress: 12.5
2021/12/11 08:32:16 Rebalance progress: 12.5
2021/12/11 08:32:21 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:32:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:32:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 08:32:21 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/11 08:32:21 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/11 08:32:26 Rebalance progress: 12.5
2021/12/11 08:32:31 Rebalance progress: 12.5
2021/12/11 08:32:36 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:32:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:32:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 08:32:36 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: kv to the cluster
2021/12/11 08:32:37 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/11 08:32:42 Rebalance progress: 12.5
2021/12/11 08:32:47 Rebalance progress: 12.5
2021/12/11 08:32:52 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:32:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:32:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 08:32:52 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/11 08:32:57 Rebalance progress: 12.5
2021/12/11 08:33:02 Rebalance progress: 12.5
2021/12/11 08:33:07 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:33:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:33:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 08:33:07 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/11 08:33:12 Rebalance progress: 12.5
2021/12/11 08:33:17 Rebalance progress: 12.5
2021/12/11 08:33:22 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:33:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:33:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 08:33:22 clusterSetup.go:253: Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/11 08:33:27 Rebalance progress: 12.5
2021/12/11 08:33:32 Rebalance progress: 12.5
2021/12/11 08:33:37 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:33:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:33:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 08:33:37 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 08:33:42 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 08:33:47 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 08:33:52 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 08:33:57 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 08:34:02 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 08:34:07 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 08:34:07 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/11 08:34:07 eventingSetup.go:46: No eventing-con process running
2021/12/11 08:34:07 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/11 08:34:09 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:34:14 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 08:34:19 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 10663
2021/12/11 08:34:19 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 08:34:21 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:34:26 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 08:34:31 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 9544
2021/12/11 08:34:36 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 9544
2021/12/11 08:34:36 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebOnUpdateBucketOpOneByOne (200.31s)
=== RUN   TestKVForceFailover
2021/12/11 08:34:51 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 08:34:51 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/11 08:34:51 eventingSetup.go:46: No eventing-con process running
2021/12/11 08:34:51 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/11 08:34:53 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:34:58 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 08:34:58 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 08:34:59 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:35:04 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 08:35:04 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 08:35:04 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/11 08:35:05 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/11 08:35:10 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:35:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 08:35:10 Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/11 08:35:11 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/11 08:35:11 Starting bucket ops to source bucket
2021/12/11 08:35:16 Waiting for app: bucket_op_on_update to get deployed
2021/12/11 08:35:21 Waiting for app: bucket_op_on_update to get deployed
2021/12/11 08:35:26 Waiting for app: bucket_op_on_update to get deployed
2021/12/11 08:35:31 Waiting for app: bucket_op_on_update to get deployed
2021/12/11 08:35:31 App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:35:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:35:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 08:35:31 clusterSetup.go:229: Failing over: [127.0.0.1:19001]
2021/12/11 08:35:51 clusterSetup.go:237: Kicking off failover recovery, type: full
2021/12/11 08:35:51 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/11 08:35:57 Rebalance progress: 12.5
2021/12/11 08:36:02 Rebalance progress: 12.5
2021/12/11 08:36:07 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:36:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:36:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 08:36:07 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002] from the cluster
2021/12/11 08:36:12 Rebalance progress: 12.5
2021/12/11 08:36:17 Rebalance progress: 12.5
2021/12/11 08:36:22 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:36:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:36:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 08:36:22 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 08:36:27 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 08:36:32 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 08:36:37 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 08:36:42 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 08:36:47 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 08:36:52 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 08:36:52 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/11 08:36:52 eventingSetup.go:46: No eventing-con process running
2021/12/11 08:36:52 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/11 08:36:54 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:36:59 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 08:37:04 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7054
2021/12/11 08:37:09 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7054
2021/12/11 08:37:09 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 08:37:11 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:37:16 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 08:37:21 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6019
2021/12/11 08:37:26 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6019
2021/12/11 08:37:26 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVForceFailover (169.95s)
=== RUN   TestKVHardFailoverRecovery
2021/12/11 08:37:36 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 08:37:36 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/11 08:37:36 eventingSetup.go:46: No eventing-con process running
2021/12/11 08:37:36 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/11 08:37:38 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:37:43 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 08:37:43 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 08:37:44 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:37:49 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 08:37:49 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:37:54 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 08:37:54 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/11 08:37:55 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/11 08:38:00 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/11 08:38:05 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/11 08:38:10 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/11 08:38:16 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/11 08:38:16 clusterSetup.go:354: App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:38:16 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:38:16 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 08:38:16 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/11 08:38:16 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/11 08:38:21 Rebalance progress: 12.5
2021/12/11 08:38:26 Rebalance progress: 12.5
2021/12/11 08:38:31 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:38:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:38:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 08:38:31 clusterSetup.go:229: Failing over: [127.0.0.1:19001]
2021/12/11 08:39:31 clusterSetup.go:237: Kicking off failover recovery, type: full
2021/12/11 08:39:36 Rebalance progress: 12.5
2021/12/11 08:39:41 Rebalance progress: 12.5
2021/12/11 08:39:46 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:39:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:39:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 08:39:46 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/11 08:39:52 Rebalance progress: 12.5
2021/12/11 08:39:57 Rebalance progress: 12.5
2021/12/11 08:40:02 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:40:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:40:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 08:40:02 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 08:40:07 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 08:40:12 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 08:40:17 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 08:40:22 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 08:40:27 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 08:40:32 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 08:40:32 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/11 08:40:32 eventingSetup.go:46: No eventing-con process running
2021/12/11 08:40:32 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/11 08:40:34 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:40:39 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 08:40:44 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12097
2021/12/11 08:40:49 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12097
2021/12/11 08:40:49 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 08:40:51 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:40:56 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 08:41:01 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 8343
2021/12/11 08:41:06 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 8343
2021/12/11 08:41:06 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVHardFailoverRecovery (220.00s)
=== RUN   TestKVRebOnUpdateBucketOpAllAtOnce
2021/12/11 08:41:16 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 08:41:16 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/11 08:41:16 eventingSetup.go:46: No eventing-con process running
2021/12/11 08:41:16 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/11 08:41:18 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:41:23 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 08:41:23 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 08:41:24 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:41:29 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 08:41:29 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:41:34 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 08:41:34 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/11 08:41:35 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/11 08:41:40 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/11 08:41:45 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/11 08:41:50 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/11 08:41:55 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/11 08:41:55 clusterSetup.go:354: App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:41:55 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:41:55 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 08:41:56 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/11 08:41:56 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/11 08:41:56 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: kv to the cluster
2021/12/11 08:41:57 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/11 08:42:02 Rebalance progress: 12.5
2021/12/11 08:42:07 Rebalance progress: 12.5
2021/12/11 08:42:12 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:42:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:42:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 08:42:12 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002 127.0.0.1:19003] from the cluster
2021/12/11 08:42:17 Rebalance progress: 12.5
2021/12/11 08:42:22 Rebalance progress: 12.5
2021/12/11 08:42:27 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:42:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:42:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 08:42:27 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 08:42:32 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 08:42:37 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 08:42:42 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 08:42:47 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 08:42:52 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 08:42:57 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 08:42:57 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/11 08:42:57 eventingSetup.go:46: No eventing-con process running
2021/12/11 08:42:57 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/11 08:42:59 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:43:04 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 08:43:09 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4611
2021/12/11 08:43:09 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 08:43:11 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:43:16 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 08:43:21 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3215
2021/12/11 08:43:26 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3215
2021/12/11 08:43:26 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebOnUpdateBucketOpAllAtOnce (139.77s)
=== RUN   TestKVSwapRebOnUpdateBucketOp
2021/12/11 08:43:36 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 08:43:36 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/11 08:43:36 eventingSetup.go:46: No eventing-con process running
2021/12/11 08:43:36 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/11 08:43:37 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:43:42 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 08:43:42 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 08:43:44 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:43:49 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 08:43:49 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:43:49 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 08:43:49 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/11 08:43:50 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/11 08:43:55 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/11 08:44:00 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/11 08:44:05 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/11 08:44:10 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/11 08:44:10 clusterSetup.go:354: App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:44:10 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:44:10 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 08:44:10 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/11 08:44:11 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/11 08:44:16 Rebalance progress: 12.5
2021/12/11 08:44:21 Rebalance progress: 12.5
2021/12/11 08:44:26 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:44:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:44:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 08:44:26 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/11 08:44:26 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/11 08:44:31 Rebalance progress: 12.5
2021/12/11 08:44:36 Rebalance progress: 12.5
2021/12/11 08:44:41 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:44:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:44:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 08:44:41 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/11 08:44:46 Rebalance progress: 12.5
2021/12/11 08:44:51 Rebalance progress: 12.5
2021/12/11 08:44:56 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:44:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:44:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 08:44:57 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 08:45:02 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 08:45:07 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 08:45:12 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 08:45:17 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 08:45:22 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 08:45:27 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 08:45:27 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/11 08:45:27 eventingSetup.go:46: No eventing-con process running
2021/12/11 08:45:27 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/11 08:45:29 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:45:34 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 08:45:39 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6120
2021/12/11 08:45:39 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 08:45:40 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:45:45 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 08:45:50 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3005
2021/12/11 08:45:55 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3005
2021/12/11 08:45:55 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 08:46:00 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 08:46:00 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/11 08:46:00 eventingSetup.go:46: No eventing-con process running
2021/12/11 08:46:00 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/11 08:46:02 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:46:07 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 08:46:07 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 08:46:09 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:46:14 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 08:46:14 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVSwapRebOnUpdateBucketOp (168.40s)
=== RUN   TestKVRebalanceWithMultipleHandlers
2021/12/11 08:46:24 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 08:46:24 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/11 08:46:24 eventingSetup.go:46: No eventing-con process running
2021/12/11 08:46:24 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/11 08:46:26 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:46:31 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 08:46:31 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 08:46:33 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:46:38 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 08:46:38 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 08:46:43 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/11 08:46:43 clusterSetup.go:405: App: sys_test_bucket_op got un-deployed
2021/12/11 08:46:43 eventingSetup.go:46: No eventing-con process running
2021/12/11 08:46:43 eventingSetup.go:418: Function: sys_test_bucket_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: sys_test_bucket_op not found"
 }
}
2021/12/11 08:46:44 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:46:49 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 08:46:49 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 08:46:51 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:46:56 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 08:46:56 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:46:56 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 08:46:56 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/11 08:46:57 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:46:57 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 08:46:57 eventingSetup.go:125: Deploying app: sys_test_bucket_op. Item count src bucket: 0 dst bucket: 0
2021/12/11 08:46:58 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'sys_test_bucket_op' in metakv",
  "warnings": null
 }
}
2021/12/11 08:47:03 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/11 08:47:08 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/11 08:47:13 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/11 08:47:18 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/11 08:47:18 clusterSetup.go:354: App: bucket_op_on_update got deployed
2021/12/11 08:47:23 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/11 08:47:28 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/11 08:47:33 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/11 08:47:33 clusterSetup.go:354: App: sys_test_bucket_op got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:47:33 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:47:33 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 08:47:33 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/11 08:47:34 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/11 08:47:39 Rebalance progress: 12.5
2021/12/11 08:47:44 Rebalance progress: 12.5
2021/12/11 08:47:49 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:47:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:47:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 08:47:49 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/11 08:47:49 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/11 08:47:54 Rebalance progress: 12.5
2021/12/11 08:47:59 Rebalance progress: 12.5
2021/12/11 08:48:04 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:48:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:48:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 08:48:04 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: kv to the cluster
2021/12/11 08:48:05 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/11 08:48:10 Rebalance progress: 12.5
2021/12/11 08:48:15 Rebalance progress: 12.5
2021/12/11 08:48:20 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:48:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:48:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 08:48:20 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/11 08:48:25 Rebalance progress: 12.5
2021/12/11 08:48:30 Rebalance progress: 12.5
2021/12/11 08:48:35 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:48:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:48:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 08:48:35 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/11 08:48:40 Rebalance progress: 12.5
2021/12/11 08:48:45 Rebalance progress: 12.5
2021/12/11 08:48:50 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:48:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:48:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 08:48:50 clusterSetup.go:253: Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/11 08:48:55 Rebalance progress: 12.5
2021/12/11 08:49:00 Rebalance progress: 12.5
2021/12/11 08:49:05 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:49:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:49:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 08:49:05 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 08:49:10 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 08:49:15 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 08:49:21 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 08:49:26 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 08:49:31 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 08:49:36 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 08:49:41 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 08:49:41 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/11 08:49:41 eventingSetup.go:46: No eventing-con process running
2021/12/11 08:49:41 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/11 08:49:44 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:49:49 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 08:49:54 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12183
2021/12/11 08:49:59 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12183
2021/12/11 08:49:59 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 08:50:01 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:50:06 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 08:50:11 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 9696
2021/12/11 08:50:16 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 9696
2021/12/11 08:50:16 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 08:50:16 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 08:50:21 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/11 08:50:26 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/11 08:50:31 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/11 08:50:36 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/11 08:50:41 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/11 08:50:46 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/11 08:50:46 clusterSetup.go:405: App: sys_test_bucket_op got un-deployed
2021/12/11 08:50:46 eventingSetup.go:46: No eventing-con process running
2021/12/11 08:50:46 eventingSetup.go:418: Function: sys_test_bucket_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: sys_test_bucket_op deleting in the background"
}
2021/12/11 08:50:48 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:50:53 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 08:50:53 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 08:50:55 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:51:00 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 08:51:00 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebalanceWithMultipleHandlers (285.70s)
=== RUN   TestKVSwapRebalanceWithMultipleHandlers
2021/12/11 08:51:10 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 08:51:10 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/11 08:51:10 eventingSetup.go:46: No eventing-con process running
2021/12/11 08:51:10 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/11 08:51:11 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:51:16 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 08:51:16 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 08:51:18 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:51:23 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 08:51:23 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 08:51:28 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/11 08:51:28 clusterSetup.go:405: App: sys_test_bucket_op got un-deployed
2021/12/11 08:51:28 eventingSetup.go:46: No eventing-con process running
2021/12/11 08:51:28 eventingSetup.go:418: Function: sys_test_bucket_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: sys_test_bucket_op not found"
 }
}
2021/12/11 08:51:30 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:51:35 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 08:51:35 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 08:51:37 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:51:42 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 08:51:42 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:51:42 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 08:51:42 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/11 08:51:43 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:51:43 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 08:51:43 eventingSetup.go:125: Deploying app: sys_test_bucket_op. Item count src bucket: 0 dst bucket: 0
2021/12/11 08:51:44 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'sys_test_bucket_op' in metakv",
  "warnings": null
 }
}
2021/12/11 08:51:49 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/11 08:51:54 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/11 08:51:59 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/11 08:52:04 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/11 08:52:04 clusterSetup.go:354: App: bucket_op_on_update got deployed
2021/12/11 08:52:09 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/11 08:52:14 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/11 08:52:19 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/11 08:52:19 clusterSetup.go:354: App: sys_test_bucket_op got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:52:19 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:52:19 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 08:52:19 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/11 08:52:20 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/11 08:52:25 Rebalance progress: 12.5
2021/12/11 08:52:30 Rebalance progress: 12.5
2021/12/11 08:52:35 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:52:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:52:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 08:52:35 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/11 08:52:35 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/11 08:52:40 Rebalance progress: 12.5
2021/12/11 08:52:45 Rebalance progress: 12.5
2021/12/11 08:52:50 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:52:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:52:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 08:52:50 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/11 08:52:55 Rebalance progress: 12.5
2021/12/11 08:53:00 Rebalance progress: 12.5
2021/12/11 08:53:05 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:53:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:53:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 08:53:05 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 08:53:10 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 08:53:15 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 08:53:21 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 08:53:26 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 08:53:31 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 08:53:31 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/11 08:53:31 eventingSetup.go:46: No eventing-con process running
2021/12/11 08:53:31 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/11 08:53:36 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:53:41 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 08:53:46 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7631
2021/12/11 08:53:46 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 08:53:48 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:53:53 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 08:53:58 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7631
2021/12/11 08:53:58 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 08:53:58 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 08:54:03 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/11 08:54:08 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/11 08:54:13 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/11 08:54:18 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/11 08:54:23 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/11 08:54:23 clusterSetup.go:405: App: sys_test_bucket_op got un-deployed
2021/12/11 08:54:23 eventingSetup.go:46: No eventing-con process running
2021/12/11 08:54:23 eventingSetup.go:418: Function: sys_test_bucket_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: sys_test_bucket_op deleting in the background"
}
2021/12/11 08:54:25 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:54:30 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 08:54:30 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 08:54:32 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:54:37 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 08:54:37 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVSwapRebalanceWithMultipleHandlers (216.84s)
=== RUN   TestKVRebStopStartKVOpsOnUpdateBucketOpOneByOne
2021/12/11 08:54:47 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 08:54:47 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/11 08:54:47 eventingSetup.go:46: No eventing-con process running
2021/12/11 08:54:47 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/11 08:54:48 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:54:53 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 08:54:53 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 08:54:55 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:55:00 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 08:55:00 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:55:05 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 08:55:05 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/11 08:55:06 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/11 08:55:11 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/11 08:55:16 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/11 08:55:21 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/11 08:55:26 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/11 08:55:26 clusterSetup.go:354: App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:55:26 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:55:26 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 08:55:26 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/11 08:55:27 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/11 08:55:47 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:55:47 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:55:47 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:55:57 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:55:57 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 08:55:57 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/11 08:56:02 Rebalance progress: 12.5
2021/12/11 08:56:07 Rebalance progress: 12.5
2021/12/11 08:56:12 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:56:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:56:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 08:56:12 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/11 08:56:13 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/11 08:56:18 Rebalance progress: 12.5
2021/12/11 08:56:23 Rebalance progress: 12.5
2021/12/11 08:56:28 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:56:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:56:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 08:56:28 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/11 08:56:33 Rebalance progress: 12.5
2021/12/11 08:56:38 Rebalance progress: 12.5
2021/12/11 08:56:43 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:56:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:56:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 08:56:43 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 08:56:48 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 08:56:53 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 08:56:58 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 08:57:03 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 08:57:08 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 08:57:13 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 08:57:13 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/11 08:57:13 eventingSetup.go:46: No eventing-con process running
2021/12/11 08:57:13 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/11 08:57:15 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:57:20 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 08:57:25 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 9135
2021/12/11 08:57:25 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 08:57:27 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:57:32 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 08:57:37 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 9135
2021/12/11 08:57:37 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebStopStartKVOpsOnUpdateBucketOpOneByOne (180.09s)
=== RUN   TestKVFailoverOnUpdateBucketOp
2021/12/11 08:57:47 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 08:57:47 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/11 08:57:47 eventingSetup.go:46: No eventing-con process running
2021/12/11 08:57:47 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/11 08:57:48 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:57:53 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 08:57:53 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 08:57:55 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:58:00 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 08:58:00 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:58:05 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 08:58:05 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/11 08:58:06 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/11 08:58:11 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/11 08:58:16 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/11 08:58:21 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/11 08:58:26 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/11 08:58:26 clusterSetup.go:354: App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:58:26 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:58:26 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 08:58:26 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/11 08:58:27 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/11 08:58:32 Rebalance progress: 12.5
2021/12/11 08:58:37 Rebalance progress: 12.5
2021/12/11 08:58:42 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:58:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:58:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 08:58:42 clusterSetup.go:229: Failing over: [127.0.0.1:19001]
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:58:52 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:58:52 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 08:58:52 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/11 08:58:57 Rebalance progress: 12.5
2021/12/11 08:59:02 Rebalance progress: 12.5
2021/12/11 08:59:07 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:59:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:59:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 08:59:07 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/11 08:59:07 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/11 08:59:12 Rebalance progress: 12.5
2021/12/11 08:59:17 Rebalance progress: 12.5
2021/12/11 08:59:22 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:59:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:59:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 08:59:22 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/11 08:59:27 Rebalance progress: 12.5
2021/12/11 08:59:32 Rebalance progress: 12.5
2021/12/11 08:59:37 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:59:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 08:59:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 08:59:38 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 08:59:43 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 08:59:48 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 08:59:53 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 08:59:58 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 09:00:03 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 09:00:03 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/11 09:00:03 eventingSetup.go:46: No eventing-con process running
2021/12/11 09:00:03 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/11 09:00:05 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 09:00:10 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 09:00:15 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 8599
2021/12/11 09:00:15 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 09:00:16 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 09:00:21 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 09:00:26 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 8599
2021/12/11 09:00:26 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVFailoverOnUpdateBucketOp (169.66s)
=== RUN   TestBootstrapAfterKVHardFailover
2021/12/11 09:00:31 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/11 09:00:31 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/11 09:00:37 Rebalance progress: 100
2021/12/11 09:00:37 Failing over: [127.0.0.1:19001]
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 09:01:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 09:01:07 Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/11 09:01:08 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/11 09:01:13 Removing node(s): [] from the cluster
2021/12/11 09:01:13 Starting bucket ops to source bucket
2021/12/11 09:01:18 Rebalance failed. See logs for detailed reason. You can try again.
2021/12/11 09:01:23 Waiting for app: bucket_op_on_update to get deployed
2021/12/11 09:01:28 Waiting for app: bucket_op_on_update to get deployed
2021/12/11 09:01:28 App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 09:01:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 09:01:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 09:01:28 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 09:01:33 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 09:01:38 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 09:01:43 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 09:01:48 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 09:01:53 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 09:01:53 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/11 09:01:53 eventingSetup.go:46: No eventing-con process running
2021/12/11 09:01:53 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/11 09:01:55 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 09:02:00 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 09:02:05 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1491
2021/12/11 09:02:05 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 09:02:07 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 09:02:12 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 09:02:17 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1491
2021/12/11 09:02:17 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestBootstrapAfterKVHardFailover (110.43s)
PASS
ok  	github.com/couchbase/eventing/tests/functional_tests	12104.414s

Rebalance tests took 3 hours, 21 minutes and 47 seconds.

Testing Succeeded


Version: versions-11.12.2021-01.20.cfg
Build Log: make-11.12.2021-01.20.log
Server Log: logs-11.12.2021-01.20.tar.gz

Finished