Building

Started building at 2021/12/08 18:27:52
Calculating base
Updating mirror
Basing run on 7.1.0-1866 1b52f26ea
Updating tree for run 08.12.2021-18.27
gocb is at 635fa6c, changes since last good build: none
libcouchbase is at f6822fc, changes since last good build: none
indexing is at f1a81ab, changes since last good build: none
kv_engine is at 6bd64be, changes since last good build: 
 6bd64be10 [Magma] Don't unconditionally read droppedCollections in saveDocs
 2a88a4359 Clang-Tidy: EPBucket::makeVBucket()
 3b6466448 MB-48816: Avoid writing stats responses from background tasks
 72bc6eac1 MB-49626: Move dropped collection stats to MagmaDbStats
 cb68839a6 MB-49746: Reschedule CompactTask after a failure
ns_server is at fcadff5, changes since last good build: none
couchstore is at 5aa7c4c, changes since last good build: none
couchdb is at fca2e6b, changes since last good build: none
testrunner is at 926f5ae, changes since last good build: 
 926f5aeb1 Eventing: System event tests
Switching eventing to unstable
eventing is at 9fce322, changes since last good build: none
Switching eventing-ee to unstable
eventing-ee is at 7060ef8, changes since last good build: none
Pulling in uncommitted change 160397 at refs/changes/97/160397/3
Total 23 (delta 15), reused 17 (delta 15)
[unstable bac8d6e8] MB-100 : Do not fail n1ql timeout test if we are catching timeout exceptions successfully
 Author: abhijpes 
 Date: Mon Aug 30 08:24:38 2021 +0530
 1 file changed, 2 insertions(+), 2 deletions(-)
Pulling in uncommitted change 163449 at refs/changes/49/163449/1
Total 48 (delta 27), reused 39 (delta 27)
[unstable c8086115] MB-100: collect data when test fails
 Author: AnkitPrabhu 
 Date: Tue Oct 12 14:02:31 2021 +0530
 1 file changed, 2 insertions(+)
Pulling in uncommitted change 165976 at refs/changes/76/165976/17
Total 4 (delta 3), reused 4 (delta 3)
[unstable 7871fe61] MB-48538: Add new audit ids for different events
 Author: AnkitPrabhu 
 Date: Tue Nov 23 14:29:10 2021 +0530
 1 file changed, 337 insertions(+), 61 deletions(-)
Pulling in uncommitted change 166697 at refs/changes/97/166697/4
Total 10 (delta 9), reused 10 (delta 9)
[unstable c2533abc] MB-49547: Check bucket instance to get kv node for dcp request
 Author: AnkitPrabhu 
 Date: Thu Dec 2 12:22:17 2021 +0530
 6 files changed, 38 insertions(+), 109 deletions(-)
Pulling in uncommitted change 166698 at refs/changes/98/166698/3
Total 4 (delta 3), reused 4 (delta 3)
[unstable 1137023f] MB-48334 : Fail rebalance unless all functions deployed on current encryption level
 Author: abhijpes 
 Date: Wed Dec 1 14:51:29 2021 +0530
 1 file changed, 9 insertions(+)
Pulling in uncommitted change 166812 at refs/changes/12/166812/5
Total 24 (delta 20), reused 24 (delta 20)
[unstable 5bf56324] MB-49547: Refresh bucket info from streaming endpoint instance
 Author: AnkitPrabhu 
 Date: Thu Dec 2 13:55:55 2021 +0530
 7 files changed, 147 insertions(+), 235 deletions(-)
Pulling in uncommitted change 166999 at refs/changes/99/166999/5
Total 56 (delta 47), reused 56 (delta 47)
[unstable e1d9bddd] MB-46304: Add functionality to disable eventing exposed javascript features
 Author: AnkitPrabhu 
 Date: Mon Dec 6 13:26:12 2021 +0530
 20 files changed, 217 insertions(+), 30 deletions(-)
Pulling in uncommitted change 167183 at refs/changes/83/167183/2
Total 5 (delta 2), reused 4 (delta 2)
[unstable 1f976df] MB-46304: Throw runtime exception when curl feature is disabled
 Author: AnkitPrabhu 
 Date: Wed Dec 8 10:39:19 2021 +0530
 1 file changed, 9 insertions(+), 3 deletions(-)
Pulling in uncommitted change 167188 at refs/changes/88/167188/1
Total 32 (delta 25), reused 32 (delta 25)
error: could not apply a2b85b7... MB-49921 : Return appropriate error for an incorrect N1QL statement
hint: after resolving the conflicts, mark the corrected paths
hint: with 'git add ' or 'git rm '
hint: and commit the result with 'git commit'
Recorded preimage for 'evaluator/impl/engine.go'
Recorded preimage for 'evaluator/impl/handlers.go'
Merge Conflict. Ignoring uncommitted change 167188 at refs/changes/88/167188/1
fatal: "167188" no such patch set
Pulling in uncommitted change 167191 at refs/changes/91/167191/1
Total 38 (delta 30), reused 38 (delta 30)
[unstable e840bc3] MB-49931 : Fix "delete from" statement regex
 Author: abhijpes 
 Date: Wed Dec 8 12:53:50 2021 +0530
 1 file changed, 1 insertion(+), 1 deletion(-)
Building cmakefiles and deps
Building main product

Testing

Started testing at 2021/12/08 18:45:17
Testing mode: rebalance,n1ql,functional
Starting server
Starting server: attempt 1

Functional tests

2021/12/08 18:46:48 Started functional
2021/12/08 18:46:57 Adding node: https://127.0.0.1:19001 with role: kv,index,n1ql to the cluster
2021/12/08 18:47:03 Error in adding nodes : , response: {"otpNode":"n_1@127.0.0.1"}
2021/12/08 18:47:03 Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/08 18:47:09 Removing node(s): [] from the cluster
2021/12/08 18:47:14 Rebalance progress: 0
2021/12/08 18:47:19 Rebalance progress: 0.777633101851852
2021/12/08 18:47:24 Rebalance progress: 5.407262731481482
2021/12/08 18:47:29 Rebalance progress: 9.25925925925926
2021/12/08 18:47:34 Rebalance progress: 13.36443865740741
2021/12/08 18:47:39 Rebalance progress: 18.19299768518519
2021/12/08 18:47:44 Rebalance progress: 21.484375
2021/12/08 18:47:49 Rebalance progress: 26.20442708333333
2021/12/08 18:47:54 Rebalance progress: 100
=== RUN   TestAdvancedGetOps
2021/12/08 18:47:59 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/08 18:48:14 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/08 18:48:29 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/08 18:48:44 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/08 18:48:59 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/08 18:49:14 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/08 18:49:29 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/08 18:49:44 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/08 18:49:59 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/08 18:50:14 Deploying app: TestAdvancedGetOps. Item count src bucket: 0 dst bucket: 0
2021/12/08 18:50:16 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedGetOps' in metakv",
  "warnings": null
 }
}
2021/12/08 18:50:21 Waiting for app: TestAdvancedGetOps to get deployed
2021/12/08 18:50:26 Waiting for app: TestAdvancedGetOps to get deployed
2021/12/08 18:50:31 Waiting for app: TestAdvancedGetOps to get deployed
2021/12/08 18:50:36 Waiting for app: TestAdvancedGetOps to get deployed
2021/12/08 18:50:36 App: TestAdvancedGetOps got deployed
2021/12/08 18:50:36 Starting bucket ops to source bucket
2021/12/08 18:50:41 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/08 18:50:46 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/08 18:50:46 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/08 18:50:46 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 18:50:46 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/08 18:50:46 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 18:50:51 Waiting for app: TestAdvancedGetOps to get un-deployed
2021/12/08 18:50:56 Waiting for app: TestAdvancedGetOps to get un-deployed
2021/12/08 18:51:01 Waiting for app: TestAdvancedGetOps to get un-deployed
2021/12/08 18:51:06 Waiting for app: TestAdvancedGetOps to get un-deployed
2021/12/08 18:51:06 App: TestAdvancedGetOps got un-deployed
2021/12/08 18:51:06 No eventing-con process running
2021/12/08 18:51:06 Function: TestAdvancedGetOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedGetOps deleting in the background"
}
2021/12/08 18:51:08 Context: Bucket flush, response code: 200 dump: 
2021/12/08 18:51:18 Waiting for dst bucket item count to get to: 0 curr count: 53
2021/12/08 18:51:23 Waiting for dst bucket item count to get to: 0 curr count: 3
2021/12/08 18:51:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 18:51:25 Context: Bucket flush, response code: 200 dump: 
2021/12/08 18:51:35 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/08 18:51:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedGetOps (215.18s)
=== RUN   TestAdvancedInsertOps
2021/12/08 18:51:35 Deploying app: TestAdvancedInsertOps. Item count src bucket: 0 dst bucket: 0
2021/12/08 18:51:36 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedInsertOps' in metakv",
  "warnings": null
 }
}
2021/12/08 18:51:41 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/08 18:51:46 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/08 18:51:51 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/08 18:51:56 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/08 18:51:56 App: TestAdvancedInsertOps got deployed
2021/12/08 18:51:56 Starting bucket ops to source bucket
2021/12/08 18:52:02 Waiting for dst bucket item count to get to: 1024 curr count: 0
2021/12/08 18:52:02 src & dst bucket item count matched up. src bucket count: 1024 dst bucket count: 1024
2021/12/08 18:52:02 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 18:52:02 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/08 18:52:02 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 18:52:07 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/08 18:52:12 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/08 18:52:17 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/08 18:52:22 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/08 18:52:22 App: TestAdvancedInsertOps got un-deployed
2021/12/08 18:52:22 No eventing-con process running
2021/12/08 18:52:22 Function: TestAdvancedInsertOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedInsertOps deleting in the background"
}
2021/12/08 18:52:23 Context: Bucket flush, response code: 200 dump: 
2021/12/08 18:52:33 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/08 18:52:38 Waiting for dst bucket item count to get to: 0 curr count: 512
2021/12/08 18:52:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 18:52:40 Context: Bucket flush, response code: 200 dump: 
2021/12/08 18:52:50 Waiting for dst bucket item count to get to: 0 curr count: 741
2021/12/08 18:52:55 Waiting for dst bucket item count to get to: 0 curr count: 741
2021/12/08 18:52:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 18:52:55 Testing insert operation on source bucket
2021/12/08 18:52:55 Deploying app: TestAdvancedInsertOps. Item count src bucket: 0 dst bucket: 0
2021/12/08 18:52:56 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedInsertOps' in metakv",
  "warnings": null
 }
}
2021/12/08 18:53:01 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/08 18:53:06 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/08 18:53:11 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/08 18:53:16 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/08 18:53:16 App: TestAdvancedInsertOps got deployed
2021/12/08 18:53:16 Starting bucket ops to source bucket
2021/12/08 18:53:22 Waiting for dst bucket item count to get to: 2048 curr count: 0
2021/12/08 18:53:27 Waiting for dst bucket item count to get to: 2048 curr count: 1770
2021/12/08 18:53:32 Waiting for dst bucket item count to get to: 2048 curr count: 1770
2021/12/08 18:53:32 src & dst bucket item count matched up. src bucket count: 2048 dst bucket count: 2048
2021/12/08 18:53:32 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 18:53:32 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/08 18:53:33 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 18:53:38 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/08 18:53:43 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/08 18:53:48 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/08 18:53:53 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/08 18:53:53 App: TestAdvancedInsertOps got un-deployed
2021/12/08 18:53:53 No eventing-con process running
2021/12/08 18:53:53 Function: TestAdvancedInsertOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedInsertOps deleting in the background"
}
2021/12/08 18:53:54 Context: Bucket flush, response code: 200 dump: 
2021/12/08 18:54:04 Waiting for dst bucket item count to get to: 0 curr count: 2048
2021/12/08 18:54:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 18:54:06 Context: Bucket flush, response code: 200 dump: 
2021/12/08 18:54:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedInsertOps (156.54s)
=== RUN   TestAdvancedUpsertOps
2021/12/08 18:54:11 Starting bucket ops to source bucket
2021/12/08 18:54:12 Deploying app: TestAdvancedUpsertOps. Item count src bucket: 0 dst bucket: 0
2021/12/08 18:54:13 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedUpsertOps' in metakv",
  "warnings": null
 }
}
2021/12/08 18:54:18 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/08 18:54:23 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/08 18:54:28 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/08 18:54:33 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/08 18:54:33 App: TestAdvancedUpsertOps got deployed
2021/12/08 18:54:33 Starting bucket ops to source bucket
2021/12/08 18:54:39 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/08 18:54:44 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/08 18:54:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 18:54:44 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 18:54:44 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/08 18:54:44 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 18:54:49 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/08 18:54:54 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/08 18:54:59 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/08 18:55:04 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/08 18:55:04 App: TestAdvancedUpsertOps got un-deployed
2021/12/08 18:55:04 No eventing-con process running
2021/12/08 18:55:04 Function: TestAdvancedUpsertOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedUpsertOps deleting in the background"
}
2021/12/08 18:55:06 Context: Bucket flush, response code: 200 dump: 
2021/12/08 18:55:16 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/08 18:55:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 18:55:18 Context: Bucket flush, response code: 200 dump: 
2021/12/08 18:55:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 18:55:23 Testing upsert operation on source bucket
2021/12/08 18:55:23 Deploying app: TestAdvancedUpsertOps. Item count src bucket: 0 dst bucket: 0
2021/12/08 18:55:24 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedUpsertOps' in metakv",
  "warnings": null
 }
}
2021/12/08 18:55:29 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/08 18:55:34 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/08 18:55:39 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/08 18:55:44 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/08 18:55:44 App: TestAdvancedUpsertOps got deployed
2021/12/08 18:55:44 Starting bucket ops to source bucket
2021/12/08 18:55:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 18:55:45 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 18:55:45 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/08 18:55:45 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 18:55:50 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/08 18:55:55 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/08 18:56:00 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/08 18:56:05 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/08 18:56:05 App: TestAdvancedUpsertOps got un-deployed
2021/12/08 18:56:05 No eventing-con process running
2021/12/08 18:56:05 Function: TestAdvancedUpsertOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedUpsertOps deleting in the background"
}
2021/12/08 18:56:07 Context: Bucket flush, response code: 200 dump: 
2021/12/08 18:56:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 18:56:14 Context: Bucket flush, response code: 200 dump: 
2021/12/08 18:56:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedUpsertOps (128.04s)
=== RUN   TestAdvancedReplaceOps
2021/12/08 18:56:19 Starting bucket ops to source bucket
2021/12/08 18:56:20 Deploying app: TestAdvancedReplaceOps. Item count src bucket: 0 dst bucket: 0
2021/12/08 18:56:21 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedReplaceOps' in metakv",
  "warnings": null
 }
}
2021/12/08 18:56:26 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/08 18:56:31 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/08 18:56:36 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/08 18:56:41 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/08 18:56:41 App: TestAdvancedReplaceOps got deployed
2021/12/08 18:56:41 Starting bucket ops to source bucket
2021/12/08 18:56:47 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/08 18:56:52 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/08 18:56:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 18:56:52 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 18:56:52 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/08 18:56:52 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 18:56:57 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/08 18:57:02 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/08 18:57:07 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/08 18:57:12 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/08 18:57:12 App: TestAdvancedReplaceOps got un-deployed
2021/12/08 18:57:12 No eventing-con process running
2021/12/08 18:57:12 Function: TestAdvancedReplaceOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedReplaceOps deleting in the background"
}
2021/12/08 18:57:14 Context: Bucket flush, response code: 200 dump: 
2021/12/08 18:57:24 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/08 18:57:29 Waiting for dst bucket item count to get to: 0 curr count: 512
2021/12/08 18:57:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 18:57:31 Context: Bucket flush, response code: 200 dump: 
2021/12/08 18:57:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 18:57:36 Testing upsert operation on source bucket
2021/12/08 18:57:36 Deploying app: TestAdvancedReplaceOps. Item count src bucket: 0 dst bucket: 0
2021/12/08 18:57:37 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedReplaceOps' in metakv",
  "warnings": null
 }
}
2021/12/08 18:57:42 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/08 18:57:47 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/08 18:57:52 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/08 18:57:57 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/08 18:57:57 App: TestAdvancedReplaceOps got deployed
2021/12/08 18:57:57 Starting bucket ops to source bucket
2021/12/08 18:57:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 18:57:58 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 18:57:58 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/08 18:57:59 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 18:58:04 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/08 18:58:09 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/08 18:58:14 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/08 18:58:19 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/08 18:58:19 App: TestAdvancedReplaceOps got un-deployed
2021/12/08 18:58:19 No eventing-con process running
2021/12/08 18:58:19 Function: TestAdvancedReplaceOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedReplaceOps deleting in the background"
}
2021/12/08 18:58:20 Context: Bucket flush, response code: 200 dump: 
2021/12/08 18:58:30 Waiting for dst bucket item count to get to: 0 curr count: 229
2021/12/08 18:58:35 Waiting for dst bucket item count to get to: 0 curr count: 229
2021/12/08 18:58:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 18:58:37 Context: Bucket flush, response code: 200 dump: 
2021/12/08 18:58:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedReplaceOps (142.93s)
=== RUN   TestAdvancedDeleteOps
2021/12/08 18:58:42 Starting bucket ops to source bucket
2021/12/08 18:58:42 Deploying app: TestAdvancedDeleteOps. Item count src bucket: 0 dst bucket: 0
2021/12/08 18:58:43 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedDeleteOps' in metakv",
  "warnings": null
 }
}
2021/12/08 18:58:48 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/08 18:58:53 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/08 18:58:58 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/08 18:59:03 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/08 18:59:03 App: TestAdvancedDeleteOps got deployed
2021/12/08 18:59:03 Starting bucket ops to source bucket
2021/12/08 18:59:08 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/08 18:59:13 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/08 18:59:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 18:59:13 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 18:59:14 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/08 18:59:14 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 18:59:19 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/08 18:59:24 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/08 18:59:29 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/08 18:59:34 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/08 18:59:34 App: TestAdvancedDeleteOps got un-deployed
2021/12/08 18:59:34 No eventing-con process running
2021/12/08 18:59:34 Function: TestAdvancedDeleteOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedDeleteOps deleting in the background"
}
2021/12/08 18:59:35 Context: Bucket flush, response code: 200 dump: 
2021/12/08 18:59:45 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/08 18:59:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 18:59:47 Context: Bucket flush, response code: 200 dump: 
2021/12/08 18:59:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 18:59:52 Testing delete operation on source bucket
2021/12/08 18:59:52 Deploying app: TestAdvancedDeleteOps. Item count src bucket: 0 dst bucket: 0
2021/12/08 18:59:53 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedDeleteOps' in metakv",
  "warnings": null
 }
}
2021/12/08 18:59:58 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/08 19:00:03 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/08 19:00:08 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/08 19:00:13 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/08 19:00:13 App: TestAdvancedDeleteOps got deployed
2021/12/08 19:00:13 Starting bucket ops to source bucket
2021/12/08 19:00:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 19:00:14 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 19:00:14 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/08 19:00:14 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 19:00:19 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/08 19:00:24 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/08 19:00:29 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/08 19:00:34 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/08 19:00:34 App: TestAdvancedDeleteOps got un-deployed
2021/12/08 19:00:34 No eventing-con process running
2021/12/08 19:00:34 Function: TestAdvancedDeleteOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedDeleteOps deleting in the background"
}
2021/12/08 19:00:35 Context: Bucket flush, response code: 200 dump: 
2021/12/08 19:00:40 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 19:00:43 Context: Bucket flush, response code: 200 dump: 
2021/12/08 19:00:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedDeleteOps (125.73s)
=== RUN   TestEnoentAdvancedGet
2021/12/08 19:00:48 Deploying app: TestEnoentAdvancedGet. Item count src bucket: 0 dst bucket: 0
2021/12/08 19:00:49 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedGet' in metakv",
  "warnings": null
 }
}
2021/12/08 19:00:54 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/08 19:00:59 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/08 19:01:04 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/08 19:01:09 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/08 19:01:09 App: TestEnoentAdvancedGet got deployed
2021/12/08 19:01:09 Starting bucket ops to source bucket
2021/12/08 19:01:14 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/08 19:01:19 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/08 19:01:19 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/08 19:01:19 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 19:01:19 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/08 19:01:19 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 19:01:24 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/08 19:01:29 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/08 19:01:34 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/08 19:01:39 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/08 19:01:39 App: TestEnoentAdvancedGet got un-deployed
2021/12/08 19:01:39 No eventing-con process running
2021/12/08 19:01:39 Function: TestEnoentAdvancedGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedGet deleting in the background"
}
2021/12/08 19:01:41 Context: Bucket flush, response code: 200 dump: 
2021/12/08 19:01:51 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/08 19:01:56 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/08 19:01:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 19:01:58 Context: Bucket flush, response code: 200 dump: 
2021/12/08 19:02:08 Waiting for dst bucket item count to get to: 0 curr count: 61
2021/12/08 19:02:13 Waiting for dst bucket item count to get to: 0 curr count: 11
2021/12/08 19:02:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 19:02:13 Testing get enoent operation on source bucket
2021/12/08 19:02:13 Deploying app: TestEnoentAdvancedGet. Item count src bucket: 0 dst bucket: 0
2021/12/08 19:02:14 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedGet' in metakv",
  "warnings": null
 }
}
2021/12/08 19:02:19 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/08 19:02:24 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/08 19:02:29 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/08 19:02:34 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/08 19:02:34 App: TestEnoentAdvancedGet got deployed
2021/12/08 19:02:34 Starting bucket ops to source bucket
2021/12/08 19:02:39 Waiting for dst bucket item count to get to: 200 curr count: 0
2021/12/08 19:02:44 Waiting for dst bucket item count to get to: 200 curr count: 100
2021/12/08 19:02:44 src & dst bucket item count matched up. src bucket count: 200 dst bucket count: 200
2021/12/08 19:02:44 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 19:02:44 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/08 19:02:44 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 19:02:49 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/08 19:02:54 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/08 19:02:59 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/08 19:03:04 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/08 19:03:04 App: TestEnoentAdvancedGet got un-deployed
2021/12/08 19:03:04 No eventing-con process running
2021/12/08 19:03:04 Function: TestEnoentAdvancedGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedGet deleting in the background"
}
2021/12/08 19:03:06 Context: Bucket flush, response code: 200 dump: 
2021/12/08 19:03:16 Waiting for dst bucket item count to get to: 0 curr count: 200
2021/12/08 19:03:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 19:03:18 Context: Bucket flush, response code: 200 dump: 
2021/12/08 19:03:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEnoentAdvancedGet (155.03s)
=== RUN   TestEnoentAdvancedReplace
2021/12/08 19:03:23 Deploying app: TestEnoentAdvancedReplace. Item count src bucket: 0 dst bucket: 0
2021/12/08 19:03:24 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedReplace' in metakv",
  "warnings": null
 }
}
2021/12/08 19:03:29 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/08 19:03:34 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/08 19:03:39 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/08 19:03:44 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/08 19:03:44 App: TestEnoentAdvancedReplace got deployed
2021/12/08 19:03:44 Starting bucket ops to source bucket
2021/12/08 19:03:49 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/08 19:03:54 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/08 19:03:54 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/08 19:03:54 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 19:03:54 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/08 19:03:54 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 19:03:59 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/08 19:04:04 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/08 19:04:09 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/08 19:04:14 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/08 19:04:14 App: TestEnoentAdvancedReplace got un-deployed
2021/12/08 19:04:14 No eventing-con process running
2021/12/08 19:04:14 Function: TestEnoentAdvancedReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedReplace deleting in the background"
}
2021/12/08 19:04:16 Context: Bucket flush, response code: 200 dump: 
2021/12/08 19:04:26 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/08 19:04:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 19:04:28 Context: Bucket flush, response code: 200 dump: 
2021/12/08 19:04:38 Waiting for dst bucket item count to get to: 0 curr count: 60
2021/12/08 19:04:43 Waiting for dst bucket item count to get to: 0 curr count: 10
2021/12/08 19:04:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 19:04:43 Testing get enoent operation on source bucket
2021/12/08 19:04:43 Deploying app: TestEnoentAdvancedReplace. Item count src bucket: 0 dst bucket: 0
2021/12/08 19:04:44 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedReplace' in metakv",
  "warnings": null
 }
}
2021/12/08 19:04:49 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/08 19:04:54 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/08 19:04:59 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/08 19:05:04 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/08 19:05:04 App: TestEnoentAdvancedReplace got deployed
2021/12/08 19:05:04 Starting bucket ops to source bucket
2021/12/08 19:05:09 Waiting for dst bucket item count to get to: 200 curr count: 0
2021/12/08 19:05:14 Waiting for dst bucket item count to get to: 200 curr count: 100
2021/12/08 19:05:14 src & dst bucket item count matched up. src bucket count: 200 dst bucket count: 200
2021/12/08 19:05:14 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 19:05:14 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/08 19:05:14 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 19:05:19 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/08 19:05:24 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/08 19:05:29 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/08 19:05:34 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/08 19:05:34 App: TestEnoentAdvancedReplace got un-deployed
2021/12/08 19:05:34 No eventing-con process running
2021/12/08 19:05:34 Function: TestEnoentAdvancedReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedReplace deleting in the background"
}
2021/12/08 19:05:36 Context: Bucket flush, response code: 200 dump: 
2021/12/08 19:05:46 Waiting for dst bucket item count to get to: 0 curr count: 200
2021/12/08 19:05:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 19:05:48 Context: Bucket flush, response code: 200 dump: 
2021/12/08 19:05:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEnoentAdvancedReplace (150.07s)
=== RUN   TestEnoentAdvancedDelete
2021/12/08 19:05:53 Deploying app: TestEnoentAdvancedDelete. Item count src bucket: 0 dst bucket: 0
2021/12/08 19:05:54 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedDelete' in metakv",
  "warnings": null
 }
}
2021/12/08 19:05:59 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/08 19:06:04 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/08 19:06:09 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/08 19:06:14 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/08 19:06:14 App: TestEnoentAdvancedDelete got deployed
2021/12/08 19:06:14 Starting bucket ops to source bucket
2021/12/08 19:06:19 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/08 19:06:24 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/08 19:06:24 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/08 19:06:24 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 19:06:24 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/08 19:06:24 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 19:06:29 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/08 19:06:34 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/08 19:06:39 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/08 19:06:44 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/08 19:06:44 App: TestEnoentAdvancedDelete got un-deployed
2021/12/08 19:06:44 No eventing-con process running
2021/12/08 19:06:44 Function: TestEnoentAdvancedDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedDelete deleting in the background"
}
2021/12/08 19:06:46 Context: Bucket flush, response code: 200 dump: 
2021/12/08 19:06:56 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/08 19:06:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 19:06:58 Context: Bucket flush, response code: 200 dump: 
2021/12/08 19:07:08 Waiting for dst bucket item count to get to: 0 curr count: 64
2021/12/08 19:07:13 Waiting for dst bucket item count to get to: 0 curr count: 14
2021/12/08 19:07:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 19:07:13 Testing get enoent operation on source bucket
2021/12/08 19:07:13 Deploying app: TestEnoentAdvancedDelete. Item count src bucket: 0 dst bucket: 0
2021/12/08 19:07:14 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedDelete' in metakv",
  "warnings": null
 }
}
2021/12/08 19:07:19 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/08 19:07:24 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/08 19:07:29 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/08 19:07:34 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/08 19:07:34 App: TestEnoentAdvancedDelete got deployed
2021/12/08 19:07:34 Starting bucket ops to source bucket
2021/12/08 19:07:39 Waiting for dst bucket item count to get to: 200 curr count: 0
2021/12/08 19:07:44 Waiting for dst bucket item count to get to: 200 curr count: 100
2021/12/08 19:07:44 src & dst bucket item count matched up. src bucket count: 200 dst bucket count: 200
2021/12/08 19:07:44 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 19:07:44 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/08 19:07:44 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 19:07:49 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/08 19:07:54 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/08 19:07:59 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/08 19:08:04 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/08 19:08:04 App: TestEnoentAdvancedDelete got un-deployed
2021/12/08 19:08:04 No eventing-con process running
2021/12/08 19:08:04 Function: TestEnoentAdvancedDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedDelete deleting in the background"
}
2021/12/08 19:08:06 Context: Bucket flush, response code: 200 dump: 
2021/12/08 19:08:16 Waiting for dst bucket item count to get to: 0 curr count: 200
2021/12/08 19:08:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 19:08:18 Context: Bucket flush, response code: 200 dump: 
2021/12/08 19:08:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEnoentAdvancedDelete (150.02s)
=== RUN   TestAdvancedInsertKeyExist
2021/12/08 19:08:23 Deploying app: TestAdvancedInsertKeyExist. Item count src bucket: 0 dst bucket: 0
2021/12/08 19:08:24 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedInsertKeyExist' in metakv",
  "warnings": null
 }
}
2021/12/08 19:08:29 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/08 19:08:34 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/08 19:08:39 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/08 19:08:44 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/08 19:08:44 App: TestAdvancedInsertKeyExist got deployed
2021/12/08 19:08:44 Starting bucket ops to source bucket
2021/12/08 19:08:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 19:08:44 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 19:08:44 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/08 19:08:44 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 19:08:49 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/08 19:08:54 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/08 19:08:59 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/08 19:09:04 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/08 19:09:04 App: TestAdvancedInsertKeyExist got un-deployed
2021/12/08 19:09:04 No eventing-con process running
2021/12/08 19:09:04 Function: TestAdvancedInsertKeyExist delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedInsertKeyExist deleting in the background"
}
2021/12/08 19:09:06 Context: Bucket flush, response code: 200 dump: 
2021/12/08 19:09:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 19:09:14 Context: Bucket flush, response code: 200 dump: 
2021/12/08 19:09:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 19:09:19 Pumping document in the destination bucket
2021/12/08 19:09:19 Starting bucket ops to source bucket
2021/12/08 19:09:19 Deploying app: TestAdvancedInsertKeyExist. Item count src bucket: 0 dst bucket: 0
2021/12/08 19:09:20 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedInsertKeyExist' in metakv",
  "warnings": null
 }
}
2021/12/08 19:09:25 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/08 19:09:30 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/08 19:09:35 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/08 19:09:40 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/08 19:09:40 App: TestAdvancedInsertKeyExist got deployed
2021/12/08 19:09:40 Starting bucket ops to source bucket
2021/12/08 19:09:45 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/08 19:09:50 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/08 19:09:55 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/08 19:09:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 19:09:55 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 19:09:55 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/08 19:09:55 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 19:10:00 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/08 19:10:05 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/08 19:10:10 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/08 19:10:15 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/08 19:10:15 App: TestAdvancedInsertKeyExist got un-deployed
2021/12/08 19:10:15 No eventing-con process running
2021/12/08 19:10:15 Function: TestAdvancedInsertKeyExist delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedInsertKeyExist deleting in the background"
}
2021/12/08 19:10:17 Context: Bucket flush, response code: 200 dump: 
2021/12/08 19:10:27 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/08 19:10:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 19:10:29 Context: Bucket flush, response code: 200 dump: 
2021/12/08 19:10:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedInsertKeyExist (130.74s)
=== RUN   TestExipryGet
2021/12/08 19:10:34 Deploying app: TestExipryGet. Item count src bucket: 0 dst bucket: 0
2021/12/08 19:10:35 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryGet' in metakv",
  "warnings": null
 }
}
2021/12/08 19:10:40 Waiting for app: TestExipryGet to get deployed
2021/12/08 19:10:45 Waiting for app: TestExipryGet to get deployed
2021/12/08 19:10:50 Waiting for app: TestExipryGet to get deployed
2021/12/08 19:10:55 Waiting for app: TestExipryGet to get deployed
2021/12/08 19:10:55 App: TestExipryGet got deployed
2021/12/08 19:10:55 Starting bucket ops to source bucket
2021/12/08 19:11:40 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/08 19:11:40 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 19:11:40 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 19:11:40 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/08 19:11:40 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 19:11:45 Waiting for app: TestExipryGet to get un-deployed
2021/12/08 19:11:50 Waiting for app: TestExipryGet to get un-deployed
2021/12/08 19:11:55 Waiting for app: TestExipryGet to get un-deployed
2021/12/08 19:12:00 Waiting for app: TestExipryGet to get un-deployed
2021/12/08 19:12:00 App: TestExipryGet got un-deployed
2021/12/08 19:12:00 No eventing-con process running
2021/12/08 19:12:00 Function: TestExipryGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryGet deleting in the background"
}
2021/12/08 19:12:02 Context: Bucket flush, response code: 200 dump: 
2021/12/08 19:12:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 19:12:09 Context: Bucket flush, response code: 200 dump: 
2021/12/08 19:12:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 19:12:14 Pumping document in the destination bucket for expiry get
2021/12/08 19:12:14 Starting bucket ops to source bucket
2021/12/08 19:12:14 Deploying app: TestExipryGet. Item count src bucket: 0 dst bucket: 0
2021/12/08 19:12:16 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryGet' in metakv",
  "warnings": null
 }
}
2021/12/08 19:12:21 Waiting for app: TestExipryGet to get deployed
2021/12/08 19:12:26 Waiting for app: TestExipryGet to get deployed
2021/12/08 19:12:31 Waiting for app: TestExipryGet to get deployed
2021/12/08 19:12:36 Waiting for app: TestExipryGet to get deployed
2021/12/08 19:12:36 App: TestExipryGet got deployed
2021/12/08 19:12:36 Starting bucket ops to source bucket
2021/12/08 19:13:21 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/08 19:13:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 19:13:21 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 19:13:21 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/08 19:13:21 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 19:13:26 Waiting for app: TestExipryGet to get un-deployed
2021/12/08 19:13:31 Waiting for app: TestExipryGet to get un-deployed
2021/12/08 19:13:36 Waiting for app: TestExipryGet to get un-deployed
2021/12/08 19:13:41 Waiting for app: TestExipryGet to get un-deployed
2021/12/08 19:13:41 App: TestExipryGet got un-deployed
2021/12/08 19:13:41 No eventing-con process running
2021/12/08 19:13:41 Function: TestExipryGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryGet deleting in the background"
}
2021/12/08 19:13:43 Context: Bucket flush, response code: 200 dump: 
2021/12/08 19:13:53 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/08 19:13:58 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/08 19:13:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 19:13:59 Context: Bucket flush, response code: 200 dump: 
2021/12/08 19:14:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestExipryGet (210.79s)
=== RUN   TestExipryInsert
2021/12/08 19:14:04 Deploying app: TestExipryInsert. Item count src bucket: 0 dst bucket: 0
2021/12/08 19:14:06 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryInsert' in metakv",
  "warnings": null
 }
}
2021/12/08 19:14:11 Waiting for app: TestExipryInsert to get deployed
2021/12/08 19:14:16 Waiting for app: TestExipryInsert to get deployed
2021/12/08 19:14:21 Waiting for app: TestExipryInsert to get deployed
2021/12/08 19:14:26 Waiting for app: TestExipryInsert to get deployed
2021/12/08 19:14:26 App: TestExipryInsert got deployed
2021/12/08 19:14:26 Starting bucket ops to source bucket
2021/12/08 19:15:11 Waiting for dst bucket item count to get to: 0 curr count: 200
2021/12/08 19:15:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 19:15:11 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 19:15:11 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/08 19:15:11 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 19:15:16 Waiting for app: TestExipryInsert to get un-deployed
2021/12/08 19:15:21 Waiting for app: TestExipryInsert to get un-deployed
2021/12/08 19:15:26 Waiting for app: TestExipryInsert to get un-deployed
2021/12/08 19:15:31 Waiting for app: TestExipryInsert to get un-deployed
2021/12/08 19:15:31 App: TestExipryInsert got un-deployed
2021/12/08 19:15:31 No eventing-con process running
2021/12/08 19:15:31 Function: TestExipryInsert delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryInsert deleting in the background"
}
2021/12/08 19:15:33 Context: Bucket flush, response code: 200 dump: 
2021/12/08 19:15:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 19:15:40 Context: Bucket flush, response code: 200 dump: 
2021/12/08 19:15:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 19:15:45 Testing on the destination bucket for expiry insert
2021/12/08 19:15:45 Deploying app: TestExipryInsert. Item count src bucket: 0 dst bucket: 0
2021/12/08 19:15:46 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryInsert' in metakv",
  "warnings": null
 }
}
2021/12/08 19:15:51 Waiting for app: TestExipryInsert to get deployed
2021/12/08 19:15:56 Waiting for app: TestExipryInsert to get deployed
2021/12/08 19:16:01 Waiting for app: TestExipryInsert to get deployed
2021/12/08 19:16:06 Waiting for app: TestExipryInsert to get deployed
2021/12/08 19:16:06 App: TestExipryInsert got deployed
2021/12/08 19:16:06 Starting bucket ops to source bucket
2021/12/08 19:16:52 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/08 19:16:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 19:16:52 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 19:16:52 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/08 19:16:52 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 19:16:57 Waiting for app: TestExipryInsert to get un-deployed
2021/12/08 19:17:02 Waiting for app: TestExipryInsert to get un-deployed
2021/12/08 19:17:07 Waiting for app: TestExipryInsert to get un-deployed
2021/12/08 19:17:12 Waiting for app: TestExipryInsert to get un-deployed
2021/12/08 19:17:12 App: TestExipryInsert got un-deployed
2021/12/08 19:17:12 No eventing-con process running
2021/12/08 19:17:12 Function: TestExipryInsert delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryInsert deleting in the background"
}
2021/12/08 19:17:13 Context: Bucket flush, response code: 200 dump: 
2021/12/08 19:17:23 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/08 19:17:28 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/08 19:17:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 19:17:30 Context: Bucket flush, response code: 200 dump: 
2021/12/08 19:17:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestExipryInsert (210.73s)
=== RUN   TestExipryReplace
2021/12/08 19:17:35 Deploying app: TestExipryReplace. Item count src bucket: 0 dst bucket: 0
2021/12/08 19:17:36 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryReplace' in metakv",
  "warnings": null
 }
}
2021/12/08 19:17:41 Waiting for app: TestExipryReplace to get deployed
2021/12/08 19:17:46 Waiting for app: TestExipryReplace to get deployed
2021/12/08 19:17:51 Waiting for app: TestExipryReplace to get deployed
2021/12/08 19:17:56 Waiting for app: TestExipryReplace to get deployed
2021/12/08 19:17:56 App: TestExipryReplace got deployed
2021/12/08 19:17:56 Starting bucket ops to source bucket
2021/12/08 19:18:42 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/08 19:18:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 19:18:42 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 19:18:42 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/08 19:18:42 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 19:18:47 Waiting for app: TestExipryReplace to get un-deployed
2021/12/08 19:18:52 Waiting for app: TestExipryReplace to get un-deployed
2021/12/08 19:18:57 Waiting for app: TestExipryReplace to get un-deployed
2021/12/08 19:19:02 Waiting for app: TestExipryReplace to get un-deployed
2021/12/08 19:19:02 App: TestExipryReplace got un-deployed
2021/12/08 19:19:02 No eventing-con process running
2021/12/08 19:19:02 Function: TestExipryReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryReplace deleting in the background"
}
2021/12/08 19:19:03 Context: Bucket flush, response code: 200 dump: 
2021/12/08 19:19:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 19:19:11 Context: Bucket flush, response code: 200 dump: 
2021/12/08 19:19:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 19:19:16 Testing on the destination bucket for expiry insert
2021/12/08 19:19:16 Deploying app: TestExipryReplace. Item count src bucket: 0 dst bucket: 0
2021/12/08 19:19:17 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryReplace' in metakv",
  "warnings": null
 }
}
2021/12/08 19:19:22 Waiting for app: TestExipryReplace to get deployed
2021/12/08 19:19:27 Waiting for app: TestExipryReplace to get deployed
2021/12/08 19:19:32 Waiting for app: TestExipryReplace to get deployed
2021/12/08 19:19:37 Waiting for app: TestExipryReplace to get deployed
2021/12/08 19:19:37 App: TestExipryReplace got deployed
2021/12/08 19:19:37 Starting bucket ops to source bucket
2021/12/08 19:20:22 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/08 19:20:27 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/08 19:20:32 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/08 19:20:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 19:20:32 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 19:20:32 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/08 19:20:32 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 19:20:37 Waiting for app: TestExipryReplace to get un-deployed
2021/12/08 19:20:42 Waiting for app: TestExipryReplace to get un-deployed
2021/12/08 19:20:47 Waiting for app: TestExipryReplace to get un-deployed
2021/12/08 19:20:52 Waiting for app: TestExipryReplace to get un-deployed
2021/12/08 19:20:52 App: TestExipryReplace got un-deployed
2021/12/08 19:20:52 No eventing-con process running
2021/12/08 19:20:52 Function: TestExipryReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryReplace deleting in the background"
}
2021/12/08 19:20:54 Context: Bucket flush, response code: 200 dump: 
2021/12/08 19:21:04 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/08 19:21:09 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/08 19:21:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 19:21:11 Context: Bucket flush, response code: 200 dump: 
2021/12/08 19:21:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestExipryReplace (220.75s)
=== RUN   TestExipryUpsert
2021/12/08 19:21:16 Deploying app: TestExipryUpsert. Item count src bucket: 0 dst bucket: 0
2021/12/08 19:21:17 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryUpsert' in metakv",
  "warnings": null
 }
}
2021/12/08 19:21:22 Waiting for app: TestExipryUpsert to get deployed
2021/12/08 19:21:27 Waiting for app: TestExipryUpsert to get deployed
2021/12/08 19:21:32 Waiting for app: TestExipryUpsert to get deployed
2021/12/08 19:21:37 Waiting for app: TestExipryUpsert to get deployed
2021/12/08 19:21:37 App: TestExipryUpsert got deployed
2021/12/08 19:21:37 Starting bucket ops to source bucket
2021/12/08 19:22:22 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/08 19:22:27 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/08 19:22:32 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/08 19:22:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 19:22:32 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 19:22:32 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/08 19:22:32 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 19:22:37 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/08 19:22:42 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/08 19:22:47 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/08 19:22:52 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/08 19:22:52 App: TestExipryUpsert got un-deployed
2021/12/08 19:22:52 No eventing-con process running
2021/12/08 19:22:53 Function: TestExipryUpsert delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryUpsert deleting in the background"
}
2021/12/08 19:22:54 Context: Bucket flush, response code: 200 dump: 
2021/12/08 19:22:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 19:23:02 Context: Bucket flush, response code: 200 dump: 
2021/12/08 19:23:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 19:23:07 Testing on the destination bucket for expiry insert
2021/12/08 19:23:07 Deploying app: TestExipryUpsert. Item count src bucket: 0 dst bucket: 0
2021/12/08 19:23:08 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryUpsert' in metakv",
  "warnings": null
 }
}
2021/12/08 19:23:13 Waiting for app: TestExipryUpsert to get deployed
2021/12/08 19:23:18 Waiting for app: TestExipryUpsert to get deployed
2021/12/08 19:23:23 Waiting for app: TestExipryUpsert to get deployed
2021/12/08 19:23:28 Waiting for app: TestExipryUpsert to get deployed
2021/12/08 19:23:28 App: TestExipryUpsert got deployed
2021/12/08 19:23:28 Starting bucket ops to source bucket
2021/12/08 19:24:13 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/08 19:24:18 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/08 19:24:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 19:24:18 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 19:24:18 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/08 19:24:18 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 19:24:23 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/08 19:24:28 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/08 19:24:33 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/08 19:24:38 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/08 19:24:38 App: TestExipryUpsert got un-deployed
2021/12/08 19:24:38 No eventing-con process running
2021/12/08 19:24:38 Function: TestExipryUpsert delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryUpsert deleting in the background"
}
2021/12/08 19:24:40 Context: Bucket flush, response code: 200 dump: 
2021/12/08 19:24:50 Waiting for dst bucket item count to get to: 0 curr count: 54
2021/12/08 19:24:55 Waiting for dst bucket item count to get to: 0 curr count: 54
2021/12/08 19:24:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 19:24:57 Context: Bucket flush, response code: 200 dump: 
2021/12/08 19:25:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestExipryUpsert (225.66s)
=== RUN   TestCasReplace
2021/12/08 19:25:02 Deploying app: TestCasReplace. Item count src bucket: 0 dst bucket: 0
2021/12/08 19:25:03 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCasReplace' in metakv",
  "warnings": null
 }
}
2021/12/08 19:25:08 Waiting for app: TestCasReplace to get deployed
2021/12/08 19:25:13 Waiting for app: TestCasReplace to get deployed
2021/12/08 19:25:18 Waiting for app: TestCasReplace to get deployed
2021/12/08 19:25:23 Waiting for app: TestCasReplace to get deployed
2021/12/08 19:25:23 App: TestCasReplace got deployed
2021/12/08 19:25:23 Starting bucket ops to source bucket
2021/12/08 19:25:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 19:25:23 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 19:25:23 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/08 19:25:23 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 19:25:28 Waiting for app: TestCasReplace to get un-deployed
2021/12/08 19:25:33 Waiting for app: TestCasReplace to get un-deployed
2021/12/08 19:25:38 Waiting for app: TestCasReplace to get un-deployed
2021/12/08 19:25:43 Waiting for app: TestCasReplace to get un-deployed
2021/12/08 19:25:43 App: TestCasReplace got un-deployed
2021/12/08 19:25:43 No eventing-con process running
2021/12/08 19:25:43 Function: TestCasReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCasReplace deleting in the background"
}
2021/12/08 19:25:45 Context: Bucket flush, response code: 200 dump: 
2021/12/08 19:25:55 Waiting for dst bucket item count to get to: 0 curr count: 17
2021/12/08 19:25:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 19:25:57 Context: Bucket flush, response code: 200 dump: 
2021/12/08 19:26:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 19:26:02 Testing on the destination bucket for upsert with cas
2021/12/08 19:26:02 Starting bucket ops to source bucket
2021/12/08 19:26:02 Deploying app: TestCasReplace. Item count src bucket: 0 dst bucket: 0
2021/12/08 19:26:03 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCasReplace' in metakv",
  "warnings": null
 }
}
2021/12/08 19:26:08 Waiting for app: TestCasReplace to get deployed
2021/12/08 19:26:13 Waiting for app: TestCasReplace to get deployed
2021/12/08 19:26:18 Waiting for app: TestCasReplace to get deployed
2021/12/08 19:26:23 Waiting for app: TestCasReplace to get deployed
2021/12/08 19:26:23 App: TestCasReplace got deployed
2021/12/08 19:26:23 Starting bucket ops to source bucket
2021/12/08 19:26:28 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/08 19:26:33 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/08 19:26:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 19:26:33 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 19:26:33 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/08 19:26:33 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 19:26:38 Waiting for app: TestCasReplace to get un-deployed
2021/12/08 19:26:43 Waiting for app: TestCasReplace to get un-deployed
2021/12/08 19:26:48 Waiting for app: TestCasReplace to get un-deployed
2021/12/08 19:26:53 Waiting for app: TestCasReplace to get un-deployed
2021/12/08 19:26:53 App: TestCasReplace got un-deployed
2021/12/08 19:26:53 No eventing-con process running
2021/12/08 19:26:53 Function: TestCasReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCasReplace deleting in the background"
}
2021/12/08 19:26:55 Context: Bucket flush, response code: 200 dump: 
2021/12/08 19:27:05 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/08 19:27:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 19:27:07 Context: Bucket flush, response code: 200 dump: 
2021/12/08 19:27:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCasReplace (130.09s)
=== RUN   TestCasDelete
2021/12/08 19:27:12 Deploying app: TestCasDelete. Item count src bucket: 0 dst bucket: 0
2021/12/08 19:27:13 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCasDelete' in metakv",
  "warnings": null
 }
}
2021/12/08 19:27:18 Waiting for app: TestCasDelete to get deployed
2021/12/08 19:27:23 Waiting for app: TestCasDelete to get deployed
2021/12/08 19:27:28 Waiting for app: TestCasDelete to get deployed
2021/12/08 19:27:33 Waiting for app: TestCasDelete to get deployed
2021/12/08 19:27:33 App: TestCasDelete got deployed
2021/12/08 19:27:33 Starting bucket ops to source bucket
2021/12/08 19:27:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 19:27:33 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 19:27:33 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/08 19:27:33 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 19:27:38 Waiting for app: TestCasDelete to get un-deployed
2021/12/08 19:27:43 Waiting for app: TestCasDelete to get un-deployed
2021/12/08 19:27:48 Waiting for app: TestCasDelete to get un-deployed
2021/12/08 19:27:53 Waiting for app: TestCasDelete to get un-deployed
2021/12/08 19:27:53 App: TestCasDelete got un-deployed
2021/12/08 19:27:53 No eventing-con process running
2021/12/08 19:27:53 Function: TestCasDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCasDelete deleting in the background"
}
2021/12/08 19:27:55 Context: Bucket flush, response code: 200 dump: 
2021/12/08 19:28:05 Waiting for dst bucket item count to get to: 0 curr count: 2
2021/12/08 19:28:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 19:28:07 Context: Bucket flush, response code: 200 dump: 
2021/12/08 19:28:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 19:28:12 Testing on the destination bucket for delete with cas
2021/12/08 19:28:12 Starting bucket ops to source bucket
2021/12/08 19:28:12 Deploying app: TestCasDelete. Item count src bucket: 0 dst bucket: 0
2021/12/08 19:28:13 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCasDelete' in metakv",
  "warnings": null
 }
}
2021/12/08 19:28:18 Waiting for app: TestCasDelete to get deployed
2021/12/08 19:28:23 Waiting for app: TestCasDelete to get deployed
2021/12/08 19:28:28 Waiting for app: TestCasDelete to get deployed
2021/12/08 19:28:33 Waiting for app: TestCasDelete to get deployed
2021/12/08 19:28:33 App: TestCasDelete got deployed
2021/12/08 19:28:33 Starting bucket ops to source bucket
2021/12/08 19:28:38 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/08 19:28:43 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/08 19:28:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 19:28:43 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 19:28:43 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/08 19:28:43 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 19:28:48 Waiting for app: TestCasDelete to get un-deployed
2021/12/08 19:28:53 Waiting for app: TestCasDelete to get un-deployed
2021/12/08 19:28:58 Waiting for app: TestCasDelete to get un-deployed
2021/12/08 19:29:03 Waiting for app: TestCasDelete to get un-deployed
2021/12/08 19:29:03 App: TestCasDelete got un-deployed
2021/12/08 19:29:03 No eventing-con process running
2021/12/08 19:29:03 Function: TestCasDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCasDelete deleting in the background"
}
2021/12/08 19:29:05 Context: Bucket flush, response code: 200 dump: 
2021/12/08 19:29:15 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/08 19:29:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 19:29:17 Context: Bucket flush, response code: 200 dump: 
2021/12/08 19:29:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCasDelete (130.13s)
=== RUN   TestCountersIncrement
2021/12/08 19:29:22 Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/08 19:29:27 Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/08 19:29:28 Removing node(s): [] from the cluster
2021/12/08 19:29:33 Rebalance progress: 100
2021/12/08 19:29:33 Deploying app: TestCountersIncrement. Item count src bucket: 0 dst bucket: 0
2021/12/08 19:29:34 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCountersIncrement' in metakv",
  "warnings": null
 }
}
2021/12/08 19:29:39 Waiting for app: TestCountersIncrement to get deployed
2021/12/08 19:29:44 Waiting for app: TestCountersIncrement to get deployed
2021/12/08 19:29:49 Waiting for app: TestCountersIncrement to get deployed
2021/12/08 19:29:54 Waiting for app: TestCountersIncrement to get deployed
2021/12/08 19:29:54 App: TestCountersIncrement got deployed
2021/12/08 19:29:54 Starting bucket ops to source bucket
2021/12/08 19:30:01 Waiting for dst bucket item count to get to: 1024 curr count: 0
2021/12/08 19:30:06 Waiting for dst bucket item count to get to: 1024 curr count: 512
2021/12/08 19:30:06 src & dst bucket item count matched up. src bucket count: 1024 dst bucket count: 1024
2021/12/08 19:30:06 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 19:30:06 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 19:30:11 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/08 19:30:16 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/08 19:30:21 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/08 19:30:26 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/08 19:30:26 App: TestCountersIncrement got un-deployed
2021/12/08 19:30:26 No eventing-con process running
2021/12/08 19:30:26 Function: TestCountersIncrement delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCountersIncrement deleting in the background"
}
2021/12/08 19:30:27 Context: Bucket flush, response code: 200 dump: 
2021/12/08 19:30:38 Waiting for dst bucket item count to get to: 0 curr count: 513
2021/12/08 19:30:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 19:30:39 Context: Bucket flush, response code: 200 dump: 
2021/12/08 19:30:49 Waiting for dst bucket item count to get to: 0 curr count: 512
2021/12/08 19:30:54 Waiting for dst bucket item count to get to: 0 curr count: 512
2021/12/08 19:30:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 19:30:54 Deploying app: TestCountersIncrement. Item count src bucket: 0 dst bucket: 0
2021/12/08 19:30:56 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCountersIncrement' in metakv",
  "warnings": null
 }
}
2021/12/08 19:31:01 Waiting for app: TestCountersIncrement to get deployed
2021/12/08 19:31:06 Waiting for app: TestCountersIncrement to get deployed
2021/12/08 19:31:11 Waiting for app: TestCountersIncrement to get deployed
2021/12/08 19:31:16 Waiting for app: TestCountersIncrement to get deployed
2021/12/08 19:31:16 App: TestCountersIncrement got deployed
2021/12/08 19:31:16 Starting bucket ops to source bucket
2021/12/08 19:31:22 Waiting for dst bucket item count to get to: 2048 curr count: 0
2021/12/08 19:31:22 src & dst bucket item count matched up. src bucket count: 2048 dst bucket count: 2048
2021/12/08 19:31:22 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 19:31:22 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 19:31:27 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/08 19:31:32 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/08 19:31:37 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/08 19:31:42 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/08 19:31:42 App: TestCountersIncrement got un-deployed
2021/12/08 19:31:42 No eventing-con process running
2021/12/08 19:31:42 Function: TestCountersIncrement delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCountersIncrement deleting in the background"
}
2021/12/08 19:31:44 Context: Bucket flush, response code: 200 dump: 
2021/12/08 19:31:54 Waiting for dst bucket item count to get to: 0 curr count: 2048
2021/12/08 19:31:59 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/08 19:31:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 19:32:00 Context: Bucket flush, response code: 200 dump: 
2021/12/08 19:32:10 Waiting for dst bucket item count to get to: 0 curr count: 1
2021/12/08 19:32:15 Waiting for dst bucket item count to get to: 0 curr count: 1
2021/12/08 19:32:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 19:32:15 Removing node(s): [http://127.0.0.1:19002 http://127.0.0.1:19003] from the cluster
2021/12/08 19:32:21 Rebalance progress: 100
--- PASS: TestCountersIncrement (178.75s)
=== RUN   TestCountersDecrement
2021/12/08 19:32:21 Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/08 19:32:21 Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/08 19:32:21 Removing node(s): [] from the cluster
2021/12/08 19:32:26 Rebalance progress: 100
2021/12/08 19:32:26 Deploying app: TestCountersDecrement. Item count src bucket: 0 dst bucket: 0
2021/12/08 19:32:27 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCountersDecrement' in metakv",
  "warnings": null
 }
}
2021/12/08 19:32:32 Waiting for app: TestCountersDecrement to get deployed
2021/12/08 19:32:37 Waiting for app: TestCountersDecrement to get deployed
2021/12/08 19:32:42 Waiting for app: TestCountersDecrement to get deployed
2021/12/08 19:32:47 Waiting for app: TestCountersDecrement to get deployed
2021/12/08 19:32:47 App: TestCountersDecrement got deployed
2021/12/08 19:32:47 Starting bucket ops to source bucket
2021/12/08 19:32:54 Waiting for dst bucket item count to get to: 1024 curr count: 0
2021/12/08 19:32:59 Waiting for dst bucket item count to get to: 1024 curr count: 512
2021/12/08 19:33:04 Waiting for dst bucket item count to get to: 1024 curr count: 512
2021/12/08 19:33:04 src & dst bucket item count matched up. src bucket count: 1024 dst bucket count: 1024
2021/12/08 19:33:04 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 19:33:04 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 19:33:09 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/08 19:33:14 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/08 19:33:19 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/08 19:33:24 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/08 19:33:24 App: TestCountersDecrement got un-deployed
2021/12/08 19:33:24 No eventing-con process running
2021/12/08 19:33:24 Function: TestCountersDecrement delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCountersDecrement deleting in the background"
}
2021/12/08 19:33:26 Context: Bucket flush, response code: 200 dump: 
2021/12/08 19:33:36 Waiting for dst bucket item count to get to: 0 curr count: 1025
2021/12/08 19:33:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 19:33:37 Context: Bucket flush, response code: 200 dump: 
2021/12/08 19:33:47 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/08 19:33:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 19:33:47 Deploying app: TestCountersDecrement. Item count src bucket: 0 dst bucket: 0
2021/12/08 19:33:49 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCountersDecrement' in metakv",
  "warnings": null
 }
}
2021/12/08 19:33:54 Waiting for app: TestCountersDecrement to get deployed
2021/12/08 19:33:59 Waiting for app: TestCountersDecrement to get deployed
2021/12/08 19:34:04 Waiting for app: TestCountersDecrement to get deployed
2021/12/08 19:34:09 Waiting for app: TestCountersDecrement to get deployed
2021/12/08 19:34:09 App: TestCountersDecrement got deployed
2021/12/08 19:34:09 Starting bucket ops to source bucket
2021/12/08 19:34:15 Waiting for dst bucket item count to get to: 2048 curr count: 0
2021/12/08 19:34:20 Waiting for dst bucket item count to get to: 2048 curr count: 563
2021/12/08 19:34:25 Waiting for dst bucket item count to get to: 2048 curr count: 1587
2021/12/08 19:34:25 src & dst bucket item count matched up. src bucket count: 2048 dst bucket count: 2048
2021/12/08 19:34:25 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 19:34:25 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 19:34:30 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/08 19:34:35 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/08 19:34:40 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/08 19:34:45 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/08 19:34:45 App: TestCountersDecrement got un-deployed
2021/12/08 19:34:45 No eventing-con process running
2021/12/08 19:34:45 Function: TestCountersDecrement delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCountersDecrement deleting in the background"
}
2021/12/08 19:34:47 Context: Bucket flush, response code: 200 dump: 
2021/12/08 19:34:57 Waiting for dst bucket item count to get to: 0 curr count: 2048
2021/12/08 19:34:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 19:34:59 Context: Bucket flush, response code: 200 dump: 
2021/12/08 19:35:09 Waiting for dst bucket item count to get to: 0 curr count: 1
2021/12/08 19:35:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 19:35:09 Removing node(s): [http://127.0.0.1:19002 http://127.0.0.1:19003] from the cluster
2021/12/08 19:35:14 Rebalance progress: 100
--- PASS: TestCountersDecrement (173.06s)
=== RUN   TestCacheGet
2021/12/08 19:35:14 Deploying app: TestCacheGet. Item count src bucket: 0 dst bucket: 0
2021/12/08 19:35:15 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCacheGet' in metakv",
  "warnings": null
 }
}
2021/12/08 19:35:20 Waiting for app: TestCacheGet to get deployed
2021/12/08 19:35:25 Waiting for app: TestCacheGet to get deployed
2021/12/08 19:35:30 Waiting for app: TestCacheGet to get deployed
2021/12/08 19:35:35 Waiting for app: TestCacheGet to get deployed
2021/12/08 19:35:35 App: TestCacheGet got deployed
2021/12/08 19:35:35 Starting bucket ops to source bucket
2021/12/08 19:35:40 Waiting for dst bucket item count to get to: 20 curr count: 0
2021/12/08 19:35:45 Waiting for dst bucket item count to get to: 20 curr count: 0
2021/12/08 19:35:50 Waiting for dst bucket item count to get to: 20 curr count: 0
2021/12/08 19:35:55 Waiting for dst bucket item count to get to: 20 curr count: 4
2021/12/08 19:36:00 Waiting for dst bucket item count to get to: 20 curr count: 12
2021/12/08 19:36:00 src & dst bucket item count matched up. src bucket count: 20 dst bucket count: 20
2021/12/08 19:36:00 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 19:36:00 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 19:36:05 Waiting for app: TestCacheGet to get un-deployed
2021/12/08 19:36:10 Waiting for app: TestCacheGet to get un-deployed
2021/12/08 19:36:15 Waiting for app: TestCacheGet to get un-deployed
2021/12/08 19:36:20 Waiting for app: TestCacheGet to get un-deployed
2021/12/08 19:36:20 App: TestCacheGet got un-deployed
2021/12/08 19:36:20 No eventing-con process running
2021/12/08 19:36:20 Function: TestCacheGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCacheGet deleting in the background"
}
2021/12/08 19:36:22 Context: Bucket flush, response code: 200 dump: 
2021/12/08 19:36:32 Waiting for dst bucket item count to get to: 0 curr count: 20
2021/12/08 19:36:37 Waiting for dst bucket item count to get to: 0 curr count: 20
2021/12/08 19:36:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 19:36:39 Context: Bucket flush, response code: 200 dump: 
2021/12/08 19:36:49 Waiting for dst bucket item count to get to: 0 curr count: 20
2021/12/08 19:36:54 Waiting for dst bucket item count to get to: 0 curr count: 10
2021/12/08 19:36:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCacheGet (100.17s)
=== RUN   TestCacheRYOW
2021/12/08 19:36:54 Deploying app: TestCacheRYOW. Item count src bucket: 0 dst bucket: 0
2021/12/08 19:36:55 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCacheRYOW' in metakv",
  "warnings": null
 }
}
2021/12/08 19:37:00 Waiting for app: TestCacheRYOW to get deployed
2021/12/08 19:37:05 Waiting for app: TestCacheRYOW to get deployed
2021/12/08 19:37:10 Waiting for app: TestCacheRYOW to get deployed
2021/12/08 19:37:15 Waiting for app: TestCacheRYOW to get deployed
2021/12/08 19:37:15 App: TestCacheRYOW got deployed
2021/12/08 19:37:15 Starting bucket ops to source bucket
2021/12/08 19:37:20 Waiting for dst bucket item count to get to: 50 curr count: 0
2021/12/08 19:37:25 Waiting for dst bucket item count to get to: 50 curr count: 25
2021/12/08 19:37:25 src & dst bucket item count matched up. src bucket count: 50 dst bucket count: 50
2021/12/08 19:37:25 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 19:37:25 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 19:37:30 Waiting for app: TestCacheRYOW to get un-deployed
2021/12/08 19:37:35 Waiting for app: TestCacheRYOW to get un-deployed
2021/12/08 19:37:40 Waiting for app: TestCacheRYOW to get un-deployed
2021/12/08 19:37:45 Waiting for app: TestCacheRYOW to get un-deployed
2021/12/08 19:37:45 App: TestCacheRYOW got un-deployed
2021/12/08 19:37:45 No eventing-con process running
2021/12/08 19:37:45 Function: TestCacheRYOW delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCacheRYOW deleting in the background"
}
2021/12/08 19:37:47 Context: Bucket flush, response code: 200 dump: 
2021/12/08 19:37:57 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/08 19:37:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 19:37:59 Context: Bucket flush, response code: 200 dump: 
2021/12/08 19:38:09 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/08 19:38:14 Waiting for dst bucket item count to get to: 0 curr count: 25
2021/12/08 19:38:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCacheRYOW (80.18s)
=== RUN   TestCacheExpiry
2021/12/08 19:38:14 Deploying app: TestCacheExpiry. Item count src bucket: 0 dst bucket: 0
2021/12/08 19:38:15 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCacheExpiry' in metakv",
  "warnings": null
 }
}
2021/12/08 19:38:20 Waiting for app: TestCacheExpiry to get deployed
2021/12/08 19:38:25 Waiting for app: TestCacheExpiry to get deployed
2021/12/08 19:38:30 Waiting for app: TestCacheExpiry to get deployed
2021/12/08 19:38:35 Waiting for app: TestCacheExpiry to get deployed
2021/12/08 19:38:35 App: TestCacheExpiry got deployed
2021/12/08 19:38:35 Starting bucket ops to source bucket
2021/12/08 19:38:40 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/08 19:38:45 Waiting for dst bucket item count to get to: 100 curr count: 19
2021/12/08 19:38:50 Waiting for dst bucket item count to get to: 100 curr count: 19
2021/12/08 19:38:55 Waiting for dst bucket item count to get to: 100 curr count: 94
2021/12/08 19:39:00 Waiting for dst bucket item count to get to: 100 curr count: 94
2021/12/08 19:39:00 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/08 19:39:00 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 19:39:01 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 19:39:06 Waiting for app: TestCacheExpiry to get un-deployed
2021/12/08 19:39:11 Waiting for app: TestCacheExpiry to get un-deployed
2021/12/08 19:39:16 Waiting for app: TestCacheExpiry to get un-deployed
2021/12/08 19:39:21 Waiting for app: TestCacheExpiry to get un-deployed
2021/12/08 19:39:21 App: TestCacheExpiry got un-deployed
2021/12/08 19:39:21 No eventing-con process running
2021/12/08 19:39:21 Function: TestCacheExpiry delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCacheExpiry deleting in the background"
}
2021/12/08 19:39:22 Context: Bucket flush, response code: 200 dump: 
2021/12/08 19:39:32 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/08 19:39:37 Waiting for dst bucket item count to get to: 0 curr count: 25
2021/12/08 19:39:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 19:39:39 Context: Bucket flush, response code: 200 dump: 
2021/12/08 19:39:49 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/08 19:39:54 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/08 19:39:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCacheExpiry (100.23s)
=== RUN   TestCacheExpiryCustom
2021/12/08 19:39:54 Deploying app: TestCacheExpiryCustom. Item count src bucket: 0 dst bucket: 0
2021/12/08 19:39:55 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCacheExpiryCustom' in metakv",
  "warnings": null
 }
}
2021/12/08 19:40:00 Waiting for app: TestCacheExpiryCustom to get deployed
2021/12/08 19:40:05 Waiting for app: TestCacheExpiryCustom to get deployed
2021/12/08 19:40:10 Waiting for app: TestCacheExpiryCustom to get deployed
2021/12/08 19:40:15 Waiting for app: TestCacheExpiryCustom to get deployed
2021/12/08 19:40:15 App: TestCacheExpiryCustom got deployed
2021/12/08 19:40:15 Starting bucket ops to source bucket
2021/12/08 19:40:21 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/08 19:40:26 Waiting for dst bucket item count to get to: 100 curr count: 19
2021/12/08 19:40:31 Waiting for dst bucket item count to get to: 100 curr count: 19
2021/12/08 19:40:36 Waiting for dst bucket item count to get to: 100 curr count: 94
2021/12/08 19:40:41 Waiting for dst bucket item count to get to: 100 curr count: 94
2021/12/08 19:40:41 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/08 19:40:41 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 19:40:41 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 19:40:46 Waiting for app: TestCacheExpiryCustom to get un-deployed
2021/12/08 19:40:51 Waiting for app: TestCacheExpiryCustom to get un-deployed
2021/12/08 19:40:56 Waiting for app: TestCacheExpiryCustom to get un-deployed
2021/12/08 19:41:01 Waiting for app: TestCacheExpiryCustom to get un-deployed
2021/12/08 19:41:01 App: TestCacheExpiryCustom got un-deployed
2021/12/08 19:41:01 No eventing-con process running
2021/12/08 19:41:01 Function: TestCacheExpiryCustom delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCacheExpiryCustom deleting in the background"
}
2021/12/08 19:41:03 Context: Bucket flush, response code: 200 dump: 
2021/12/08 19:41:13 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/08 19:41:18 Waiting for dst bucket item count to get to: 0 curr count: 25
2021/12/08 19:41:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 19:41:19 Context: Bucket flush, response code: 200 dump: 
2021/12/08 19:41:29 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/08 19:41:34 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/08 19:41:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCacheExpiryCustom (100.19s)
=== RUN   TestCacheOverflowCustom
2021/12/08 19:41:34 Deploying app: TestCacheOverflowCustom. Item count src bucket: 0 dst bucket: 0
2021/12/08 19:41:36 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCacheOverflowCustom' in metakv",
  "warnings": null
 }
}
2021/12/08 19:41:41 Waiting for app: TestCacheOverflowCustom to get deployed
2021/12/08 19:41:46 Waiting for app: TestCacheOverflowCustom to get deployed
2021/12/08 19:41:51 Waiting for app: TestCacheOverflowCustom to get deployed
2021/12/08 19:41:56 Waiting for app: TestCacheOverflowCustom to get deployed
2021/12/08 19:41:56 App: TestCacheOverflowCustom got deployed
2021/12/08 19:41:56 Starting bucket ops to source bucket
2021/12/08 19:42:01 Waiting for dst bucket item count to get to: 35 curr count: 0
2021/12/08 19:42:06 Waiting for dst bucket item count to get to: 35 curr count: 13
2021/12/08 19:42:11 Waiting for dst bucket item count to get to: 35 curr count: 13
2021/12/08 19:42:16 Waiting for dst bucket item count to get to: 35 curr count: 27
2021/12/08 19:42:21 Waiting for dst bucket item count to get to: 35 curr count: 27
2021/12/08 19:42:21 src & dst bucket item count matched up. src bucket count: 35 dst bucket count: 35
2021/12/08 19:42:21 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 19:42:21 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 19:42:26 Waiting for app: TestCacheOverflowCustom to get un-deployed
2021/12/08 19:42:31 Waiting for app: TestCacheOverflowCustom to get un-deployed
2021/12/08 19:42:36 Waiting for app: TestCacheOverflowCustom to get un-deployed
2021/12/08 19:42:41 Waiting for app: TestCacheOverflowCustom to get un-deployed
2021/12/08 19:42:41 App: TestCacheOverflowCustom got un-deployed
2021/12/08 19:42:41 No eventing-con process running
2021/12/08 19:42:41 Function: TestCacheOverflowCustom delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCacheOverflowCustom deleting in the background"
}
2021/12/08 19:42:43 Context: Bucket flush, response code: 200 dump: 
2021/12/08 19:42:53 Waiting for dst bucket item count to get to: 0 curr count: 5
2021/12/08 19:42:58 Waiting for dst bucket item count to get to: 0 curr count: 3
2021/12/08 19:42:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 19:43:00 Context: Bucket flush, response code: 200 dump: 
2021/12/08 19:43:10 Waiting for dst bucket item count to get to: 0 curr count: 17
2021/12/08 19:43:15 Waiting for dst bucket item count to get to: 0 curr count: 17
2021/12/08 19:43:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCacheOverflowCustom (100.30s)
=== RUN   TestStrictMode
2021/12/08 19:43:15 Deploying app: TestStrictMode. Item count src bucket: 0 dst bucket: 0
2021/12/08 19:43:16 Context: Post to main store, response code: 422 dump: {
 "name": "ERR_HANDLER_COMPILATION",
 "code": 27,
 "description": "Function compilation failed",
 "attributes": null,
 "runtime_info": {
  "code": 27,
  "info": {
   "area": "handlerCode",
   "column_number": 63,
   "compile_success": false,
   "description": "        destination[meta.id] = 00000000000000000000000000000101;\n                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nSyntaxError: Octal literals are not allowed in strict mode.\n",
   "index": 132,
   "language": "JavaScript",
   "line_number": 4
  }
 }
}
--- PASS: TestStrictMode (1.06s)
=== RUN   TestDataTypes
2021/12/08 19:43:16 Deploying app: TestDataTypes. Item count src bucket: 0 dst bucket: 0
2021/12/08 19:43:17 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDataTypes' in metakv",
  "warnings": null
 }
}
2021/12/08 19:43:22 Waiting for app: TestDataTypes to get deployed
2021/12/08 19:43:27 Waiting for app: TestDataTypes to get deployed
2021/12/08 19:43:32 Waiting for app: TestDataTypes to get deployed
2021/12/08 19:43:37 Waiting for app: TestDataTypes to get deployed
2021/12/08 19:43:37 App: TestDataTypes got deployed
2021/12/08 19:43:37 Starting bucket ops to source bucket
2021/12/08 19:43:42 Waiting for dst bucket item count to get to: 18 curr count: 0
2021/12/08 19:43:42 src & dst bucket item count matched up. src bucket count: 18 dst bucket count: 18
2021/12/08 19:43:42 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 19:43:42 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 19:43:47 Waiting for app: TestDataTypes to get un-deployed
2021/12/08 19:43:52 Waiting for app: TestDataTypes to get un-deployed
2021/12/08 19:43:57 Waiting for app: TestDataTypes to get un-deployed
2021/12/08 19:44:02 Waiting for app: TestDataTypes to get un-deployed
2021/12/08 19:44:02 App: TestDataTypes got un-deployed
2021/12/08 19:44:02 No eventing-con process running
2021/12/08 19:44:02 Function: TestDataTypes delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDataTypes deleting in the background"
}
2021/12/08 19:44:04 Context: Bucket flush, response code: 200 dump: 
2021/12/08 19:44:14 Waiting for dst bucket item count to get to: 0 curr count: 1
2021/12/08 19:44:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 19:44:16 Context: Bucket flush, response code: 200 dump: 
2021/12/08 19:44:26 Waiting for dst bucket item count to get to: 0 curr count: 18
2021/12/08 19:44:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDataTypes (70.14s)
=== RUN   TestEnoentGet
2021/12/08 19:44:26 Deploying app: TestEnoentGet. Item count src bucket: 0 dst bucket: 0
2021/12/08 19:44:27 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentGet' in metakv",
  "warnings": null
 }
}
2021/12/08 19:44:32 Waiting for app: TestEnoentGet to get deployed
2021/12/08 19:44:37 Waiting for app: TestEnoentGet to get deployed
2021/12/08 19:44:42 Waiting for app: TestEnoentGet to get deployed
2021/12/08 19:44:47 Waiting for app: TestEnoentGet to get deployed
2021/12/08 19:44:47 App: TestEnoentGet got deployed
2021/12/08 19:44:47 Starting bucket ops to source bucket
2021/12/08 19:44:52 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/08 19:44:57 Waiting for dst bucket item count to get to: 100 curr count: 81
2021/12/08 19:45:02 Waiting for dst bucket item count to get to: 100 curr count: 81
2021/12/08 19:45:02 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/08 19:45:02 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 19:45:02 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 19:45:07 Waiting for app: TestEnoentGet to get un-deployed
2021/12/08 19:45:12 Waiting for app: TestEnoentGet to get un-deployed
2021/12/08 19:45:17 Waiting for app: TestEnoentGet to get un-deployed
2021/12/08 19:45:22 Waiting for app: TestEnoentGet to get un-deployed
2021/12/08 19:45:22 App: TestEnoentGet got un-deployed
2021/12/08 19:45:22 No eventing-con process running
2021/12/08 19:45:22 Function: TestEnoentGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentGet deleting in the background"
}
2021/12/08 19:45:24 Context: Bucket flush, response code: 200 dump: 
2021/12/08 19:45:34 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/08 19:45:39 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/08 19:45:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 19:45:41 Context: Bucket flush, response code: 200 dump: 
2021/12/08 19:45:51 Waiting for dst bucket item count to get to: 0 curr count: 98
2021/12/08 19:45:56 Waiting for dst bucket item count to get to: 0 curr count: 98
2021/12/08 19:45:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 19:45:56 Deploying app: TestEnoentGet. Item count src bucket: 0 dst bucket: 0
2021/12/08 19:45:57 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentGet' in metakv",
  "warnings": null
 }
}
2021/12/08 19:46:02 Waiting for app: TestEnoentGet to get deployed
2021/12/08 19:46:07 Waiting for app: TestEnoentGet to get deployed
2021/12/08 19:46:12 Waiting for app: TestEnoentGet to get deployed
2021/12/08 19:46:17 Waiting for app: TestEnoentGet to get deployed
2021/12/08 19:46:17 App: TestEnoentGet got deployed
2021/12/08 19:46:17 Starting bucket ops to source bucket
2021/12/08 19:46:22 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/08 19:46:27 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/08 19:46:32 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/08 19:46:32 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/08 19:46:32 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 19:46:33 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 19:46:38 Waiting for app: TestEnoentGet to get un-deployed
2021/12/08 19:46:43 Waiting for app: TestEnoentGet to get un-deployed
2021/12/08 19:46:48 Waiting for app: TestEnoentGet to get un-deployed
2021/12/08 19:46:53 Waiting for app: TestEnoentGet to get un-deployed
2021/12/08 19:46:53 App: TestEnoentGet got un-deployed
2021/12/08 19:46:53 No eventing-con process running
2021/12/08 19:46:53 Function: TestEnoentGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentGet deleting in the background"
}
2021/12/08 19:46:54 Context: Bucket flush, response code: 200 dump: 
2021/12/08 19:47:04 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/08 19:47:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 19:47:06 Context: Bucket flush, response code: 200 dump: 
2021/12/08 19:47:16 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/08 19:47:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEnoentGet (170.30s)
=== RUN   TestEnoentDelete
2021/12/08 19:47:16 Deploying app: TestEnoentDelete. Item count src bucket: 0 dst bucket: 0
2021/12/08 19:47:17 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentDelete' in metakv",
  "warnings": null
 }
}
2021/12/08 19:47:22 Waiting for app: TestEnoentDelete to get deployed
2021/12/08 19:47:27 Waiting for app: TestEnoentDelete to get deployed
2021/12/08 19:47:32 Waiting for app: TestEnoentDelete to get deployed
2021/12/08 19:47:37 Waiting for app: TestEnoentDelete to get deployed
2021/12/08 19:47:37 App: TestEnoentDelete got deployed
2021/12/08 19:47:37 Starting bucket ops to source bucket
2021/12/08 19:47:43 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/08 19:47:48 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/08 19:47:53 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/08 19:47:53 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/08 19:47:53 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 19:47:53 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 19:47:58 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/08 19:48:03 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/08 19:48:08 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/08 19:48:13 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/08 19:48:13 App: TestEnoentDelete got un-deployed
2021/12/08 19:48:13 No eventing-con process running
2021/12/08 19:48:13 Function: TestEnoentDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentDelete deleting in the background"
}
2021/12/08 19:48:14 Context: Bucket flush, response code: 200 dump: 
2021/12/08 19:48:24 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/08 19:48:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 19:48:26 Context: Bucket flush, response code: 200 dump: 
2021/12/08 19:48:36 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/08 19:48:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 19:48:36 Deploying app: TestEnoentDelete. Item count src bucket: 0 dst bucket: 0
2021/12/08 19:48:37 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentDelete' in metakv",
  "warnings": null
 }
}
2021/12/08 19:48:42 Waiting for app: TestEnoentDelete to get deployed
2021/12/08 19:48:47 Waiting for app: TestEnoentDelete to get deployed
2021/12/08 19:48:52 Waiting for app: TestEnoentDelete to get deployed
2021/12/08 19:48:57 Waiting for app: TestEnoentDelete to get deployed
2021/12/08 19:48:57 App: TestEnoentDelete got deployed
2021/12/08 19:48:57 Starting bucket ops to source bucket
2021/12/08 19:49:03 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/08 19:49:08 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/08 19:49:13 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/08 19:49:13 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/08 19:49:13 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 19:49:13 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 19:49:18 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/08 19:49:23 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/08 19:49:28 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/08 19:49:33 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/08 19:49:33 App: TestEnoentDelete got un-deployed
2021/12/08 19:49:33 No eventing-con process running
2021/12/08 19:49:33 Function: TestEnoentDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentDelete deleting in the background"
}
2021/12/08 19:49:35 Context: Bucket flush, response code: 200 dump: 
2021/12/08 19:49:45 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/08 19:49:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 19:49:46 Context: Bucket flush, response code: 200 dump: 
2021/12/08 19:49:56 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/08 19:49:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEnoentDelete (160.17s)
=== RUN   TestError
2021/12/08 19:50:06 Waiting for app: TestError to get un-deployed
2021/12/08 19:50:06 App: TestError got un-deployed
2021/12/08 19:50:06 No eventing-con process running
2021/12/08 19:50:06 Function: TestError delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestError not found"
 }
}
2021/12/08 19:50:08 Context: Bucket flush, response code: 200 dump: 
2021/12/08 19:50:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 19:50:15 Context: Bucket flush, response code: 200 dump: 
2021/12/08 19:50:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 19:50:20 Deploying app: TestError. Item count src bucket: 0 dst bucket: 0
2021/12/08 19:50:21 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestError' in metakv",
  "warnings": null
 }
}
2021/12/08 19:50:26 Waiting for app: TestError to get deployed
2021/12/08 19:50:31 Waiting for app: TestError to get deployed
2021/12/08 19:50:36 Waiting for app: TestError to get deployed
2021/12/08 19:50:41 Waiting for app: TestError to get deployed
2021/12/08 19:50:41 App: TestError got deployed
2021/12/08 19:50:41 Starting bucket ops to source bucket
2021/12/08 19:50:46 Waiting for dst bucket item count to get to: 300 curr count: 0
2021/12/08 19:50:51 Waiting for dst bucket item count to get to: 300 curr count: 0
2021/12/08 19:50:51 src & dst bucket item count matched up. src bucket count: 300 dst bucket count: 300
2021/12/08 19:50:51 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 19:50:51 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 19:50:56 Waiting for app: TestError to get un-deployed
2021/12/08 19:51:01 Waiting for app: TestError to get un-deployed
2021/12/08 19:51:06 Waiting for app: TestError to get un-deployed
2021/12/08 19:51:12 Waiting for app: TestError to get un-deployed
2021/12/08 19:51:12 App: TestError got un-deployed
2021/12/08 19:51:12 No eventing-con process running
2021/12/08 19:51:12 Function: TestError delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestError deleting in the background"
}
2021/12/08 19:51:13 Context: Bucket flush, response code: 200 dump: 
2021/12/08 19:51:23 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/08 19:51:28 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/08 19:51:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 19:51:30 Context: Bucket flush, response code: 200 dump: 
2021/12/08 19:51:40 Waiting for dst bucket item count to get to: 0 curr count: 203
2021/12/08 19:51:45 Waiting for dst bucket item count to get to: 0 curr count: 203
2021/12/08 19:51:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestError (108.73s)
=== RUN   TestCRLF
2021/12/08 19:51:55 Waiting for app: TestCRLF to get un-deployed
2021/12/08 19:51:55 App: TestCRLF got un-deployed
2021/12/08 19:51:55 No eventing-con process running
2021/12/08 19:51:55 Function: TestCRLF delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestCRLF not found"
 }
}
2021/12/08 19:51:57 Context: Bucket flush, response code: 200 dump: 
2021/12/08 19:52:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 19:52:04 Context: Bucket flush, response code: 200 dump: 
2021/12/08 19:52:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 19:52:09 Deploying app: TestCRLF. Item count src bucket: 0 dst bucket: 0
2021/12/08 19:52:10 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCRLF' in metakv",
  "warnings": null
 }
}
2021/12/08 19:52:15 Waiting for app: TestCRLF to get deployed
2021/12/08 19:52:20 Waiting for app: TestCRLF to get deployed
2021/12/08 19:52:25 Waiting for app: TestCRLF to get deployed
2021/12/08 19:52:30 Waiting for app: TestCRLF to get deployed
2021/12/08 19:52:30 App: TestCRLF got deployed
2021/12/08 19:52:30 Starting bucket ops to source bucket
2021/12/08 19:52:35 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/08 19:52:40 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/08 19:52:45 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/08 19:52:45 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/08 19:52:45 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 19:52:45 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 19:52:50 Waiting for app: TestCRLF to get un-deployed
2021/12/08 19:52:55 Waiting for app: TestCRLF to get un-deployed
2021/12/08 19:53:00 Waiting for app: TestCRLF to get un-deployed
2021/12/08 19:53:05 Waiting for app: TestCRLF to get un-deployed
2021/12/08 19:53:05 App: TestCRLF got un-deployed
2021/12/08 19:53:05 No eventing-con process running
2021/12/08 19:53:05 Function: TestCRLF delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCRLF deleting in the background"
}
2021/12/08 19:53:07 Context: Bucket flush, response code: 200 dump: 
2021/12/08 19:53:17 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/08 19:53:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 19:53:19 Context: Bucket flush, response code: 200 dump: 
2021/12/08 19:53:29 Waiting for dst bucket item count to get to: 0 curr count: 94
2021/12/08 19:53:34 Waiting for dst bucket item count to get to: 0 curr count: 44
2021/12/08 19:53:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCRLF (108.74s)
=== RUN   TestImportExport
2021/12/08 19:53:44 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/08 19:53:44 App: TestImportExport_n1ql_insert_on_update got un-deployed
2021/12/08 19:53:44 No eventing-con process running
2021/12/08 19:53:44 Function: TestImportExport_n1ql_insert_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestImportExport_n1ql_insert_on_update not found"
 }
}
2021/12/08 19:53:46 Context: Bucket flush, response code: 200 dump: 
2021/12/08 19:53:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 19:53:52 Context: Bucket flush, response code: 200 dump: 
2021/12/08 19:53:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 19:54:02 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/08 19:54:02 App: TestImportExport_bucket_op_on_update got un-deployed
2021/12/08 19:54:03 No eventing-con process running
2021/12/08 19:54:03 Function: TestImportExport_bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestImportExport_bucket_op_on_update not found"
 }
}
2021/12/08 19:54:04 Context: Bucket flush, response code: 200 dump: 
2021/12/08 19:54:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 19:54:11 Context: Bucket flush, response code: 200 dump: 
2021/12/08 19:54:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 19:54:16 Deploying app: TestImportExport_n1ql_insert_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/08 19:54:17 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestImportExport_n1ql_insert_on_update' in metakv",
  "warnings": null
 }
}
2021/12/08 19:54:17 Deploying app: TestImportExport_bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/08 19:54:19 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestImportExport_bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/08 19:54:24 Waiting for app: TestImportExport_n1ql_insert_on_update to get deployed
2021/12/08 19:54:29 Waiting for app: TestImportExport_n1ql_insert_on_update to get deployed
2021/12/08 19:54:34 Waiting for app: TestImportExport_n1ql_insert_on_update to get deployed
2021/12/08 19:54:39 Waiting for app: TestImportExport_n1ql_insert_on_update to get deployed
2021/12/08 19:54:39 App: TestImportExport_n1ql_insert_on_update got deployed
2021/12/08 19:54:44 Waiting for app: TestImportExport_bucket_op_on_update to get deployed
2021/12/08 19:54:49 Waiting for app: TestImportExport_bucket_op_on_update to get deployed
2021/12/08 19:54:54 Waiting for app: TestImportExport_bucket_op_on_update to get deployed
2021/12/08 19:54:54 App: TestImportExport_bucket_op_on_update got deployed
2021/12/08 19:54:54 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 19:54:59 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/08 19:55:04 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/08 19:55:09 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/08 19:55:14 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/08 19:55:14 App: TestImportExport_n1ql_insert_on_update got un-deployed
2021/12/08 19:55:14 No eventing-con process running
2021/12/08 19:55:14 Function: TestImportExport_n1ql_insert_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestImportExport_n1ql_insert_on_update deleting in the background"
}
2021/12/08 19:55:14 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 19:55:19 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/08 19:55:24 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/08 19:55:29 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/08 19:55:34 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/08 19:55:34 App: TestImportExport_bucket_op_on_update got un-deployed
2021/12/08 19:55:34 No eventing-con process running
2021/12/08 19:55:34 Function: TestImportExport_bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestImportExport_bucket_op_on_update deleting in the background"
}
2021/12/08 19:55:51 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/08 19:55:51 App: TestImportExport_bucket_op_on_update got un-deployed
2021/12/08 19:55:51 No eventing-con process running
2021/12/08 19:55:51 Function: TestImportExport_bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestImportExport_bucket_op_on_update deleting in the background"
}
2021/12/08 19:55:53 Context: Bucket flush, response code: 200 dump: 
2021/12/08 19:55:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 19:56:00 Context: Bucket flush, response code: 200 dump: 
2021/12/08 19:56:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 19:56:10 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/08 19:56:10 App: TestImportExport_n1ql_insert_on_update got un-deployed
2021/12/08 19:56:10 No eventing-con process running
2021/12/08 19:56:10 Function: TestImportExport_n1ql_insert_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestImportExport_n1ql_insert_on_update deleting in the background"
}
2021/12/08 19:56:12 Context: Bucket flush, response code: 200 dump: 
2021/12/08 19:56:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 19:56:19 Context: Bucket flush, response code: 200 dump: 
2021/12/08 19:56:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestImportExport (169.73s)
=== RUN   TestDeployUndeployLoopNonDefaultSettings
2021/12/08 19:56:34 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/08 19:56:34 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/08 19:56:34 No eventing-con process running
2021/12/08 19:56:34 Function: TestDeployUndeployLoopNonDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDeployUndeployLoopNonDefaultSettings not found"
 }
}
2021/12/08 19:56:35 Context: Bucket flush, response code: 200 dump: 
2021/12/08 19:56:40 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 19:56:42 Context: Bucket flush, response code: 200 dump: 
2021/12/08 19:56:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 19:56:47 Deploying app: TestDeployUndeployLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/08 19:56:48 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/08 19:56:48 Starting bucket ops to source bucket
2021/12/08 19:56:56 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 19:57:01 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 19:57:06 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 19:57:11 Waiting for dst bucket item count to get to: 5000 curr count: 9
2021/12/08 19:57:16 Waiting for dst bucket item count to get to: 5000 curr count: 2509
2021/12/08 19:57:16 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/08 19:57:21 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get deployed
2021/12/08 19:57:21 App: TestDeployUndeployLoopNonDefaultSettings got deployed
2021/12/08 19:57:21 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 19:57:21 Undeploying app: bucket_op_on_update
2021/12/08 19:57:21 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 19:57:26 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/08 19:57:31 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/08 19:57:36 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/08 19:57:41 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/08 19:57:41 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/08 19:57:41 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 19:57:41 No eventing-con process running
2021/12/08 19:57:43 Context: Bucket flush, response code: 200 dump: 
2021/12/08 19:57:50 Context: Bucket flush, response code: 200 dump: 
2021/12/08 19:58:25 Deploying app: TestDeployUndeployLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/08 19:58:27 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/08 19:58:27 Starting bucket ops to source bucket
2021/12/08 19:58:35 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 19:58:40 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 19:58:45 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 19:58:50 Waiting for dst bucket item count to get to: 5000 curr count: 960
2021/12/08 19:58:55 Waiting for dst bucket item count to get to: 5000 curr count: 960
2021/12/08 19:58:55 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/08 19:59:00 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get deployed
2021/12/08 19:59:00 App: TestDeployUndeployLoopNonDefaultSettings got deployed
2021/12/08 19:59:00 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 19:59:00 Undeploying app: bucket_op_on_update
2021/12/08 19:59:00 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 19:59:05 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/08 19:59:10 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/08 19:59:15 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/08 19:59:20 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/08 19:59:20 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/08 19:59:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 19:59:20 No eventing-con process running
2021/12/08 19:59:22 Context: Bucket flush, response code: 200 dump: 
2021/12/08 19:59:29 Context: Bucket flush, response code: 200 dump: 
2021/12/08 20:00:04 Deploying app: TestDeployUndeployLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/08 20:00:05 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/08 20:00:05 Starting bucket ops to source bucket
2021/12/08 20:00:13 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 20:00:18 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 20:00:23 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 20:00:28 Waiting for dst bucket item count to get to: 5000 curr count: 2985
2021/12/08 20:00:33 Waiting for dst bucket item count to get to: 5000 curr count: 2985
2021/12/08 20:00:33 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/08 20:00:38 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get deployed
2021/12/08 20:00:38 App: TestDeployUndeployLoopNonDefaultSettings got deployed
2021/12/08 20:00:38 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 20:00:38 Undeploying app: bucket_op_on_update
2021/12/08 20:00:38 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 20:00:43 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/08 20:00:48 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/08 20:00:53 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/08 20:00:58 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/08 20:00:58 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/08 20:00:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 20:00:58 No eventing-con process running
2021/12/08 20:01:00 Context: Bucket flush, response code: 200 dump: 
2021/12/08 20:01:07 Context: Bucket flush, response code: 200 dump: 
2021/12/08 20:01:42 Deploying app: TestDeployUndeployLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/08 20:01:43 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/08 20:01:43 Starting bucket ops to source bucket
2021/12/08 20:01:51 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 20:01:56 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 20:02:01 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 20:02:06 Waiting for dst bucket item count to get to: 5000 curr count: 2249
2021/12/08 20:02:11 Waiting for dst bucket item count to get to: 5000 curr count: 4749
2021/12/08 20:02:11 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/08 20:02:16 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get deployed
2021/12/08 20:02:16 App: TestDeployUndeployLoopNonDefaultSettings got deployed
2021/12/08 20:02:16 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 20:02:16 Undeploying app: bucket_op_on_update
2021/12/08 20:02:16 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 20:02:21 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/08 20:02:26 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/08 20:02:31 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/08 20:02:36 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/08 20:02:36 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/08 20:02:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 20:02:37 No eventing-con process running
2021/12/08 20:02:38 Context: Bucket flush, response code: 200 dump: 
2021/12/08 20:02:46 Context: Bucket flush, response code: 200 dump: 
2021/12/08 20:03:21 Deploying app: TestDeployUndeployLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/08 20:03:22 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/08 20:03:22 Starting bucket ops to source bucket
2021/12/08 20:03:30 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 20:03:35 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 20:03:40 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 20:03:45 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 20:03:45 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/08 20:03:50 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get deployed
2021/12/08 20:03:50 App: TestDeployUndeployLoopNonDefaultSettings got deployed
2021/12/08 20:03:50 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 20:03:50 Undeploying app: bucket_op_on_update
2021/12/08 20:03:50 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 20:03:55 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/08 20:04:00 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/08 20:04:05 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/08 20:04:10 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/08 20:04:10 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/08 20:04:10 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 20:04:10 No eventing-con process running
2021/12/08 20:04:12 Context: Bucket flush, response code: 200 dump: 
2021/12/08 20:04:19 Context: Bucket flush, response code: 200 dump: 
2021/12/08 20:04:54 Function: TestDeployUndeployLoopNonDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDeployUndeployLoopNonDefaultSettings deleting in the background"
}
--- PASS: TestDeployUndeployLoopNonDefaultSettings (510.39s)
=== RUN   TestOnUpdateN1QLOp
2021/12/08 20:05:04 Waiting for app: TestOnUpdateN1QLOp to get un-deployed
2021/12/08 20:05:04 App: TestOnUpdateN1QLOp got un-deployed
2021/12/08 20:05:04 No eventing-con process running
2021/12/08 20:05:04 Function: TestOnUpdateN1QLOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateN1QLOp not found"
 }
}
2021/12/08 20:05:06 Context: Bucket flush, response code: 200 dump: 
2021/12/08 20:05:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 20:05:13 Context: Bucket flush, response code: 200 dump: 
2021/12/08 20:05:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 20:05:18 Deploying app: TestOnUpdateN1QLOp. Item count src bucket: 0 dst bucket: 0
2021/12/08 20:05:19 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateN1QLOp' in metakv",
  "warnings": null
 }
}
2021/12/08 20:05:19 Starting bucket ops to source bucket
2021/12/08 20:05:27 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 20:05:32 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 20:05:37 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 20:05:42 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 20:05:47 Waiting for dst bucket item count to get to: 5000 curr count: 2553
2021/12/08 20:05:47 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/08 20:05:47 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 20:05:47 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 20:05:52 Waiting for app: TestOnUpdateN1QLOp to get un-deployed
2021/12/08 20:05:57 Waiting for app: TestOnUpdateN1QLOp to get un-deployed
2021/12/08 20:06:02 Waiting for app: TestOnUpdateN1QLOp to get un-deployed
2021/12/08 20:06:07 Waiting for app: TestOnUpdateN1QLOp to get un-deployed
2021/12/08 20:06:07 App: TestOnUpdateN1QLOp got un-deployed
2021/12/08 20:06:07 No eventing-con process running
2021/12/08 20:06:07 Function: TestOnUpdateN1QLOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateN1QLOp deleting in the background"
}
2021/12/08 20:06:09 Context: Bucket flush, response code: 200 dump: 
2021/12/08 20:06:19 Waiting for dst bucket item count to get to: 0 curr count: 4608
2021/12/08 20:06:24 Waiting for dst bucket item count to get to: 0 curr count: 2108
2021/12/08 20:06:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 20:06:25 Context: Bucket flush, response code: 200 dump: 
2021/12/08 20:06:36 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/08 20:06:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateN1QLOp (101.59s)
=== RUN   TestOnUpdateBucketOpDefaultSettings
2021/12/08 20:06:46 Waiting for app: TestOnUpdateBucketOpDefaultSettings to get un-deployed
2021/12/08 20:06:46 App: TestOnUpdateBucketOpDefaultSettings got un-deployed
2021/12/08 20:06:46 No eventing-con process running
2021/12/08 20:06:46 Function: TestOnUpdateBucketOpDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateBucketOpDefaultSettings not found"
 }
}
2021/12/08 20:06:47 Context: Bucket flush, response code: 200 dump: 
2021/12/08 20:06:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 20:06:54 Context: Bucket flush, response code: 200 dump: 
2021/12/08 20:06:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 20:06:59 Deploying app: TestOnUpdateBucketOpDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/08 20:07:00 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateBucketOpDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/08 20:07:00 Starting bucket ops to source bucket
2021/12/08 20:07:08 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 20:07:13 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 20:07:18 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 20:07:23 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 20:07:23 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/08 20:07:23 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 20:07:23 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 20:07:28 Waiting for app: TestOnUpdateBucketOpDefaultSettings to get un-deployed
2021/12/08 20:07:33 Waiting for app: TestOnUpdateBucketOpDefaultSettings to get un-deployed
2021/12/08 20:07:38 Waiting for app: TestOnUpdateBucketOpDefaultSettings to get un-deployed
2021/12/08 20:07:43 Waiting for app: TestOnUpdateBucketOpDefaultSettings to get un-deployed
2021/12/08 20:07:43 App: TestOnUpdateBucketOpDefaultSettings got un-deployed
2021/12/08 20:07:43 No eventing-con process running
2021/12/08 20:07:43 Function: TestOnUpdateBucketOpDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateBucketOpDefaultSettings deleting in the background"
}
2021/12/08 20:07:45 Context: Bucket flush, response code: 200 dump: 
2021/12/08 20:07:55 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/08 20:07:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 20:07:57 Context: Bucket flush, response code: 200 dump: 
2021/12/08 20:08:07 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/08 20:08:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateBucketOpDefaultSettings (91.47s)
=== RUN   TestOnUpdateBucketOpNonDefaultSettings
2021/12/08 20:08:17 Waiting for app: TestOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/08 20:08:17 App: TestOnUpdateBucketOpNonDefaultSettings got un-deployed
2021/12/08 20:08:17 No eventing-con process running
2021/12/08 20:08:17 Function: TestOnUpdateBucketOpNonDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateBucketOpNonDefaultSettings not found"
 }
}
2021/12/08 20:08:19 Context: Bucket flush, response code: 200 dump: 
2021/12/08 20:08:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 20:08:26 Context: Bucket flush, response code: 200 dump: 
2021/12/08 20:08:31 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 20:08:31 Deploying app: TestOnUpdateBucketOpNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/08 20:08:32 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateBucketOpNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/08 20:08:32 Starting bucket ops to source bucket
2021/12/08 20:08:40 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 20:08:45 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 20:08:50 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 20:08:55 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 20:08:55 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/08 20:08:55 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 20:08:55 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 20:09:00 Waiting for app: TestOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/08 20:09:05 Waiting for app: TestOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/08 20:09:10 Waiting for app: TestOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/08 20:09:15 Waiting for app: TestOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/08 20:09:15 App: TestOnUpdateBucketOpNonDefaultSettings got un-deployed
2021/12/08 20:09:15 No eventing-con process running
2021/12/08 20:09:15 Function: TestOnUpdateBucketOpNonDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateBucketOpNonDefaultSettings deleting in the background"
}
2021/12/08 20:09:17 Context: Bucket flush, response code: 200 dump: 
2021/12/08 20:09:27 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/08 20:09:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 20:09:28 Context: Bucket flush, response code: 200 dump: 
2021/12/08 20:09:38 Waiting for dst bucket item count to get to: 0 curr count: 4298
2021/12/08 20:09:43 Waiting for dst bucket item count to get to: 0 curr count: 1798
2021/12/08 20:09:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateBucketOpNonDefaultSettings (96.49s)
=== RUN   TestOnUpdateBucketOpDefaultSettings10K
2021/12/08 20:09:53 Waiting for app: TestOnUpdateBucketOpDefaultSettings10K to get un-deployed
2021/12/08 20:09:53 App: TestOnUpdateBucketOpDefaultSettings10K got un-deployed
2021/12/08 20:09:53 No eventing-con process running
2021/12/08 20:09:54 Function: TestOnUpdateBucketOpDefaultSettings10K delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateBucketOpDefaultSettings10K not found"
 }
}
2021/12/08 20:09:55 Context: Bucket flush, response code: 200 dump: 
2021/12/08 20:10:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 20:10:02 Context: Bucket flush, response code: 200 dump: 
2021/12/08 20:10:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 20:10:07 Deploying app: TestOnUpdateBucketOpDefaultSettings10K. Item count src bucket: 0 dst bucket: 0
2021/12/08 20:10:08 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateBucketOpDefaultSettings10K' in metakv",
  "warnings": null
 }
}
2021/12/08 20:10:08 Starting bucket ops to source bucket
2021/12/08 20:10:16 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 20:10:21 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 20:10:26 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 20:10:31 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 20:10:36 Waiting for dst bucket item count to get to: 5000 curr count: 2500
2021/12/08 20:10:36 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/08 20:10:36 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 20:10:36 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 20:10:41 Waiting for app: TestOnUpdateBucketOpDefaultSettings10K to get un-deployed
2021/12/08 20:10:46 Waiting for app: TestOnUpdateBucketOpDefaultSettings10K to get un-deployed
2021/12/08 20:10:51 Waiting for app: TestOnUpdateBucketOpDefaultSettings10K to get un-deployed
2021/12/08 20:10:56 Waiting for app: TestOnUpdateBucketOpDefaultSettings10K to get un-deployed
2021/12/08 20:10:57 App: TestOnUpdateBucketOpDefaultSettings10K got un-deployed
2021/12/08 20:10:57 No eventing-con process running
2021/12/08 20:10:57 Function: TestOnUpdateBucketOpDefaultSettings10K delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateBucketOpDefaultSettings10K deleting in the background"
}
2021/12/08 20:10:58 Context: Bucket flush, response code: 200 dump: 
2021/12/08 20:11:08 Waiting for dst bucket item count to get to: 0 curr count: 3897
2021/12/08 20:11:13 Waiting for dst bucket item count to get to: 0 curr count: 1397
2021/12/08 20:11:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 20:11:15 Context: Bucket flush, response code: 200 dump: 
2021/12/08 20:11:25 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/08 20:11:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateBucketOpDefaultSettings10K (101.73s)
=== RUN   TestOnUpdateBucketOpDefaultSettings100K
2021/12/08 20:11:35 Waiting for app: TestOnUpdateBucketOpDefaultSettings100K to get un-deployed
2021/12/08 20:11:35 App: TestOnUpdateBucketOpDefaultSettings100K got un-deployed
2021/12/08 20:11:35 No eventing-con process running
2021/12/08 20:11:35 Function: TestOnUpdateBucketOpDefaultSettings100K delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateBucketOpDefaultSettings100K not found"
 }
}
2021/12/08 20:11:37 Context: Bucket flush, response code: 200 dump: 
2021/12/08 20:11:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 20:11:44 Context: Bucket flush, response code: 200 dump: 
2021/12/08 20:11:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 20:11:49 Deploying app: TestOnUpdateBucketOpDefaultSettings100K. Item count src bucket: 0 dst bucket: 0
2021/12/08 20:11:50 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateBucketOpDefaultSettings100K' in metakv",
  "warnings": null
 }
}
2021/12/08 20:11:50 Starting bucket ops to source bucket
2021/12/08 20:11:58 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 20:12:03 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 20:12:08 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 20:12:13 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 20:12:18 Waiting for dst bucket item count to get to: 5000 curr count: 2500
2021/12/08 20:12:18 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/08 20:12:18 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 20:12:19 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 20:12:24 Waiting for app: TestOnUpdateBucketOpDefaultSettings100K to get un-deployed
2021/12/08 20:12:29 Waiting for app: TestOnUpdateBucketOpDefaultSettings100K to get un-deployed
2021/12/08 20:12:34 Waiting for app: TestOnUpdateBucketOpDefaultSettings100K to get un-deployed
2021/12/08 20:12:39 Waiting for app: TestOnUpdateBucketOpDefaultSettings100K to get un-deployed
2021/12/08 20:12:39 App: TestOnUpdateBucketOpDefaultSettings100K got un-deployed
2021/12/08 20:12:39 No eventing-con process running
2021/12/08 20:12:39 Function: TestOnUpdateBucketOpDefaultSettings100K delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateBucketOpDefaultSettings100K deleting in the background"
}
2021/12/08 20:12:41 Context: Bucket flush, response code: 200 dump: 
2021/12/08 20:12:51 Waiting for dst bucket item count to get to: 0 curr count: 4139
2021/12/08 20:12:56 Waiting for dst bucket item count to get to: 0 curr count: 4139
2021/12/08 20:12:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 20:12:57 Context: Bucket flush, response code: 200 dump: 
2021/12/08 20:13:07 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/08 20:13:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateBucketOpDefaultSettings100K (102.24s)
=== RUN   TestOnDeleteBucketOp
2021/12/08 20:13:17 Waiting for app: TestOnDeleteBucketOp to get un-deployed
2021/12/08 20:13:17 App: TestOnDeleteBucketOp got un-deployed
2021/12/08 20:13:17 No eventing-con process running
2021/12/08 20:13:17 Function: TestOnDeleteBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnDeleteBucketOp not found"
 }
}
2021/12/08 20:13:19 Context: Bucket flush, response code: 200 dump: 
2021/12/08 20:13:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 20:13:26 Context: Bucket flush, response code: 200 dump: 
2021/12/08 20:13:31 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 20:13:31 Deploying app: TestOnDeleteBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/08 20:13:32 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnDeleteBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/08 20:13:32 Starting bucket ops to source bucket
2021/12/08 20:13:43 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 20:13:48 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 20:13:53 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 20:13:53 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/08 20:13:53 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 20:13:53 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 20:13:58 Waiting for app: TestOnDeleteBucketOp to get un-deployed
2021/12/08 20:14:03 Waiting for app: TestOnDeleteBucketOp to get un-deployed
2021/12/08 20:14:08 Waiting for app: TestOnDeleteBucketOp to get un-deployed
2021/12/08 20:14:13 Waiting for app: TestOnDeleteBucketOp to get un-deployed
2021/12/08 20:14:13 App: TestOnDeleteBucketOp got un-deployed
2021/12/08 20:14:13 No eventing-con process running
2021/12/08 20:14:13 Function: TestOnDeleteBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnDeleteBucketOp deleting in the background"
}
2021/12/08 20:14:15 Context: Bucket flush, response code: 200 dump: 
2021/12/08 20:14:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 20:14:22 Context: Bucket flush, response code: 200 dump: 
2021/12/08 20:14:32 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/08 20:14:37 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/08 20:14:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnDeleteBucketOp (89.81s)
=== RUN   TestOnDeleteBucketOp5K
2021/12/08 20:14:47 Waiting for app: TestOnDeleteBucketOp5K to get un-deployed
2021/12/08 20:14:47 App: TestOnDeleteBucketOp5K got un-deployed
2021/12/08 20:14:47 No eventing-con process running
2021/12/08 20:14:47 Function: TestOnDeleteBucketOp5K delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnDeleteBucketOp5K not found"
 }
}
2021/12/08 20:14:49 Context: Bucket flush, response code: 200 dump: 
2021/12/08 20:14:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 20:14:56 Context: Bucket flush, response code: 200 dump: 
2021/12/08 20:15:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 20:15:01 Deploying app: TestOnDeleteBucketOp5K. Item count src bucket: 0 dst bucket: 0
2021/12/08 20:15:02 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnDeleteBucketOp5K' in metakv",
  "warnings": null
 }
}
2021/12/08 20:15:02 Starting bucket ops to source bucket
2021/12/08 20:15:13 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 20:15:18 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 20:15:23 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 20:15:23 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/08 20:15:23 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 20:15:23 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 20:15:28 Waiting for app: TestOnDeleteBucketOp5K to get un-deployed
2021/12/08 20:15:33 Waiting for app: TestOnDeleteBucketOp5K to get un-deployed
2021/12/08 20:15:38 Waiting for app: TestOnDeleteBucketOp5K to get un-deployed
2021/12/08 20:15:43 Waiting for app: TestOnDeleteBucketOp5K to get un-deployed
2021/12/08 20:15:43 App: TestOnDeleteBucketOp5K got un-deployed
2021/12/08 20:15:43 No eventing-con process running
2021/12/08 20:15:43 Function: TestOnDeleteBucketOp5K delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnDeleteBucketOp5K deleting in the background"
}
2021/12/08 20:15:45 Context: Bucket flush, response code: 200 dump: 
2021/12/08 20:15:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 20:15:52 Context: Bucket flush, response code: 200 dump: 
2021/12/08 20:16:02 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/08 20:16:07 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/08 20:16:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnDeleteBucketOp5K (89.83s)
=== RUN   TestTimerBucketOp
2021/12/08 20:16:17 Waiting for app: TestTimerBucketOp to get un-deployed
2021/12/08 20:16:17 App: TestTimerBucketOp got un-deployed
2021/12/08 20:16:17 No eventing-con process running
2021/12/08 20:16:17 Function: TestTimerBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestTimerBucketOp not found"
 }
}
2021/12/08 20:16:19 Context: Bucket flush, response code: 200 dump: 
2021/12/08 20:16:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 20:16:26 Context: Bucket flush, response code: 200 dump: 
2021/12/08 20:16:31 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 20:16:31 Deploying app: TestTimerBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/08 20:16:32 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestTimerBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/08 20:16:32 Starting bucket ops to source bucket
2021/12/08 20:16:40 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 20:16:45 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 20:16:50 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 20:16:55 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 20:17:00 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 20:17:05 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 20:17:10 Waiting for dst bucket item count to get to: 5000 curr count: 1047
2021/12/08 20:17:15 Waiting for dst bucket item count to get to: 5000 curr count: 1047
2021/12/08 20:17:15 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/08 20:17:15 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 20:17:15 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 20:17:20 Waiting for app: TestTimerBucketOp to get un-deployed
2021/12/08 20:17:25 Waiting for app: TestTimerBucketOp to get un-deployed
2021/12/08 20:17:30 Waiting for app: TestTimerBucketOp to get un-deployed
2021/12/08 20:17:35 Waiting for app: TestTimerBucketOp to get un-deployed
2021/12/08 20:17:35 App: TestTimerBucketOp got un-deployed
2021/12/08 20:17:35 No eventing-con process running
2021/12/08 20:17:35 Function: TestTimerBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestTimerBucketOp deleting in the background"
}
2021/12/08 20:17:37 Context: Bucket flush, response code: 200 dump: 
2021/12/08 20:17:47 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/08 20:17:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 20:17:49 Context: Bucket flush, response code: 200 dump: 
2021/12/08 20:17:59 Waiting for dst bucket item count to get to: 0 curr count: 4552
2021/12/08 20:18:04 Waiting for dst bucket item count to get to: 0 curr count: 2052
2021/12/08 20:18:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestTimerBucketOp (116.54s)
=== RUN   TestTimerInPastBucketOp
2021/12/08 20:18:14 Waiting for app: TestTimerInPastBucketOp to get un-deployed
2021/12/08 20:18:14 App: TestTimerInPastBucketOp got un-deployed
2021/12/08 20:18:14 No eventing-con process running
2021/12/08 20:18:14 Function: TestTimerInPastBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestTimerInPastBucketOp not found"
 }
}
2021/12/08 20:18:15 Context: Bucket flush, response code: 200 dump: 
2021/12/08 20:18:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 20:18:22 Context: Bucket flush, response code: 200 dump: 
2021/12/08 20:18:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 20:18:27 Deploying app: TestTimerInPastBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/08 20:18:28 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestTimerInPastBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/08 20:18:28 Starting bucket ops to source bucket
2021/12/08 20:18:36 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 20:18:41 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 20:18:46 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 20:18:51 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 20:18:56 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 20:19:01 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 20:19:06 Waiting for dst bucket item count to get to: 5000 curr count: 2377
2021/12/08 20:19:11 Waiting for dst bucket item count to get to: 5000 curr count: 4877
2021/12/08 20:19:11 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/08 20:19:11 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 20:19:12 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 20:19:17 Waiting for app: TestTimerInPastBucketOp to get un-deployed
2021/12/08 20:19:22 Waiting for app: TestTimerInPastBucketOp to get un-deployed
2021/12/08 20:19:27 Waiting for app: TestTimerInPastBucketOp to get un-deployed
2021/12/08 20:19:32 Waiting for app: TestTimerInPastBucketOp to get un-deployed
2021/12/08 20:19:32 App: TestTimerInPastBucketOp got un-deployed
2021/12/08 20:19:32 No eventing-con process running
2021/12/08 20:19:32 Function: TestTimerInPastBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestTimerInPastBucketOp deleting in the background"
}
2021/12/08 20:19:33 Context: Bucket flush, response code: 200 dump: 
2021/12/08 20:19:43 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/08 20:19:48 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/08 20:19:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 20:19:50 Context: Bucket flush, response code: 200 dump: 
2021/12/08 20:20:00 Waiting for dst bucket item count to get to: 0 curr count: 3410
2021/12/08 20:20:05 Waiting for dst bucket item count to get to: 0 curr count: 3410
2021/12/08 20:20:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestTimerInPastBucketOp (121.56s)
=== RUN   TestDeployUndeployLoopTimer
2021/12/08 20:20:15 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/08 20:20:15 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/08 20:20:15 No eventing-con process running
2021/12/08 20:20:15 Function: TestDeployUndeployLoopTimer delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDeployUndeployLoopTimer not found"
 }
}
2021/12/08 20:20:17 Context: Bucket flush, response code: 200 dump: 
2021/12/08 20:20:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 20:20:24 Context: Bucket flush, response code: 200 dump: 
2021/12/08 20:20:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 20:20:29 Deploying app: TestDeployUndeployLoopTimer. Item count src bucket: 0 dst bucket: 0
2021/12/08 20:20:30 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopTimer' in metakv",
  "warnings": null
 }
}
2021/12/08 20:20:30 Starting bucket ops to source bucket
2021/12/08 20:20:38 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 20:20:43 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 20:20:48 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 20:20:53 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 20:20:58 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 20:21:03 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 20:21:08 Waiting for dst bucket item count to get to: 5000 curr count: 792
2021/12/08 20:21:13 Waiting for dst bucket item count to get to: 5000 curr count: 3292
2021/12/08 20:21:13 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/08 20:21:18 Waiting for app: TestDeployUndeployLoopTimer to get deployed
2021/12/08 20:21:18 App: TestDeployUndeployLoopTimer got deployed
2021/12/08 20:21:18 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 20:21:18 Undeploying app: bucket_op_with_timer
2021/12/08 20:21:18 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 20:21:23 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/08 20:21:28 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/08 20:21:33 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/08 20:21:38 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/08 20:21:38 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/08 20:21:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 20:21:38 No eventing-con process running
2021/12/08 20:21:40 Context: Bucket flush, response code: 200 dump: 
2021/12/08 20:21:47 Context: Bucket flush, response code: 200 dump: 
2021/12/08 20:22:22 Deploying app: TestDeployUndeployLoopTimer. Item count src bucket: 0 dst bucket: 0
2021/12/08 20:22:23 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopTimer' in metakv",
  "warnings": null
 }
}
2021/12/08 20:22:23 Starting bucket ops to source bucket
2021/12/08 20:22:31 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 20:22:36 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 20:22:41 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 20:22:46 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 20:22:51 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 20:22:56 Waiting for dst bucket item count to get to: 5000 curr count: 1474
2021/12/08 20:23:01 Waiting for dst bucket item count to get to: 5000 curr count: 2980
2021/12/08 20:23:06 Waiting for dst bucket item count to get to: 5000 curr count: 4006
2021/12/08 20:23:06 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/08 20:23:11 Waiting for app: TestDeployUndeployLoopTimer to get deployed
2021/12/08 20:23:11 App: TestDeployUndeployLoopTimer got deployed
2021/12/08 20:23:11 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 20:23:11 Undeploying app: bucket_op_with_timer
2021/12/08 20:23:11 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 20:23:16 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/08 20:23:21 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/08 20:23:26 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/08 20:23:31 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/08 20:23:31 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/08 20:23:31 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 20:23:31 No eventing-con process running
2021/12/08 20:23:33 Context: Bucket flush, response code: 200 dump: 
2021/12/08 20:23:40 Context: Bucket flush, response code: 200 dump: 
2021/12/08 20:24:16 Deploying app: TestDeployUndeployLoopTimer. Item count src bucket: 0 dst bucket: 0
2021/12/08 20:24:17 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopTimer' in metakv",
  "warnings": null
 }
}
2021/12/08 20:24:17 Starting bucket ops to source bucket
2021/12/08 20:24:25 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 20:24:30 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 20:24:35 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 20:24:40 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 20:24:45 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 20:24:50 Waiting for dst bucket item count to get to: 5000 curr count: 436
2021/12/08 20:24:55 Waiting for dst bucket item count to get to: 5000 curr count: 436
2021/12/08 20:24:55 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/08 20:25:00 Waiting for app: TestDeployUndeployLoopTimer to get deployed
2021/12/08 20:25:00 App: TestDeployUndeployLoopTimer got deployed
2021/12/08 20:25:00 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 20:25:00 Undeploying app: bucket_op_with_timer
2021/12/08 20:25:00 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 20:25:05 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/08 20:25:10 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/08 20:25:15 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/08 20:25:20 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/08 20:25:20 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/08 20:25:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 20:25:20 No eventing-con process running
2021/12/08 20:25:21 Context: Bucket flush, response code: 200 dump: 
2021/12/08 20:25:29 Context: Bucket flush, response code: 200 dump: 
2021/12/08 20:26:04 Deploying app: TestDeployUndeployLoopTimer. Item count src bucket: 0 dst bucket: 0
2021/12/08 20:26:05 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopTimer' in metakv",
  "warnings": null
 }
}
2021/12/08 20:26:05 Starting bucket ops to source bucket
2021/12/08 20:26:13 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 20:26:18 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 20:26:23 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 20:26:28 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 20:26:33 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 20:26:38 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 20:26:43 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 20:26:43 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/08 20:26:48 Waiting for app: TestDeployUndeployLoopTimer to get deployed
2021/12/08 20:26:48 App: TestDeployUndeployLoopTimer got deployed
2021/12/08 20:26:48 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 20:26:48 Undeploying app: bucket_op_with_timer
2021/12/08 20:26:48 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 20:26:53 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/08 20:26:58 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/08 20:27:03 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/08 20:27:08 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/08 20:27:08 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/08 20:27:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 20:27:08 No eventing-con process running
2021/12/08 20:27:10 Context: Bucket flush, response code: 200 dump: 
2021/12/08 20:27:17 Context: Bucket flush, response code: 200 dump: 
2021/12/08 20:27:52 Deploying app: TestDeployUndeployLoopTimer. Item count src bucket: 0 dst bucket: 0
2021/12/08 20:27:53 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopTimer' in metakv",
  "warnings": null
 }
}
2021/12/08 20:27:53 Starting bucket ops to source bucket
2021/12/08 20:28:01 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 20:28:06 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 20:28:11 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 20:28:16 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 20:28:21 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 20:28:26 Waiting for dst bucket item count to get to: 5000 curr count: 152
2021/12/08 20:28:31 Waiting for dst bucket item count to get to: 5000 curr count: 766
2021/12/08 20:28:36 Waiting for dst bucket item count to get to: 5000 curr count: 3114
2021/12/08 20:28:36 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/08 20:28:41 Waiting for app: TestDeployUndeployLoopTimer to get deployed
2021/12/08 20:28:41 App: TestDeployUndeployLoopTimer got deployed
2021/12/08 20:28:41 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 20:28:41 Undeploying app: bucket_op_with_timer
2021/12/08 20:28:42 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 20:28:47 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/08 20:28:52 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/08 20:28:57 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/08 20:29:02 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/08 20:29:02 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/08 20:29:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 20:29:02 No eventing-con process running
2021/12/08 20:29:03 Context: Bucket flush, response code: 200 dump: 
2021/12/08 20:29:11 Context: Bucket flush, response code: 200 dump: 
2021/12/08 20:29:46 Function: TestDeployUndeployLoopTimer delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDeployUndeployLoopTimer deleting in the background"
}
--- PASS: TestDeployUndeployLoopTimer (580.47s)
=== RUN   TestMultipleHandlers
2021/12/08 20:29:56 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get un-deployed
2021/12/08 20:29:56 App: TestMultipleHandlers_bucket_op_on_update got un-deployed
2021/12/08 20:29:56 No eventing-con process running
2021/12/08 20:29:56 Function: TestMultipleHandlers_bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMultipleHandlers_bucket_op_on_update not found"
 }
}
2021/12/08 20:29:57 Context: Bucket flush, response code: 200 dump: 
2021/12/08 20:30:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 20:30:04 Context: Bucket flush, response code: 200 dump: 
2021/12/08 20:30:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 20:30:14 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get un-deployed
2021/12/08 20:30:14 App: TestMultipleHandlers_bucket_op_on_delete got un-deployed
2021/12/08 20:30:14 No eventing-con process running
2021/12/08 20:30:14 Function: TestMultipleHandlers_bucket_op_on_delete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMultipleHandlers_bucket_op_on_delete not found"
 }
}
2021/12/08 20:30:16 Context: Bucket flush, response code: 200 dump: 
2021/12/08 20:30:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 20:30:23 Context: Bucket flush, response code: 200 dump: 
2021/12/08 20:30:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 20:30:28 Deploying app: TestMultipleHandlers_bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/08 20:30:29 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMultipleHandlers_bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/08 20:30:29 Deploying app: TestMultipleHandlers_bucket_op_on_delete. Item count src bucket: 0 dst bucket: 0
2021/12/08 20:30:30 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMultipleHandlers_bucket_op_on_delete' in metakv",
  "warnings": null
 }
}
2021/12/08 20:30:35 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get deployed
2021/12/08 20:30:40 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get deployed
2021/12/08 20:30:45 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get deployed
2021/12/08 20:30:50 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get deployed
2021/12/08 20:30:50 App: TestMultipleHandlers_bucket_op_on_update got deployed
2021/12/08 20:30:55 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get deployed
2021/12/08 20:31:00 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get deployed
2021/12/08 20:31:05 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get deployed
2021/12/08 20:31:05 App: TestMultipleHandlers_bucket_op_on_delete got deployed
2021/12/08 20:31:05 Starting bucket ops to source bucket
2021/12/08 20:31:15 Waiting for dst bucket item count to get to: 5000 curr count: 937
2021/12/08 20:31:20 Waiting for dst bucket item count to get to: 5000 curr count: 2987
2021/12/08 20:31:25 Waiting for dst bucket item count to get to: 5000 curr count: 4550
2021/12/08 20:31:25 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/08 20:31:25 Starting bucket ops to source bucket
2021/12/08 20:31:40 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/08 20:31:45 Waiting for dst bucket item count to get to: 10000 curr count: 7504
2021/12/08 20:31:45 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/08 20:31:45 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 20:31:45 Function: TestMultipleHandlers_bucket_op_on_update update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/08 20:31:46 Function: TestMultipleHandlers_bucket_op_on_delete update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/08 20:31:46 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 20:31:51 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get un-deployed
2021/12/08 20:31:56 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get un-deployed
2021/12/08 20:32:01 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get un-deployed
2021/12/08 20:32:06 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get un-deployed
2021/12/08 20:32:06 App: TestMultipleHandlers_bucket_op_on_update got un-deployed
2021/12/08 20:32:06 No eventing-con process running
2021/12/08 20:32:06 Function: TestMultipleHandlers_bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMultipleHandlers_bucket_op_on_update deleting in the background"
}
2021/12/08 20:32:07 Context: Bucket flush, response code: 200 dump: 
2021/12/08 20:32:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 20:32:15 Context: Bucket flush, response code: 200 dump: 
2021/12/08 20:32:25 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/08 20:32:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 20:32:25 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 20:32:30 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get un-deployed
2021/12/08 20:32:35 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get un-deployed
2021/12/08 20:32:40 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get un-deployed
2021/12/08 20:32:45 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get un-deployed
2021/12/08 20:32:45 App: TestMultipleHandlers_bucket_op_on_delete got un-deployed
2021/12/08 20:32:45 No eventing-con process running
2021/12/08 20:32:45 Function: TestMultipleHandlers_bucket_op_on_delete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMultipleHandlers_bucket_op_on_delete deleting in the background"
}
2021/12/08 20:32:47 Context: Bucket flush, response code: 200 dump: 
2021/12/08 20:32:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 20:32:53 Context: Bucket flush, response code: 200 dump: 
2021/12/08 20:32:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestMultipleHandlers (192.81s)
=== RUN   TestPauseResumeLoopDefaultSettings
2021/12/08 20:33:08 Waiting for app: TestPauseResumeLoopDefaultSettings to get un-deployed
2021/12/08 20:33:08 App: TestPauseResumeLoopDefaultSettings got un-deployed
2021/12/08 20:33:08 No eventing-con process running
2021/12/08 20:33:08 Function: TestPauseResumeLoopDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestPauseResumeLoopDefaultSettings not found"
 }
}
2021/12/08 20:33:10 Context: Bucket flush, response code: 200 dump: 
2021/12/08 20:33:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 20:33:17 Context: Bucket flush, response code: 200 dump: 
2021/12/08 20:33:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 20:33:22 Deploying app: TestPauseResumeLoopDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/08 20:33:23 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestPauseResumeLoopDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/08 20:33:23 Starting bucket ops to source bucket
2021/12/08 20:33:31 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 20:33:36 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 20:33:41 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 20:33:46 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 20:33:46 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/08 20:33:46 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 20:33:51 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to deployed
2021/12/08 20:33:51 Function: TestPauseResumeLoopDefaultSettings status changed to deployed
2021/12/08 20:33:51 Pausing app: bucket_op_on_update
2021/12/08 20:33:51 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/08 20:33:56 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to paused
2021/12/08 20:33:56 Function: TestPauseResumeLoopDefaultSettings status changed to paused
2021/12/08 20:33:56 Resuming app: bucket_op_on_update
2021/12/08 20:33:57 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/08 20:33:57 Starting bucket ops to source bucket
2021/12/08 20:34:05 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/08 20:34:10 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/08 20:34:15 Waiting for dst bucket item count to get to: 10000 curr count: 7500
2021/12/08 20:34:15 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/08 20:34:15 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 20:34:20 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to deployed
2021/12/08 20:34:20 Function: TestPauseResumeLoopDefaultSettings status changed to deployed
2021/12/08 20:34:20 Pausing app: bucket_op_on_update
2021/12/08 20:34:21 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/08 20:34:26 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to paused
2021/12/08 20:34:26 Function: TestPauseResumeLoopDefaultSettings status changed to paused
2021/12/08 20:34:26 Resuming app: bucket_op_on_update
2021/12/08 20:34:27 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/08 20:34:27 Starting bucket ops to source bucket
2021/12/08 20:34:35 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/08 20:34:40 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/08 20:34:45 Waiting for dst bucket item count to get to: 15000 curr count: 12500
2021/12/08 20:34:45 src & dst bucket item count matched up. src bucket count: 15000 dst bucket count: 15000
2021/12/08 20:34:45 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 20:34:50 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to deployed
2021/12/08 20:34:50 Function: TestPauseResumeLoopDefaultSettings status changed to deployed
2021/12/08 20:34:50 Pausing app: bucket_op_on_update
2021/12/08 20:34:50 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/08 20:34:55 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to paused
2021/12/08 20:34:55 Function: TestPauseResumeLoopDefaultSettings status changed to paused
2021/12/08 20:34:55 Resuming app: bucket_op_on_update
2021/12/08 20:34:56 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/08 20:34:56 Starting bucket ops to source bucket
2021/12/08 20:35:04 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/08 20:35:09 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/08 20:35:14 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/08 20:35:14 src & dst bucket item count matched up. src bucket count: 20000 dst bucket count: 20000
2021/12/08 20:35:14 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 20:35:19 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to deployed
2021/12/08 20:35:19 Function: TestPauseResumeLoopDefaultSettings status changed to deployed
2021/12/08 20:35:19 Pausing app: bucket_op_on_update
2021/12/08 20:35:19 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/08 20:35:24 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to paused
2021/12/08 20:35:24 Function: TestPauseResumeLoopDefaultSettings status changed to paused
2021/12/08 20:35:24 Resuming app: bucket_op_on_update
2021/12/08 20:35:25 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/08 20:35:25 Starting bucket ops to source bucket
2021/12/08 20:35:33 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/08 20:35:38 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/08 20:35:43 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/08 20:35:48 Waiting for dst bucket item count to get to: 25000 curr count: 22500
2021/12/08 20:35:48 src & dst bucket item count matched up. src bucket count: 25000 dst bucket count: 25000
2021/12/08 20:35:48 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 20:35:53 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to deployed
2021/12/08 20:35:53 Function: TestPauseResumeLoopDefaultSettings status changed to deployed
2021/12/08 20:35:53 Pausing app: bucket_op_on_update
2021/12/08 20:35:53 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/08 20:35:58 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to paused
2021/12/08 20:35:58 Function: TestPauseResumeLoopDefaultSettings status changed to paused
2021/12/08 20:35:58 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 20:36:03 Waiting for app: TestPauseResumeLoopDefaultSettings to get un-deployed
2021/12/08 20:36:09 Waiting for app: TestPauseResumeLoopDefaultSettings to get un-deployed
2021/12/08 20:36:14 Waiting for app: TestPauseResumeLoopDefaultSettings to get un-deployed
2021/12/08 20:36:19 Waiting for app: TestPauseResumeLoopDefaultSettings to get un-deployed
2021/12/08 20:36:19 App: TestPauseResumeLoopDefaultSettings got un-deployed
2021/12/08 20:36:19 No eventing-con process running
2021/12/08 20:36:19 Function: TestPauseResumeLoopDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestPauseResumeLoopDefaultSettings deleting in the background"
}
2021/12/08 20:36:20 Context: Bucket flush, response code: 200 dump: 
2021/12/08 20:36:30 Waiting for dst bucket item count to get to: 0 curr count: 18859
2021/12/08 20:36:35 Waiting for dst bucket item count to get to: 0 curr count: 18859
2021/12/08 20:36:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 20:36:37 Context: Bucket flush, response code: 200 dump: 
2021/12/08 20:36:47 Waiting for dst bucket item count to get to: 0 curr count: 25000
2021/12/08 20:36:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestPauseResumeLoopDefaultSettings (228.80s)
=== RUN   TestPauseResumeLoopNonDefaultSettings
2021/12/08 20:36:57 Waiting for app: TestPauseResumeLoopNonDefaultSettings to get un-deployed
2021/12/08 20:36:57 App: TestPauseResumeLoopNonDefaultSettings got un-deployed
2021/12/08 20:36:57 No eventing-con process running
2021/12/08 20:36:57 Function: TestPauseResumeLoopNonDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestPauseResumeLoopNonDefaultSettings not found"
 }
}
2021/12/08 20:36:59 Context: Bucket flush, response code: 200 dump: 
2021/12/08 20:37:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 20:37:06 Context: Bucket flush, response code: 200 dump: 
2021/12/08 20:37:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 20:37:11 Deploying app: TestPauseResumeLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/08 20:37:12 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestPauseResumeLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/08 20:37:12 Starting bucket ops to source bucket
2021/12/08 20:37:20 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 20:37:25 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 20:37:30 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 20:37:35 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 20:37:40 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 20:37:45 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 20:37:50 Waiting for dst bucket item count to get to: 5000 curr count: 3084
2021/12/08 20:37:55 Waiting for dst bucket item count to get to: 5000 curr count: 3084
2021/12/08 20:37:55 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/08 20:37:55 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 20:38:00 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to deployed
2021/12/08 20:38:00 Function: TestPauseResumeLoopNonDefaultSettings status changed to deployed
2021/12/08 20:38:00 Pausing app: bucket_op_with_timer
2021/12/08 20:38:00 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/08 20:38:05 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to paused
2021/12/08 20:38:05 Function: TestPauseResumeLoopNonDefaultSettings status changed to paused
2021/12/08 20:38:05 Resuming app: bucket_op_with_timer
2021/12/08 20:38:06 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/08 20:38:06 Starting bucket ops to source bucket
2021/12/08 20:38:14 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/08 20:38:19 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/08 20:38:24 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/08 20:38:29 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/08 20:38:34 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/08 20:38:39 Waiting for dst bucket item count to get to: 10000 curr count: 8440
2021/12/08 20:38:44 Waiting for dst bucket item count to get to: 10000 curr count: 8440
2021/12/08 20:38:44 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/08 20:38:44 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 20:38:49 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to deployed
2021/12/08 20:38:49 Function: TestPauseResumeLoopNonDefaultSettings status changed to deployed
2021/12/08 20:38:49 Pausing app: bucket_op_with_timer
2021/12/08 20:38:50 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/08 20:38:55 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to paused
2021/12/08 20:38:55 Function: TestPauseResumeLoopNonDefaultSettings status changed to paused
2021/12/08 20:38:55 Resuming app: bucket_op_with_timer
2021/12/08 20:38:56 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/08 20:38:56 Starting bucket ops to source bucket
2021/12/08 20:39:04 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/08 20:39:09 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/08 20:39:14 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/08 20:39:19 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/08 20:39:24 Waiting for dst bucket item count to get to: 15000 curr count: 11795
2021/12/08 20:39:29 Waiting for dst bucket item count to get to: 15000 curr count: 14295
2021/12/08 20:39:29 src & dst bucket item count matched up. src bucket count: 15000 dst bucket count: 15000
2021/12/08 20:39:29 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 20:39:34 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to deployed
2021/12/08 20:39:34 Function: TestPauseResumeLoopNonDefaultSettings status changed to deployed
2021/12/08 20:39:34 Pausing app: bucket_op_with_timer
2021/12/08 20:39:34 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/08 20:39:39 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to paused
2021/12/08 20:39:39 Function: TestPauseResumeLoopNonDefaultSettings status changed to paused
2021/12/08 20:39:39 Resuming app: bucket_op_with_timer
2021/12/08 20:39:40 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/08 20:39:40 Starting bucket ops to source bucket
2021/12/08 20:39:48 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/08 20:39:53 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/08 20:39:58 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/08 20:40:03 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/08 20:40:08 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/08 20:40:13 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/08 20:40:18 Waiting for dst bucket item count to get to: 20000 curr count: 17500
2021/12/08 20:40:18 src & dst bucket item count matched up. src bucket count: 20000 dst bucket count: 20000
2021/12/08 20:40:18 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 20:40:23 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to deployed
2021/12/08 20:40:23 Function: TestPauseResumeLoopNonDefaultSettings status changed to deployed
2021/12/08 20:40:23 Pausing app: bucket_op_with_timer
2021/12/08 20:40:23 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/08 20:40:28 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to paused
2021/12/08 20:40:28 Function: TestPauseResumeLoopNonDefaultSettings status changed to paused
2021/12/08 20:40:28 Resuming app: bucket_op_with_timer
2021/12/08 20:40:29 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/08 20:40:29 Starting bucket ops to source bucket
2021/12/08 20:40:37 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/08 20:40:42 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/08 20:40:47 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/08 20:40:52 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/08 20:40:57 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/08 20:41:02 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/08 20:41:07 Waiting for dst bucket item count to get to: 25000 curr count: 22500
2021/12/08 20:41:07 src & dst bucket item count matched up. src bucket count: 25000 dst bucket count: 25000
2021/12/08 20:41:07 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 20:41:12 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to deployed
2021/12/08 20:41:12 Function: TestPauseResumeLoopNonDefaultSettings status changed to deployed
2021/12/08 20:41:12 Pausing app: bucket_op_with_timer
2021/12/08 20:41:12 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/08 20:41:17 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to paused
2021/12/08 20:41:17 Function: TestPauseResumeLoopNonDefaultSettings status changed to paused
2021/12/08 20:41:17 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 20:41:22 Waiting for app: TestPauseResumeLoopNonDefaultSettings to get un-deployed
2021/12/08 20:41:27 Waiting for app: TestPauseResumeLoopNonDefaultSettings to get un-deployed
2021/12/08 20:41:32 Waiting for app: TestPauseResumeLoopNonDefaultSettings to get un-deployed
2021/12/08 20:41:37 Waiting for app: TestPauseResumeLoopNonDefaultSettings to get un-deployed
2021/12/08 20:41:38 App: TestPauseResumeLoopNonDefaultSettings got un-deployed
2021/12/08 20:41:38 No eventing-con process running
2021/12/08 20:41:38 Function: TestPauseResumeLoopNonDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestPauseResumeLoopNonDefaultSettings deleting in the background"
}
2021/12/08 20:41:39 Context: Bucket flush, response code: 200 dump: 
2021/12/08 20:41:49 Waiting for dst bucket item count to get to: 0 curr count: 25000
2021/12/08 20:41:54 Waiting for dst bucket item count to get to: 0 curr count: 12500
2021/12/08 20:41:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 20:41:56 Context: Bucket flush, response code: 200 dump: 
2021/12/08 20:42:06 Waiting for dst bucket item count to get to: 0 curr count: 25000
2021/12/08 20:42:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestPauseResumeLoopNonDefaultSettings (318.83s)
=== RUN   TestPauseAndResumeWithWorkerCountChange
2021/12/08 20:42:16 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get un-deployed
2021/12/08 20:42:16 App: TestPauseAndResumeWithWorkerCountChange got un-deployed
2021/12/08 20:42:16 No eventing-con process running
2021/12/08 20:42:16 Function: TestPauseAndResumeWithWorkerCountChange delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestPauseAndResumeWithWorkerCountChange not found"
 }
}
2021/12/08 20:42:18 Context: Bucket flush, response code: 200 dump: 
2021/12/08 20:42:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 20:42:25 Context: Bucket flush, response code: 200 dump: 
2021/12/08 20:42:30 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 20:42:30 Deploying app: TestPauseAndResumeWithWorkerCountChange. Item count src bucket: 0 dst bucket: 0
2021/12/08 20:42:31 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestPauseAndResumeWithWorkerCountChange' in metakv",
  "warnings": null
 }
}
2021/12/08 20:42:36 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get deployed
2021/12/08 20:42:41 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get deployed
2021/12/08 20:42:46 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get deployed
2021/12/08 20:42:51 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get deployed
2021/12/08 20:42:51 App: TestPauseAndResumeWithWorkerCountChange got deployed
2021/12/08 20:42:51 Starting bucket ops to source bucket
2021/12/08 20:43:00 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 20:43:05 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 20:43:10 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 20:43:15 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 20:43:20 Waiting for dst bucket item count to get to: 5000 curr count: 4187
2021/12/08 20:43:25 Waiting for dst bucket item count to get to: 5000 curr count: 4187
2021/12/08 20:43:25 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/08 20:43:25 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 20:43:25 Pausing app: bucket_op_with_timer
2021/12/08 20:43:25 Function: TestPauseAndResumeWithWorkerCountChange update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/08 20:43:30 Waiting for function: TestPauseAndResumeWithWorkerCountChange status to change to paused
2021/12/08 20:43:30 Function: TestPauseAndResumeWithWorkerCountChange status changed to paused
2021/12/08 20:43:30 Resuming app: bucket_op_with_timer
2021/12/08 20:43:32 Function: TestPauseAndResumeWithWorkerCountChange update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:6] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/08 20:43:32 Starting bucket ops to source bucket
2021/12/08 20:43:43 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/08 20:43:48 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/08 20:43:53 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/08 20:43:58 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/08 20:44:03 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/08 20:44:03 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/08 20:44:03 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 20:44:08 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get un-deployed
2021/12/08 20:44:13 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get un-deployed
2021/12/08 20:44:18 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get un-deployed
2021/12/08 20:44:23 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get un-deployed
2021/12/08 20:44:23 App: TestPauseAndResumeWithWorkerCountChange got un-deployed
2021/12/08 20:44:23 No eventing-con process running
2021/12/08 20:44:23 Function: TestPauseAndResumeWithWorkerCountChange delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestPauseAndResumeWithWorkerCountChange deleting in the background"
}
2021/12/08 20:44:25 Context: Bucket flush, response code: 200 dump: 
2021/12/08 20:44:35 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/08 20:44:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 20:44:37 Context: Bucket flush, response code: 200 dump: 
2021/12/08 20:44:47 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/08 20:44:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestPauseAndResumeWithWorkerCountChange (160.73s)
=== RUN   TestPauseResumeWithEventingReb
2021/12/08 20:44:57 Waiting for app: TestPauseResumeWithEventingReb to get un-deployed
2021/12/08 20:44:57 App: TestPauseResumeWithEventingReb got un-deployed
2021/12/08 20:44:57 No eventing-con process running
2021/12/08 20:44:57 Function: TestPauseResumeWithEventingReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestPauseResumeWithEventingReb not found"
 }
}
2021/12/08 20:44:59 Context: Bucket flush, response code: 200 dump: 
2021/12/08 20:45:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 20:45:05 Context: Bucket flush, response code: 200 dump: 
2021/12/08 20:45:10 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 20:45:10 Deploying app: TestPauseResumeWithEventingReb. Item count src bucket: 0 dst bucket: 0
2021/12/08 20:45:12 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestPauseResumeWithEventingReb' in metakv",
  "warnings": null
 }
}
2021/12/08 20:45:17 Waiting for app: TestPauseResumeWithEventingReb to get deployed
2021/12/08 20:45:22 Waiting for app: TestPauseResumeWithEventingReb to get deployed
2021/12/08 20:45:27 Waiting for app: TestPauseResumeWithEventingReb to get deployed
2021/12/08 20:45:32 Waiting for app: TestPauseResumeWithEventingReb to get deployed
2021/12/08 20:45:32 App: TestPauseResumeWithEventingReb got deployed
2021/12/08 20:45:32 Starting bucket ops to source bucket
2021/12/08 20:45:41 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 20:45:46 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 20:45:51 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 20:45:56 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 20:46:01 Waiting for dst bucket item count to get to: 5000 curr count: 4566
2021/12/08 20:46:06 Waiting for dst bucket item count to get to: 5000 curr count: 4566
2021/12/08 20:46:06 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/08 20:46:06 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 20:46:06 Pausing app: bucket_op_with_timer
2021/12/08 20:46:06 Function: TestPauseResumeWithEventingReb update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/08 20:46:11 Waiting for function: TestPauseResumeWithEventingReb status to change to paused
2021/12/08 20:46:11 Function: TestPauseResumeWithEventingReb status changed to paused
2021/12/08 20:46:11 Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/08 20:46:11 Removing node(s): [] from the cluster
2021/12/08 20:46:16 Rebalance progress: 29.16666666666666
2021/12/08 20:46:21 Rebalance progress: 29.16666666666666
2021/12/08 20:46:26 Rebalance progress: 100
2021/12/08 20:46:26 Resuming app: bucket_op_with_timer
2021/12/08 20:46:27 Function: TestPauseResumeWithEventingReb update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:6] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/08 20:46:27 Starting bucket ops to source bucket
2021/12/08 20:46:39 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/08 20:46:44 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/08 20:46:49 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/08 20:46:54 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/08 20:46:59 Waiting for dst bucket item count to get to: 10000 curr count: 5345
2021/12/08 20:47:04 Waiting for dst bucket item count to get to: 10000 curr count: 5345
2021/12/08 20:47:04 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/08 20:47:04 Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/08 20:47:09 Rebalance progress: 29.16666666666666
2021/12/08 20:47:14 Rebalance progress: 29.16666666666666
2021/12/08 20:47:19 Rebalance progress: 100

VB distribution across eventing nodes and their workers::

2021/12/08 20:47:24 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:46} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/08 20:47:39 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:46} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/08 20:47:54 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:46} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/08 20:48:09 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:46} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/08 20:48:24 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:46} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/08 20:48:39 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:46} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/08 20:48:54 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:46} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/08 20:49:09 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:46} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/08 20:49:24 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:46} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/08 20:49:39 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:46} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/08 20:49:54 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:46} {Name:#primary Bucket:hello-world Completion:100}]

DCP Stream statuses:
	stream status: running
	len: 1024
	vb list dump: [0-1023]

vbucket curr owner:
Producer node: 127.0.0.1:9303	Node UUID: fea44f1e380744ec9b51e95c0d01a6dc
	workerID: worker_TestPauseResumeWithEventingReb_3
	len: 57
	v: [513-569]
	workerID: worker_TestPauseResumeWithEventingReb_4
	len: 57
	v: [570-626]
	workerID: worker_TestPauseResumeWithEventingReb_5
	len: 56
	v: [627-682]
	workerID: worker_TestPauseResumeWithEventingReb_0
	len: 57
	v: [342-398]
	workerID: worker_TestPauseResumeWithEventingReb_1
	len: 57
	v: [399-455]
	workerID: worker_TestPauseResumeWithEventingReb_2
	len: 57
	v: [456-512]
Producer node: 127.0.0.1:9302	Node UUID: 625a7c391dce83661389b9255eef38f4
	workerID: worker_TestPauseResumeWithEventingReb_0
	len: 57
	v: [0-56]
	workerID: worker_TestPauseResumeWithEventingReb_1
	len: 57
	v: [57-113]
	workerID: worker_TestPauseResumeWithEventingReb_2
	len: 57
	v: [114-170]
	workerID: worker_TestPauseResumeWithEventingReb_3
	len: 57
	v: [171-227]
	workerID: worker_TestPauseResumeWithEventingReb_4
	len: 57
	v: [228-284]
	workerID: worker_TestPauseResumeWithEventingReb_5
	len: 57
	v: [285-341]
Producer node: 172.31.2.134:9300	Node UUID: 7d48dccf03cc30ee41a2c09be793acce
	workerID: worker_TestPauseResumeWithEventingReb_3
	len: 57
	v: [854-910]
	workerID: worker_TestPauseResumeWithEventingReb_4
	len: 57
	v: [911-967]
	workerID: worker_TestPauseResumeWithEventingReb_5
	len: 56
	v: [968-1023]
	workerID: worker_TestPauseResumeWithEventingReb_0
	len: 57
	v: [683-739]
	workerID: worker_TestPauseResumeWithEventingReb_1
	len: 57
	v: [740-796]
	workerID: worker_TestPauseResumeWithEventingReb_2
	len: 57
	v: [797-853]


2021/12/08 20:50:09 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 20:50:10 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 20:50:15 clusterSetup.go:397: Waiting for app: TestPauseResumeWithEventingReb to get un-deployed
2021/12/08 20:50:20 clusterSetup.go:397: Waiting for app: TestPauseResumeWithEventingReb to get un-deployed
2021/12/08 20:50:25 clusterSetup.go:397: Waiting for app: TestPauseResumeWithEventingReb to get un-deployed
2021/12/08 20:50:30 clusterSetup.go:397: Waiting for app: TestPauseResumeWithEventingReb to get un-deployed
2021/12/08 20:50:30 clusterSetup.go:405: App: TestPauseResumeWithEventingReb got un-deployed
2021/12/08 20:50:30 eventingSetup.go:46: No eventing-con process running
2021/12/08 20:50:30 eventingSetup.go:418: Function: TestPauseResumeWithEventingReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestPauseResumeWithEventingReb deleting in the background"
}
2021/12/08 20:50:31 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/08 20:50:41 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/08 20:50:46 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/08 20:50:46 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 20:50:48 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/08 20:50:58 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 8042
2021/12/08 20:51:03 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3042
2021/12/08 20:51:03 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestPauseResumeWithEventingReb (376.45s)
=== RUN   TestBucketDeleteAfterPause
2021/12/08 20:51:13 clusterSetup.go:397: Waiting for app: TestBucketDeleteAfterPause to get un-deployed
2021/12/08 20:51:13 clusterSetup.go:405: App: TestBucketDeleteAfterPause got un-deployed
2021/12/08 20:51:13 eventingSetup.go:46: No eventing-con process running
2021/12/08 20:51:13 eventingSetup.go:418: Function: TestBucketDeleteAfterPause delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestBucketDeleteAfterPause not found"
 }
}
2021/12/08 20:51:15 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/08 20:51:20 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 20:51:22 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/08 20:51:27 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 20:51:27 eventingSetup.go:125: Deploying app: TestBucketDeleteAfterPause. Item count src bucket: 0 dst bucket: 0
2021/12/08 20:51:28 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestBucketDeleteAfterPause' in metakv",
  "warnings": null
 }
}
2021/12/08 20:51:33 clusterSetup.go:346: Waiting for app: TestBucketDeleteAfterPause to get deployed
2021/12/08 20:51:38 clusterSetup.go:346: Waiting for app: TestBucketDeleteAfterPause to get deployed
2021/12/08 20:51:43 clusterSetup.go:346: Waiting for app: TestBucketDeleteAfterPause to get deployed
2021/12/08 20:51:48 clusterSetup.go:346: Waiting for app: TestBucketDeleteAfterPause to get deployed
2021/12/08 20:51:48 clusterSetup.go:354: App: TestBucketDeleteAfterPause got deployed
2021/12/08 20:51:48 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/08 20:51:58 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 625
2021/12/08 20:52:03 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 3125
2021/12/08 20:52:08 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 3125
2021/12/08 20:52:08 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/08 20:52:08 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 20:52:08 handler_test.go:712: Pausing app: bucket_op_on_update
2021/12/08 20:52:08 eventingSetup.go:270: Function: TestBucketDeleteAfterPause update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/08 20:52:13 eventingSetup.go:719: Waiting for function: TestBucketDeleteAfterPause status to change to paused
2021/12/08 20:52:13 eventingSetup.go:727: Function: TestBucketDeleteAfterPause status changed to paused
2021/12/08 20:52:13 handler_test.go:716: Deleting source bucket: default
2021/12/08 20:52:19 eventingSetup.go:719: Waiting for function: TestBucketDeleteAfterPause status to change to undeployed
2021/12/08 20:52:19 eventingSetup.go:727: Function: TestBucketDeleteAfterPause status changed to undeployed
2021/12/08 20:52:24 clusterSetup.go:397: Waiting for app: TestBucketDeleteAfterPause to get un-deployed
2021/12/08 20:52:24 clusterSetup.go:405: App: TestBucketDeleteAfterPause got un-deployed
2021/12/08 20:52:24 eventingSetup.go:46: No eventing-con process running
2021/12/08 20:52:24 eventingSetup.go:418: Function: TestBucketDeleteAfterPause delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestBucketDeleteAfterPause deleting in the background"
}
2021/12/08 20:52:26 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/08 20:52:36 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/08 20:52:36 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 20:52:38 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/08 20:52:48 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/08 20:52:48 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestBucketDeleteAfterPause (104.32s)
=== RUN   TestChangeFnCodeBetweenPauseResume
2021/12/08 20:52:58 clusterSetup.go:397: Waiting for app: TestChangeFnCodeBetweenPauseResume to get un-deployed
2021/12/08 20:52:58 clusterSetup.go:405: App: TestChangeFnCodeBetweenPauseResume got un-deployed
2021/12/08 20:52:58 eventingSetup.go:46: No eventing-con process running
2021/12/08 20:52:58 eventingSetup.go:418: Function: TestChangeFnCodeBetweenPauseResume delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestChangeFnCodeBetweenPauseResume not found"
 }
}
2021/12/08 20:52:59 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/08 20:53:04 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 20:53:06 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/08 20:53:11 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 20:53:11 eventingSetup.go:125: Deploying app: TestChangeFnCodeBetweenPauseResume. Item count src bucket: 0 dst bucket: 0
2021/12/08 20:53:12 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestChangeFnCodeBetweenPauseResume' in metakv",
  "warnings": null
 }
}
2021/12/08 20:53:17 clusterSetup.go:346: Waiting for app: TestChangeFnCodeBetweenPauseResume to get deployed
2021/12/08 20:53:22 clusterSetup.go:346: Waiting for app: TestChangeFnCodeBetweenPauseResume to get deployed
2021/12/08 20:53:27 clusterSetup.go:346: Waiting for app: TestChangeFnCodeBetweenPauseResume to get deployed
2021/12/08 20:53:32 clusterSetup.go:346: Waiting for app: TestChangeFnCodeBetweenPauseResume to get deployed
2021/12/08 20:53:32 clusterSetup.go:354: App: TestChangeFnCodeBetweenPauseResume got deployed
2021/12/08 20:53:32 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/08 20:54:07 handler_test.go:739: Pausing function: TestChangeFnCodeBetweenPauseResume
2021/12/08 20:54:07 eventingSetup.go:270: Function: TestChangeFnCodeBetweenPauseResume update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/08 20:54:12 eventingSetup.go:719: Waiting for function: TestChangeFnCodeBetweenPauseResume status to change to paused
2021/12/08 20:54:12 eventingSetup.go:727: Function: TestChangeFnCodeBetweenPauseResume status changed to paused
2021/12/08 20:57:12 handler_test.go:748: Resuming function: TestChangeFnCodeBetweenPauseResume
2021/12/08 20:57:12 eventingSetup.go:125: Deploying app: TestChangeFnCodeBetweenPauseResume. Item count src bucket: 5000 dst bucket: 0
2021/12/08 20:57:13 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestChangeFnCodeBetweenPauseResume' in metakv",
  "warnings": null
 }
}
2021/12/08 20:57:18 eventingSetup.go:795: Waiting for function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter to get to 5000
2021/12/08 20:57:18 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 20:57:18 eventingSetup.go:799: Function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter got to 0 expected 5000
2021/12/08 20:57:23 eventingSetup.go:795: Waiting for function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter to get to 5000
2021/12/08 20:57:23 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 20:57:23 eventingSetup.go:799: Function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter got to 2912 expected 5000
2021/12/08 20:57:28 eventingSetup.go:795: Waiting for function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter to get to 5000
2021/12/08 20:57:28 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 20:57:28 eventingSetup.go:799: Function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter got to 3901 expected 5000
2021/12/08 20:57:33 eventingSetup.go:795: Waiting for function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter to get to 5000
2021/12/08 20:57:33 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 20:57:33 eventingSetup.go:804: Function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter got to 5000
2021/12/08 20:57:33 handler_test.go:753: Undeploying function: TestChangeFnCodeBetweenPauseResume
2021/12/08 20:57:33 eventingSetup.go:270: Function: TestChangeFnCodeBetweenPauseResume update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/08 20:57:43 clusterSetup.go:397: Waiting for app: TestChangeFnCodeBetweenPauseResume to get un-deployed
2021/12/08 20:57:48 clusterSetup.go:397: Waiting for app: TestChangeFnCodeBetweenPauseResume to get un-deployed
2021/12/08 20:57:53 clusterSetup.go:397: Waiting for app: TestChangeFnCodeBetweenPauseResume to get un-deployed
2021/12/08 20:57:53 clusterSetup.go:405: App: TestChangeFnCodeBetweenPauseResume got un-deployed
2021/12/08 20:57:53 eventingSetup.go:46: No eventing-con process running
2021/12/08 20:57:53 eventingSetup.go:418: Function: TestChangeFnCodeBetweenPauseResume delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestChangeFnCodeBetweenPauseResume deleting in the background"
}
2021/12/08 20:57:55 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/08 20:58:05 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/08 20:58:05 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 20:58:06 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/08 20:58:11 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestChangeFnCodeBetweenPauseResume (323.80s)
=== RUN   TestCommentUnCommentOnDelete
2021/12/08 20:58:21 clusterSetup.go:397: Waiting for app: TestCommentUnCommentOnDelete to get un-deployed
2021/12/08 20:58:21 clusterSetup.go:405: App: TestCommentUnCommentOnDelete got un-deployed
2021/12/08 20:58:21 eventingSetup.go:46: No eventing-con process running
2021/12/08 20:58:21 eventingSetup.go:418: Function: TestCommentUnCommentOnDelete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestCommentUnCommentOnDelete not found"
 }
}
2021/12/08 20:58:23 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/08 20:58:28 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 20:58:30 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/08 20:58:35 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 20:58:35 eventingSetup.go:125: Deploying app: TestCommentUnCommentOnDelete. Item count src bucket: 0 dst bucket: 0
2021/12/08 20:58:36 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCommentUnCommentOnDelete' in metakv",
  "warnings": null
 }
}
2021/12/08 20:58:41 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/08 20:58:46 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/08 20:58:51 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/08 20:58:56 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/08 20:58:56 clusterSetup.go:354: App: TestCommentUnCommentOnDelete got deployed
2021/12/08 20:58:56 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/08 20:59:05 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 20:59:10 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 3155
2021/12/08 20:59:15 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 3155
2021/12/08 20:59:15 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/08 20:59:15 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 20:59:15 handler_test.go:780: Undeploying app: TestCommentUnCommentOnDelete
2021/12/08 20:59:15 eventingSetup.go:270: Function: TestCommentUnCommentOnDelete update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/08 20:59:45 eventingSetup.go:125: Deploying app: TestCommentUnCommentOnDelete. Item count src bucket: 5000 dst bucket: 5000
2021/12/08 20:59:46 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCommentUnCommentOnDelete' in metakv",
  "warnings": null
 }
}
2021/12/08 20:59:51 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/08 20:59:56 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/08 21:00:01 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/08 21:00:06 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/08 21:00:06 clusterSetup.go:354: App: TestCommentUnCommentOnDelete got deployed
2021/12/08 21:00:06 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/08 21:00:19 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/08 21:00:24 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/08 21:00:24 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 21:00:24 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 21:00:24 handler_test.go:799: Undeploying app: TestCommentUnCommentOnDelete
2021/12/08 21:00:24 eventingSetup.go:270: Function: TestCommentUnCommentOnDelete update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/08 21:00:34 clusterSetup.go:397: Waiting for app: TestCommentUnCommentOnDelete to get un-deployed
2021/12/08 21:00:39 clusterSetup.go:397: Waiting for app: TestCommentUnCommentOnDelete to get un-deployed
2021/12/08 21:00:44 clusterSetup.go:397: Waiting for app: TestCommentUnCommentOnDelete to get un-deployed
2021/12/08 21:00:44 clusterSetup.go:405: App: TestCommentUnCommentOnDelete got un-deployed
2021/12/08 21:00:44 eventingSetup.go:46: No eventing-con process running
2021/12/08 21:00:44 eventingSetup.go:418: Function: TestCommentUnCommentOnDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCommentUnCommentOnDelete deleting in the background"
}
2021/12/08 21:00:45 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/08 21:00:50 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 21:00:52 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/08 21:00:57 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCommentUnCommentOnDelete (165.48s)
=== RUN   TestCPPWorkerCleanup
2021/12/08 21:01:07 clusterSetup.go:397: Waiting for app: TestCPPWorkerCleanup to get un-deployed
2021/12/08 21:01:07 clusterSetup.go:405: App: TestCPPWorkerCleanup got un-deployed
2021/12/08 21:01:07 eventingSetup.go:46: No eventing-con process running
2021/12/08 21:01:07 eventingSetup.go:418: Function: TestCPPWorkerCleanup delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestCPPWorkerCleanup not found"
 }
}
2021/12/08 21:01:08 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/08 21:01:13 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 21:01:15 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/08 21:01:20 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 21:01:20 eventingSetup.go:125: Deploying app: TestCPPWorkerCleanup. Item count src bucket: 0 dst bucket: 0
2021/12/08 21:01:21 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCPPWorkerCleanup' in metakv",
  "warnings": null
 }
}
2021/12/08 21:01:21 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/08 21:01:28 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 21:01:33 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 21:01:38 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 21:01:43 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 21:01:48 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 3936
2021/12/08 21:01:53 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 3936
2021/12/08 21:01:53 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/08 21:01:53 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 21:01:53 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 21:01:58 clusterSetup.go:397: Waiting for app: TestCPPWorkerCleanup to get un-deployed
2021/12/08 21:02:03 clusterSetup.go:397: Waiting for app: TestCPPWorkerCleanup to get un-deployed
2021/12/08 21:02:08 clusterSetup.go:397: Waiting for app: TestCPPWorkerCleanup to get un-deployed
2021/12/08 21:02:13 clusterSetup.go:397: Waiting for app: TestCPPWorkerCleanup to get un-deployed
2021/12/08 21:02:13 clusterSetup.go:405: App: TestCPPWorkerCleanup got un-deployed
2021/12/08 21:02:13 eventingSetup.go:46: No eventing-con process running
2021/12/08 21:02:13 eventingSetup.go:418: Function: TestCPPWorkerCleanup delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCPPWorkerCleanup deleting in the background"
}
2021/12/08 21:02:15 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/08 21:02:25 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/08 21:02:25 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 21:02:26 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/08 21:02:36 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/08 21:02:36 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCPPWorkerCleanup (129.59s)
=== RUN   TestWithUserXattrs
2021/12/08 21:03:16 clusterSetup.go:397: Waiting for app: TestWithUserXattrs to get un-deployed
2021/12/08 21:03:16 clusterSetup.go:405: App: TestWithUserXattrs got un-deployed
2021/12/08 21:03:16 eventingSetup.go:46: No eventing-con process running
2021/12/08 21:03:16 eventingSetup.go:418: Function: TestWithUserXattrs delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestWithUserXattrs not found"
 }
}
2021/12/08 21:03:18 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/08 21:03:23 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 21:03:25 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/08 21:03:30 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 21:03:30 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/08 21:03:31 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/08 21:03:33 eventingSetup.go:125: Deploying app: TestWithUserXattrs. Item count src bucket: 0 dst bucket: 0
2021/12/08 21:03:35 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestWithUserXattrs' in metakv",
  "warnings": null
 }
}
2021/12/08 21:03:40 clusterSetup.go:346: Waiting for app: TestWithUserXattrs to get deployed
2021/12/08 21:03:45 clusterSetup.go:346: Waiting for app: TestWithUserXattrs to get deployed
2021/12/08 21:03:50 clusterSetup.go:346: Waiting for app: TestWithUserXattrs to get deployed
2021/12/08 21:03:55 clusterSetup.go:346: Waiting for app: TestWithUserXattrs to get deployed
2021/12/08 21:03:55 clusterSetup.go:354: App: TestWithUserXattrs got deployed
2021/12/08 21:03:55 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/08 21:04:04 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 21:04:09 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 4021
2021/12/08 21:04:14 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 4021
2021/12/08 21:04:14 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/08 21:04:14 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 21:04:14 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 21:04:19 clusterSetup.go:397: Waiting for app: TestWithUserXattrs to get un-deployed
2021/12/08 21:04:24 clusterSetup.go:397: Waiting for app: TestWithUserXattrs to get un-deployed
2021/12/08 21:04:29 clusterSetup.go:397: Waiting for app: TestWithUserXattrs to get un-deployed
2021/12/08 21:04:34 clusterSetup.go:397: Waiting for app: TestWithUserXattrs to get un-deployed
2021/12/08 21:04:34 clusterSetup.go:405: App: TestWithUserXattrs got un-deployed
2021/12/08 21:04:34 eventingSetup.go:46: No eventing-con process running
2021/12/08 21:04:34 eventingSetup.go:418: Function: TestWithUserXattrs delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestWithUserXattrs deleting in the background"
}
2021/12/08 21:04:35 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/08 21:04:45 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/08 21:04:45 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 21:04:47 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/08 21:04:57 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/08 21:04:57 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestWithUserXattrs (110.75s)
=== RUN   TestEventProcessingPostBucketFlush
2021/12/08 21:05:07 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/08 21:05:07 clusterSetup.go:405: App: TestEventProcessingPostBucketFlush got un-deployed
2021/12/08 21:05:07 eventingSetup.go:46: No eventing-con process running
2021/12/08 21:05:07 eventingSetup.go:418: Function: TestEventProcessingPostBucketFlush delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventProcessingPostBucketFlush not found"
 }
}
2021/12/08 21:05:09 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/08 21:05:14 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 21:05:15 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/08 21:05:20 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 21:05:20 eventingSetup.go:125: Deploying app: TestEventProcessingPostBucketFlush. Item count src bucket: 0 dst bucket: 0
2021/12/08 21:05:21 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventProcessingPostBucketFlush' in metakv",
  "warnings": null
 }
}
2021/12/08 21:05:26 clusterSetup.go:346: Waiting for app: TestEventProcessingPostBucketFlush to get deployed
2021/12/08 21:05:31 clusterSetup.go:346: Waiting for app: TestEventProcessingPostBucketFlush to get deployed
2021/12/08 21:05:36 clusterSetup.go:346: Waiting for app: TestEventProcessingPostBucketFlush to get deployed
2021/12/08 21:05:41 clusterSetup.go:346: Waiting for app: TestEventProcessingPostBucketFlush to get deployed
2021/12/08 21:05:42 clusterSetup.go:354: App: TestEventProcessingPostBucketFlush got deployed
2021/12/08 21:05:42 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/08 21:05:50 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 21:05:55 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 21:05:55 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/08 21:05:55 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 21:05:59 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/08 21:06:09 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3455
2021/12/08 21:06:14 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 955
2021/12/08 21:06:14 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 21:06:16 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/08 21:06:26 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/08 21:06:26 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 21:06:26 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/08 21:06:51 eventingSetup.go:441: Waiting for dst bucket item count to get to: 25000 curr count: 15328
2021/12/08 21:06:56 eventingSetup.go:441: Waiting for dst bucket item count to get to: 25000 curr count: 15328
2021/12/08 21:06:56 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 25000 dst bucket count: 25000
2021/12/08 21:06:56 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 21:06:56 eventingSetup.go:270: Function: TestEventProcessingPostBucketFlush update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/08 21:07:01 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/08 21:07:06 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/08 21:07:11 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/08 21:07:16 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/08 21:07:16 clusterSetup.go:405: App: TestEventProcessingPostBucketFlush got un-deployed
2021/12/08 21:07:16 eventingSetup.go:46: No eventing-con process running
2021/12/08 21:07:21 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/08 21:07:21 clusterSetup.go:405: App: TestEventProcessingPostBucketFlush got un-deployed
2021/12/08 21:07:21 eventingSetup.go:46: No eventing-con process running
2021/12/08 21:07:21 eventingSetup.go:418: Function: TestEventProcessingPostBucketFlush delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventProcessingPostBucketFlush deleting in the background"
}
2021/12/08 21:07:23 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/08 21:07:33 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 25000
2021/12/08 21:07:38 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12500
2021/12/08 21:07:38 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 21:07:39 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/08 21:07:49 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12500
2021/12/08 21:07:54 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12500
2021/12/08 21:07:54 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventProcessingPostBucketFlush (177.22s)
=== RUN   TestMetaBucketDelete
2021/12/08 21:08:04 clusterSetup.go:397: Waiting for app: TestMetaBucketDelete to get un-deployed
2021/12/08 21:08:04 clusterSetup.go:405: App: TestMetaBucketDelete got un-deployed
2021/12/08 21:08:04 eventingSetup.go:46: No eventing-con process running
2021/12/08 21:08:04 eventingSetup.go:418: Function: TestMetaBucketDelete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMetaBucketDelete not found"
 }
}
2021/12/08 21:08:06 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/08 21:08:11 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 21:08:13 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/08 21:08:18 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 21:08:18 eventingSetup.go:125: Deploying app: TestMetaBucketDelete. Item count src bucket: 0 dst bucket: 0
2021/12/08 21:08:19 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMetaBucketDelete' in metakv",
  "warnings": null
 }
}
2021/12/08 21:08:24 clusterSetup.go:346: Waiting for app: TestMetaBucketDelete to get deployed
2021/12/08 21:08:29 clusterSetup.go:346: Waiting for app: TestMetaBucketDelete to get deployed
2021/12/08 21:08:34 clusterSetup.go:346: Waiting for app: TestMetaBucketDelete to get deployed
2021/12/08 21:08:39 clusterSetup.go:346: Waiting for app: TestMetaBucketDelete to get deployed
2021/12/08 21:08:39 clusterSetup.go:354: App: TestMetaBucketDelete got deployed
2021/12/08 21:08:39 handler_test.go:902: Deleting metadata bucket: eventing
2021/12/08 21:08:40 handler_test.go:904: Deleted metadata bucket: eventing
2021/12/08 21:08:45 clusterSetup.go:397: Waiting for app: TestMetaBucketDelete to get un-deployed
2021/12/08 21:08:45 clusterSetup.go:405: App: TestMetaBucketDelete got un-deployed
2021/12/08 21:09:00 clusterSetup.go:397: Waiting for app: TestMetaBucketDelete to get un-deployed
2021/12/08 21:09:00 clusterSetup.go:405: App: TestMetaBucketDelete got un-deployed
2021/12/08 21:09:00 eventingSetup.go:46: No eventing-con process running
2021/12/08 21:09:00 eventingSetup.go:418: Function: TestMetaBucketDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMetaBucketDelete deleting in the background"
}
2021/12/08 21:09:02 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/08 21:09:07 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 21:09:08 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/08 21:09:13 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestMetaBucketDelete (79.03s)
=== RUN   TestMetaBucketDeleteWithBootstrap
2021/12/08 21:09:23 clusterSetup.go:397: Waiting for app: TestMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/08 21:09:23 clusterSetup.go:405: App: TestMetaBucketDeleteWithBootstrap got un-deployed
2021/12/08 21:09:23 eventingSetup.go:46: No eventing-con process running
2021/12/08 21:09:23 eventingSetup.go:418: Function: TestMetaBucketDeleteWithBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMetaBucketDeleteWithBootstrap not found"
 }
}
2021/12/08 21:09:25 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/08 21:09:30 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 21:09:32 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/08 21:09:37 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 21:09:37 eventingSetup.go:125: Deploying app: TestMetaBucketDeleteWithBootstrap. Item count src bucket: 0 dst bucket: 0
2021/12/08 21:09:38 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMetaBucketDeleteWithBootstrap' in metakv",
  "warnings": null
 }
}
2021/12/08 21:09:57 handler_test.go:923: Deleting metadata bucket: eventing
2021/12/08 21:09:57 handler_test.go:925: Deleted metadata bucket: eventing
2021/12/08 21:10:12 clusterSetup.go:397: Waiting for app: TestMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/08 21:10:12 clusterSetup.go:405: App: TestMetaBucketDeleteWithBootstrap got un-deployed
2021/12/08 21:10:27 clusterSetup.go:397: Waiting for app: TestMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/08 21:10:27 clusterSetup.go:405: App: TestMetaBucketDeleteWithBootstrap got un-deployed
2021/12/08 21:10:27 eventingSetup.go:46: No eventing-con process running
2021/12/08 21:10:27 eventingSetup.go:418: Function: TestMetaBucketDeleteWithBootstrap delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMetaBucketDeleteWithBootstrap deleting in the background"
}
2021/12/08 21:10:28 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/08 21:10:33 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 21:10:35 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/08 21:10:40 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestMetaBucketDeleteWithBootstrap (86.85s)
=== RUN   TestSourceBucketDelete
2021/12/08 21:10:50 clusterSetup.go:397: Waiting for app: TestSourceBucketDelete to get un-deployed
2021/12/08 21:10:50 clusterSetup.go:405: App: TestSourceBucketDelete got un-deployed
2021/12/08 21:10:50 eventingSetup.go:46: No eventing-con process running
2021/12/08 21:10:50 eventingSetup.go:418: Function: TestSourceBucketDelete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestSourceBucketDelete not found"
 }
}
2021/12/08 21:10:52 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/08 21:10:57 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 21:10:58 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/08 21:11:03 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 21:11:03 eventingSetup.go:125: Deploying app: TestSourceBucketDelete. Item count src bucket: 0 dst bucket: 0
2021/12/08 21:11:05 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestSourceBucketDelete' in metakv",
  "warnings": null
 }
}
2021/12/08 21:11:10 clusterSetup.go:346: Waiting for app: TestSourceBucketDelete to get deployed
2021/12/08 21:11:15 clusterSetup.go:346: Waiting for app: TestSourceBucketDelete to get deployed
2021/12/08 21:11:20 clusterSetup.go:346: Waiting for app: TestSourceBucketDelete to get deployed
2021/12/08 21:11:25 clusterSetup.go:346: Waiting for app: TestSourceBucketDelete to get deployed
2021/12/08 21:11:25 clusterSetup.go:354: App: TestSourceBucketDelete got deployed
2021/12/08 21:11:25 handler_test.go:945: Deleting source bucket: default
2021/12/08 21:11:26 handler_test.go:947: Deleted source bucket: default
2021/12/08 21:11:31 clusterSetup.go:397: Waiting for app: TestSourceBucketDelete to get un-deployed
2021/12/08 21:11:31 clusterSetup.go:405: App: TestSourceBucketDelete got un-deployed
2021/12/08 21:11:46 clusterSetup.go:397: Waiting for app: TestSourceBucketDelete to get un-deployed
2021/12/08 21:11:46 clusterSetup.go:405: App: TestSourceBucketDelete got un-deployed
2021/12/08 21:11:46 eventingSetup.go:46: No eventing-con process running
2021/12/08 21:11:46 eventingSetup.go:418: Function: TestSourceBucketDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestSourceBucketDelete deleting in the background"
}
2021/12/08 21:11:47 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/08 21:11:52 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 21:11:54 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/08 21:11:59 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestSourceBucketDelete (78.66s)
=== RUN   TestSourceBucketDeleteWithBootstrap
2021/12/08 21:12:09 clusterSetup.go:397: Waiting for app: TestSourceBucketDeleteWithBootstrap to get un-deployed
2021/12/08 21:12:09 clusterSetup.go:405: App: TestSourceBucketDeleteWithBootstrap got un-deployed
2021/12/08 21:12:09 eventingSetup.go:46: No eventing-con process running
2021/12/08 21:12:09 eventingSetup.go:418: Function: TestSourceBucketDeleteWithBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestSourceBucketDeleteWithBootstrap not found"
 }
}
2021/12/08 21:12:10 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/08 21:12:15 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 21:12:17 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/08 21:12:22 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 21:12:22 eventingSetup.go:125: Deploying app: TestSourceBucketDeleteWithBootstrap. Item count src bucket: 0 dst bucket: 0
2021/12/08 21:12:23 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestSourceBucketDeleteWithBootstrap' in metakv",
  "warnings": null
 }
}
2021/12/08 21:12:42 handler_test.go:966: Deleting source bucket: default
2021/12/08 21:12:43 handler_test.go:968: Deleted source bucket: default
2021/12/08 21:12:58 clusterSetup.go:397: Waiting for app: TestSourceBucketDeleteWithBootstrap to get un-deployed
2021/12/08 21:12:58 clusterSetup.go:405: App: TestSourceBucketDeleteWithBootstrap got un-deployed
2021/12/08 21:13:13 clusterSetup.go:397: Waiting for app: TestSourceBucketDeleteWithBootstrap to get un-deployed
2021/12/08 21:13:13 clusterSetup.go:405: App: TestSourceBucketDeleteWithBootstrap got un-deployed
2021/12/08 21:13:13 eventingSetup.go:46: No eventing-con process running
2021/12/08 21:13:14 eventingSetup.go:418: Function: TestSourceBucketDeleteWithBootstrap delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestSourceBucketDeleteWithBootstrap deleting in the background"
}
2021/12/08 21:13:15 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/08 21:13:20 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 21:13:22 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/08 21:13:27 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestSourceBucketDeleteWithBootstrap (87.64s)
=== RUN   TestSourceAndMetaBucketDeleteWithBootstrap
2021/12/08 21:13:37 clusterSetup.go:397: Waiting for app: TestSourceAndMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/08 21:13:37 clusterSetup.go:405: App: TestSourceAndMetaBucketDeleteWithBootstrap got un-deployed
2021/12/08 21:13:37 eventingSetup.go:46: No eventing-con process running
2021/12/08 21:13:37 eventingSetup.go:418: Function: TestSourceAndMetaBucketDeleteWithBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestSourceAndMetaBucketDeleteWithBootstrap not found"
 }
}
2021/12/08 21:13:38 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/08 21:13:43 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 21:13:45 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/08 21:13:50 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 21:13:50 eventingSetup.go:125: Deploying app: TestSourceAndMetaBucketDeleteWithBootstrap. Item count src bucket: 0 dst bucket: 0
2021/12/08 21:13:51 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestSourceAndMetaBucketDeleteWithBootstrap' in metakv",
  "warnings": null
 }
}
2021/12/08 21:14:10 handler_test.go:988: Deleting source bucket: default
2021/12/08 21:14:11 handler_test.go:990: Deleted source bucket: default
2021/12/08 21:14:11 handler_test.go:992: Deleting metadata bucket: eventing
2021/12/08 21:14:12 handler_test.go:994: Deleted metadata bucket: eventing
2021/12/08 21:14:27 clusterSetup.go:397: Waiting for app: TestSourceAndMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/08 21:14:27 clusterSetup.go:405: App: TestSourceAndMetaBucketDeleteWithBootstrap got un-deployed
2021/12/08 21:14:42 clusterSetup.go:397: Waiting for app: TestSourceAndMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/08 21:14:42 clusterSetup.go:405: App: TestSourceAndMetaBucketDeleteWithBootstrap got un-deployed
2021/12/08 21:14:42 eventingSetup.go:46: No eventing-con process running
2021/12/08 21:14:42 eventingSetup.go:418: Function: TestSourceAndMetaBucketDeleteWithBootstrap delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestSourceAndMetaBucketDeleteWithBootstrap deleting in the background"
}
2021/12/08 21:14:43 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/08 21:14:48 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 21:14:50 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/08 21:14:55 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestSourceAndMetaBucketDeleteWithBootstrap (88.25s)
=== RUN   TestUndeployDuringBootstrap
2021/12/08 21:15:05 clusterSetup.go:397: Waiting for app: TestUndeployDuringBootstrap to get un-deployed
2021/12/08 21:15:05 clusterSetup.go:405: App: TestUndeployDuringBootstrap got un-deployed
2021/12/08 21:15:05 eventingSetup.go:46: No eventing-con process running
2021/12/08 21:15:05 eventingSetup.go:418: Function: TestUndeployDuringBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestUndeployDuringBootstrap not found"
 }
}
2021/12/08 21:15:06 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/08 21:15:11 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 21:15:13 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/08 21:15:18 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 21:15:18 eventingSetup.go:125: Deploying app: TestUndeployDuringBootstrap. Item count src bucket: 0 dst bucket: 0
2021/12/08 21:15:19 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestUndeployDuringBootstrap' in metakv",
  "warnings": null
 }
}
2021/12/08 21:15:19 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/08 21:15:31 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 21:15:31 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 21:15:31 clusterSetup.go:375: Apps undergoing bootstrap: true
2021/12/08 21:15:31 clusterSetup.go:378: App: TestUndeployDuringBootstrap started bootstrap
2021/12/08 21:15:31 clusterSetup.go:375: Apps undergoing bootstrap: true
2021/12/08 21:15:36 clusterSetup.go:367: No apps undergoing bootstrap
2021/12/08 21:15:41 clusterSetup.go:397: Waiting for app: TestUndeployDuringBootstrap to get un-deployed
2021/12/08 21:15:46 clusterSetup.go:397: Waiting for app: TestUndeployDuringBootstrap to get un-deployed
2021/12/08 21:15:51 clusterSetup.go:397: Waiting for app: TestUndeployDuringBootstrap to get un-deployed
2021/12/08 21:15:51 clusterSetup.go:405: App: TestUndeployDuringBootstrap got un-deployed
2021/12/08 21:15:51 eventingSetup.go:46: No eventing-con process running
2021/12/08 21:16:16 clusterSetup.go:397: Waiting for app: TestUndeployDuringBootstrap to get un-deployed
2021/12/08 21:16:16 clusterSetup.go:405: App: TestUndeployDuringBootstrap got un-deployed
2021/12/08 21:16:16 eventingSetup.go:46: No eventing-con process running
2021/12/08 21:16:16 eventingSetup.go:418: Function: TestUndeployDuringBootstrap delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestUndeployDuringBootstrap deleting in the background"
}
2021/12/08 21:16:17 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/08 21:16:27 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/08 21:16:27 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 21:16:29 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/08 21:16:39 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/08 21:16:44 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/08 21:16:44 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestUndeployDuringBootstrap (109.41s)
=== RUN   TestDeleteBeforeUndeploy
2021/12/08 21:16:49 eventingSetup.go:125: Deploying app: TestDeleteBeforeUndeploy. Item count src bucket: 0 dst bucket: 0
2021/12/08 21:16:50 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeleteBeforeUndeploy' in metakv",
  "warnings": null
 }
}
2021/12/08 21:16:55 clusterSetup.go:346: Waiting for app: TestDeleteBeforeUndeploy to get deployed
2021/12/08 21:17:00 clusterSetup.go:346: Waiting for app: TestDeleteBeforeUndeploy to get deployed
2021/12/08 21:17:05 clusterSetup.go:346: Waiting for app: TestDeleteBeforeUndeploy to get deployed
2021/12/08 21:17:10 clusterSetup.go:346: Waiting for app: TestDeleteBeforeUndeploy to get deployed
2021/12/08 21:17:10 clusterSetup.go:354: App: TestDeleteBeforeUndeploy got deployed
2021/12/08 21:17:10 eventingSetup.go:418: Function: TestDeleteBeforeUndeploy delete from main store request response code: 422 dump: {
 "name": "ERR_APP_NOT_UNDEPLOYED",
 "code": 22,
 "description": "Function hasn't undeployed",
 "attributes": null,
 "runtime_info": {
  "code": 22,
  "info": "Function: TestDeleteBeforeUndeploy skipping delete request from primary store, as it hasn't been undeployed"
 }
}
2021/12/08 21:17:11 eventingSetup.go:270: Function: TestDeleteBeforeUndeploy update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/08 21:17:11 eventingSetup.go:418: Function: TestDeleteBeforeUndeploy delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDeleteBeforeUndeploy deleting in the background"
}
2021/12/08 21:17:16 clusterSetup.go:397: Waiting for app: TestDeleteBeforeUndeploy to get un-deployed
2021/12/08 21:17:16 clusterSetup.go:405: App: TestDeleteBeforeUndeploy got un-deployed
2021/12/08 21:17:16 eventingSetup.go:46: No eventing-con process running
2021/12/08 21:17:16 eventingSetup.go:418: Function: TestDeleteBeforeUndeploy delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDeleteBeforeUndeploy not found"
 }
}
2021/12/08 21:17:17 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/08 21:17:22 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 21:17:24 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/08 21:17:29 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDeleteBeforeUndeploy (44.59s)
=== RUN   TestUndeployWhenTimersAreFired
2021/12/08 21:17:39 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/08 21:17:39 clusterSetup.go:405: App: TestUndeployWhenTimersAreFired got un-deployed
2021/12/08 21:17:39 eventingSetup.go:46: No eventing-con process running
2021/12/08 21:17:39 eventingSetup.go:418: Function: TestUndeployWhenTimersAreFired delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestUndeployWhenTimersAreFired not found"
 }
}
2021/12/08 21:17:40 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/08 21:17:45 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 21:17:47 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/08 21:17:52 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 21:17:52 eventingSetup.go:125: Deploying app: TestUndeployWhenTimersAreFired. Item count src bucket: 0 dst bucket: 0
2021/12/08 21:17:54 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestUndeployWhenTimersAreFired' in metakv",
  "warnings": null
 }
}
2021/12/08 21:17:59 clusterSetup.go:346: Waiting for app: TestUndeployWhenTimersAreFired to get deployed
2021/12/08 21:18:04 clusterSetup.go:346: Waiting for app: TestUndeployWhenTimersAreFired to get deployed
2021/12/08 21:18:09 clusterSetup.go:346: Waiting for app: TestUndeployWhenTimersAreFired to get deployed
2021/12/08 21:18:14 clusterSetup.go:346: Waiting for app: TestUndeployWhenTimersAreFired to get deployed
2021/12/08 21:18:14 clusterSetup.go:354: App: TestUndeployWhenTimersAreFired got deployed
2021/12/08 21:18:14 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/08 21:18:44 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 21:18:49 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/08 21:18:54 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/08 21:18:59 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/08 21:19:04 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/08 21:19:04 clusterSetup.go:405: App: TestUndeployWhenTimersAreFired got un-deployed
2021/12/08 21:19:04 eventingSetup.go:46: No eventing-con process running
2021/12/08 21:20:49 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/08 21:20:49 clusterSetup.go:405: App: TestUndeployWhenTimersAreFired got un-deployed
2021/12/08 21:20:49 eventingSetup.go:46: No eventing-con process running
2021/12/08 21:20:49 eventingSetup.go:418: Function: TestUndeployWhenTimersAreFired delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestUndeployWhenTimersAreFired deleting in the background"
}
2021/12/08 21:20:50 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/08 21:21:00 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4244
2021/12/08 21:21:05 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4244
2021/12/08 21:21:05 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 21:21:07 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/08 21:21:17 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/08 21:21:17 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestUndeployWhenTimersAreFired (228.33s)
=== RUN   TestUndeployWithKVFailover
2021/12/08 21:21:27 clusterSetup.go:397: Waiting for app: TestUndeployWithKVFailover to get un-deployed
2021/12/08 21:21:27 clusterSetup.go:405: App: TestUndeployWithKVFailover got un-deployed
2021/12/08 21:21:27 eventingSetup.go:46: No eventing-con process running
2021/12/08 21:21:27 eventingSetup.go:418: Function: TestUndeployWithKVFailover delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestUndeployWithKVFailover not found"
 }
}
2021/12/08 21:21:29 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/08 21:21:34 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 21:21:35 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/08 21:21:40 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 21:21:40 eventingSetup.go:125: Deploying app: TestUndeployWithKVFailover. Item count src bucket: 0 dst bucket: 0
2021/12/08 21:21:42 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestUndeployWithKVFailover' in metakv",
  "warnings": null
 }
}
2021/12/08 21:21:42 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/08 21:21:49 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 21:21:54 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 21:21:59 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 21:22:04 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 21:22:09 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 21:22:14 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 21:22:19 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 1338
2021/12/08 21:22:24 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 1338
2021/12/08 21:22:24 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/08 21:22:24 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: kv to the cluster
2021/12/08 21:22:24 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/08 21:22:29 Rebalance progress: 29.16666666666666
2021/12/08 21:22:34 Rebalance progress: 29.16666666666666
2021/12/08 21:22:39 Rebalance progress: 100
2021/12/08 21:22:39 Failing over: [127.0.0.1:19003]
2021/12/08 21:22:49 Function: TestUndeployWithKVFailover update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/08 21:23:49 Removing node(s): [] from the cluster
2021/12/08 21:23:54 Rebalance progress: 29.16666666666666
2021/12/08 21:23:59 Rebalance progress: 29.16666666666666
2021/12/08 21:24:04 Rebalance progress: 100
2021/12/08 21:24:09 Waiting for app: TestUndeployWithKVFailover to get un-deployed
2021/12/08 21:24:09 App: TestUndeployWithKVFailover got un-deployed
2021/12/08 21:24:09 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 21:24:14 Waiting for app: TestUndeployWithKVFailover to get un-deployed
2021/12/08 21:24:14 App: TestUndeployWithKVFailover got un-deployed
2021/12/08 21:24:14 No eventing-con process running
2021/12/08 21:24:14 Function: TestUndeployWithKVFailover delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestUndeployWithKVFailover deleting in the background"
}
2021/12/08 21:24:15 Context: Bucket flush, response code: 200 dump: 
2021/12/08 21:24:25 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/08 21:24:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 21:24:27 Context: Bucket flush, response code: 200 dump: 
2021/12/08 21:24:37 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/08 21:24:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestUndeployWithKVFailover (200.09s)
=== RUN   TestBucketFlushWhileFnDeployed
2021/12/08 21:24:47 Waiting for app: TestBucketFlushWhileFnDeployed to get un-deployed
2021/12/08 21:24:47 App: TestBucketFlushWhileFnDeployed got un-deployed
2021/12/08 21:24:47 No eventing-con process running
2021/12/08 21:24:47 Function: TestBucketFlushWhileFnDeployed delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestBucketFlushWhileFnDeployed not found"
 }
}
2021/12/08 21:24:49 Context: Bucket flush, response code: 200 dump: 
2021/12/08 21:24:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 21:24:55 Context: Bucket flush, response code: 200 dump: 
2021/12/08 21:25:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 21:25:01 Deploying app: TestBucketFlushWhileFnDeployed. Item count src bucket: 0 dst bucket: 0
2021/12/08 21:25:02 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestBucketFlushWhileFnDeployed' in metakv",
  "warnings": null
 }
}
2021/12/08 21:25:07 Waiting for app: TestBucketFlushWhileFnDeployed to get deployed
2021/12/08 21:25:12 Waiting for app: TestBucketFlushWhileFnDeployed to get deployed
2021/12/08 21:25:17 Waiting for app: TestBucketFlushWhileFnDeployed to get deployed
2021/12/08 21:25:22 Waiting for app: TestBucketFlushWhileFnDeployed to get deployed
2021/12/08 21:25:22 App: TestBucketFlushWhileFnDeployed got deployed
2021/12/08 21:25:22 Starting bucket ops to source bucket
2021/12/08 21:25:42 Waiting for dst bucket item count to get to: 20000 curr count: 8523
2021/12/08 21:25:42 src & dst bucket item count matched up. src bucket count: 20000 dst bucket count: 20000
2021/12/08 21:25:42 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 21:25:44 Context: Bucket flush, response code: 200 dump: 
2021/12/08 21:25:54 Waiting for dst bucket item count to get to: 0 curr count: 20000
2021/12/08 21:25:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 21:25:56 Context: Bucket flush, response code: 200 dump: 
2021/12/08 21:26:06 Waiting for dst bucket item count to get to: 0 curr count: 20000
2021/12/08 21:26:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 21:26:06 Starting bucket ops to source bucket
2021/12/08 21:26:19 Waiting for dst bucket item count to get to: 10000 curr count: 2233
2021/12/08 21:26:24 Waiting for dst bucket item count to get to: 10000 curr count: 2233
2021/12/08 21:26:24 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/08 21:26:24 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 21:26:25 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 21:26:30 Waiting for app: TestBucketFlushWhileFnDeployed to get un-deployed
2021/12/08 21:26:35 Waiting for app: TestBucketFlushWhileFnDeployed to get un-deployed
2021/12/08 21:26:40 Waiting for app: TestBucketFlushWhileFnDeployed to get un-deployed
2021/12/08 21:26:45 Waiting for app: TestBucketFlushWhileFnDeployed to get un-deployed
2021/12/08 21:26:45 App: TestBucketFlushWhileFnDeployed got un-deployed
2021/12/08 21:26:45 No eventing-con process running
2021/12/08 21:26:45 Function: TestBucketFlushWhileFnDeployed delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestBucketFlushWhileFnDeployed deleting in the background"
}
2021/12/08 21:26:46 Context: Bucket flush, response code: 200 dump: 
2021/12/08 21:26:56 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/08 21:26:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 21:26:58 Context: Bucket flush, response code: 200 dump: 
2021/12/08 21:27:08 Waiting for dst bucket item count to get to: 0 curr count: 5329
2021/12/08 21:27:13 Waiting for dst bucket item count to get to: 0 curr count: 329
2021/12/08 21:27:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestBucketFlushWhileFnDeployed (155.42s)
=== RUN   TestTimerOverwrite
2021/12/08 21:27:23 Waiting for app: TestTimerOverwrite to get un-deployed
2021/12/08 21:27:23 App: TestTimerOverwrite got un-deployed
2021/12/08 21:27:23 No eventing-con process running
2021/12/08 21:27:23 Function: TestTimerOverwrite delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestTimerOverwrite not found"
 }
}
2021/12/08 21:27:24 Context: Bucket flush, response code: 200 dump: 
2021/12/08 21:27:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 21:27:31 Context: Bucket flush, response code: 200 dump: 
2021/12/08 21:27:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 21:27:36 Deploying app: TestTimerOverwrite. Item count src bucket: 0 dst bucket: 0
2021/12/08 21:27:37 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestTimerOverwrite' in metakv",
  "warnings": null
 }
}
2021/12/08 21:27:42 Waiting for app: TestTimerOverwrite to get deployed
2021/12/08 21:27:47 Waiting for app: TestTimerOverwrite to get deployed
2021/12/08 21:27:52 Waiting for app: TestTimerOverwrite to get deployed
2021/12/08 21:27:57 Waiting for app: TestTimerOverwrite to get deployed
2021/12/08 21:27:57 App: TestTimerOverwrite got deployed
2021/12/08 21:28:07 Starting bucket ops to source bucket
2021/12/08 21:28:16 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 21:28:21 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 21:28:26 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 21:28:31 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 21:28:36 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 21:28:41 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 21:28:46 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 21:28:51 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 21:28:56 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 21:29:01 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 21:29:06 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 21:29:11 Waiting for dst bucket item count to get to: 5000 curr count: 524
2021/12/08 21:29:16 Waiting for dst bucket item count to get to: 5000 curr count: 524
2021/12/08 21:29:16 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/08 21:29:26 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 21:29:27 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 21:29:32 Waiting for app: TestTimerOverwrite to get un-deployed
2021/12/08 21:29:37 Waiting for app: TestTimerOverwrite to get un-deployed
2021/12/08 21:29:42 Waiting for app: TestTimerOverwrite to get un-deployed
2021/12/08 21:29:47 Waiting for app: TestTimerOverwrite to get un-deployed
2021/12/08 21:29:47 App: TestTimerOverwrite got un-deployed
2021/12/08 21:29:47 No eventing-con process running
2021/12/08 21:29:47 Function: TestTimerOverwrite delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestTimerOverwrite deleting in the background"
}
2021/12/08 21:29:48 Context: Bucket flush, response code: 200 dump: 
2021/12/08 21:29:58 Waiting for dst bucket item count to get to: 0 curr count: 3486
2021/12/08 21:30:03 Waiting for dst bucket item count to get to: 0 curr count: 986
2021/12/08 21:30:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 21:30:05 Context: Bucket flush, response code: 200 dump: 
2021/12/08 21:30:15 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/08 21:30:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestTimerOverwrite (182.02s)
=== RUN   TestUndeployBackdoorDuringBootstrap
2021/12/08 21:30:20 Adding node: https://127.0.0.1:19003 with role: kv to the cluster
2021/12/08 21:30:20 Removing node(s): [] from the cluster
2021/12/08 21:30:25 Rebalance progress: 100
2021/12/08 21:30:25 Failing over: [127.0.0.1:19003]
2021/12/08 21:30:30 Waiting for app: TestUndeployBackdoorDuringBootstrap to get un-deployed
2021/12/08 21:30:30 App: TestUndeployBackdoorDuringBootstrap got un-deployed
2021/12/08 21:30:30 No eventing-con process running
2021/12/08 21:30:30 Function: TestUndeployBackdoorDuringBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestUndeployBackdoorDuringBootstrap not found"
 }
}
2021/12/08 21:30:32 Context: Bucket flush, response code: 200 dump: 
2021/12/08 21:30:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 21:30:38 Context: Bucket flush, response code: 200 dump: 
2021/12/08 21:30:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 21:30:43 Deploying app: TestUndeployBackdoorDuringBootstrap. Item count src bucket: 0 dst bucket: 0
2021/12/08 21:30:45 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestUndeployBackdoorDuringBootstrap' in metakv",
  "warnings": null
 }
}
2021/12/08 21:30:45 Starting bucket ops to source bucket
2021/12/08 21:30:55 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 21:30:55 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 21:30:55 Context: Post to set retry counter, response code: 200 dump: 
2021/12/08 21:30:55 set retry counter, response &{[] }
2021/12/08 21:32:00 Waiting for app: TestUndeployBackdoorDuringBootstrap to get un-deployed
2021/12/08 21:32:00 App: TestUndeployBackdoorDuringBootstrap got un-deployed
2021/12/08 21:32:00 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 21:32:00 Function: TestUndeployBackdoorDuringBootstrap delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestUndeployBackdoorDuringBootstrap deleting in the background"
}
2021/12/08 21:32:05 Waiting for app: TestUndeployBackdoorDuringBootstrap to get un-deployed
2021/12/08 21:32:05 App: TestUndeployBackdoorDuringBootstrap got un-deployed
2021/12/08 21:32:05 No eventing-con process running
2021/12/08 21:32:05 Function: TestUndeployBackdoorDuringBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestUndeployBackdoorDuringBootstrap not found"
 }
}
2021/12/08 21:32:06 Context: Bucket flush, response code: 200 dump: 
2021/12/08 21:32:16 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/08 21:32:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 21:32:18 Context: Bucket flush, response code: 200 dump: 
2021/12/08 21:32:28 Waiting for dst bucket item count to get to: 0 curr count: 2733
2021/12/08 21:32:33 Waiting for dst bucket item count to get to: 0 curr count: 233
2021/12/08 21:32:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 21:32:33 Removing node(s): [] from the cluster
2021/12/08 21:32:38 Rebalance progress: 100
--- PASS: TestUndeployBackdoorDuringBootstrap (143.05s)
=== RUN   TestOnUpdateSrcMutation
2021/12/08 21:32:48 Waiting for app: TestOnUpdateSrcMutation to get un-deployed
2021/12/08 21:32:48 App: TestOnUpdateSrcMutation got un-deployed
2021/12/08 21:32:48 No eventing-con process running
2021/12/08 21:32:48 Function: TestOnUpdateSrcMutation delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateSrcMutation not found"
 }
}
2021/12/08 21:32:49 Context: Bucket flush, response code: 200 dump: 
2021/12/08 21:32:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 21:32:56 Context: Bucket flush, response code: 200 dump: 
2021/12/08 21:33:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 21:33:01 Deploying app: TestOnUpdateSrcMutation. Item count src bucket: 0 dst bucket: 0
2021/12/08 21:33:02 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateSrcMutation' in metakv",
  "warnings": null
 }
}
2021/12/08 21:33:02 Starting bucket ops to source bucket
2021/12/08 21:33:24 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/08 21:33:24 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 21:33:24 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 21:33:29 Waiting for app: TestOnUpdateSrcMutation to get un-deployed
2021/12/08 21:33:34 Waiting for app: TestOnUpdateSrcMutation to get un-deployed
2021/12/08 21:33:39 Waiting for app: TestOnUpdateSrcMutation to get un-deployed
2021/12/08 21:33:44 Waiting for app: TestOnUpdateSrcMutation to get un-deployed
2021/12/08 21:33:44 App: TestOnUpdateSrcMutation got un-deployed
2021/12/08 21:33:44 No eventing-con process running
2021/12/08 21:33:44 Function: TestOnUpdateSrcMutation delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateSrcMutation deleting in the background"
}
2021/12/08 21:33:45 Context: Bucket flush, response code: 200 dump: 
2021/12/08 21:33:55 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/08 21:33:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 21:33:57 Context: Bucket flush, response code: 200 dump: 
2021/12/08 21:34:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateSrcMutation (84.48s)
=== RUN   TestOnDeleteSrcMutation
2021/12/08 21:34:12 Waiting for app: TestOnDeleteSrcMutation to get un-deployed
2021/12/08 21:34:12 App: TestOnDeleteSrcMutation got un-deployed
2021/12/08 21:34:12 No eventing-con process running
2021/12/08 21:34:12 Function: TestOnDeleteSrcMutation delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnDeleteSrcMutation not found"
 }
}
2021/12/08 21:34:14 Context: Bucket flush, response code: 200 dump: 
2021/12/08 21:34:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 21:34:20 Context: Bucket flush, response code: 200 dump: 
2021/12/08 21:34:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 21:34:25 Deploying app: TestOnDeleteSrcMutation. Item count src bucket: 0 dst bucket: 0
2021/12/08 21:34:27 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnDeleteSrcMutation' in metakv",
  "warnings": null
 }
}
2021/12/08 21:34:27 Starting bucket ops to source bucket
2021/12/08 21:34:55 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/08 21:34:55 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 21:34:55 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 21:35:00 Waiting for app: TestOnDeleteSrcMutation to get un-deployed
2021/12/08 21:35:05 Waiting for app: TestOnDeleteSrcMutation to get un-deployed
2021/12/08 21:35:10 Waiting for app: TestOnDeleteSrcMutation to get un-deployed
2021/12/08 21:35:15 Waiting for app: TestOnDeleteSrcMutation to get un-deployed
2021/12/08 21:35:15 App: TestOnDeleteSrcMutation got un-deployed
2021/12/08 21:35:15 No eventing-con process running
2021/12/08 21:35:16 Function: TestOnDeleteSrcMutation delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnDeleteSrcMutation deleting in the background"
}
2021/12/08 21:35:17 Context: Bucket flush, response code: 200 dump: 
2021/12/08 21:35:27 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/08 21:35:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 21:35:29 Context: Bucket flush, response code: 200 dump: 
2021/12/08 21:35:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnDeleteSrcMutation (91.35s)
=== RUN   TestOnUpdateSrcMutationWithTimer
2021/12/08 21:35:44 Waiting for app: TestOnUpdateSrcMutationWithTimer to get un-deployed
2021/12/08 21:35:44 App: TestOnUpdateSrcMutationWithTimer got un-deployed
2021/12/08 21:35:44 No eventing-con process running
2021/12/08 21:35:44 Function: TestOnUpdateSrcMutationWithTimer delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateSrcMutationWithTimer not found"
 }
}
2021/12/08 21:35:45 Context: Bucket flush, response code: 200 dump: 
2021/12/08 21:35:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 21:35:52 Context: Bucket flush, response code: 200 dump: 
2021/12/08 21:35:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 21:35:57 Deploying app: TestOnUpdateSrcMutationWithTimer. Item count src bucket: 0 dst bucket: 0
2021/12/08 21:35:58 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateSrcMutationWithTimer' in metakv",
  "warnings": null
 }
}
2021/12/08 21:35:58 Starting bucket ops to source bucket
2021/12/08 21:36:35 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/08 21:36:35 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 21:36:35 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 21:36:40 Waiting for app: TestOnUpdateSrcMutationWithTimer to get un-deployed
2021/12/08 21:36:45 Waiting for app: TestOnUpdateSrcMutationWithTimer to get un-deployed
2021/12/08 21:36:50 Waiting for app: TestOnUpdateSrcMutationWithTimer to get un-deployed
2021/12/08 21:36:55 Waiting for app: TestOnUpdateSrcMutationWithTimer to get un-deployed
2021/12/08 21:36:55 App: TestOnUpdateSrcMutationWithTimer got un-deployed
2021/12/08 21:36:55 No eventing-con process running
2021/12/08 21:36:55 Function: TestOnUpdateSrcMutationWithTimer delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateSrcMutationWithTimer deleting in the background"
}
2021/12/08 21:36:56 Context: Bucket flush, response code: 200 dump: 
2021/12/08 21:37:06 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/08 21:37:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 21:37:08 Context: Bucket flush, response code: 200 dump: 
2021/12/08 21:37:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateSrcMutationWithTimer (99.51s)
=== RUN   TestOnDeleteSrcMutationsWithTimer
2021/12/08 21:37:23 Waiting for app: TestOnDeleteSrcMutationsWithTimer to get un-deployed
2021/12/08 21:37:23 App: TestOnDeleteSrcMutationsWithTimer got un-deployed
2021/12/08 21:37:23 No eventing-con process running
2021/12/08 21:37:23 Function: TestOnDeleteSrcMutationsWithTimer delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnDeleteSrcMutationsWithTimer not found"
 }
}
2021/12/08 21:37:24 Context: Bucket flush, response code: 200 dump: 
2021/12/08 21:37:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 21:37:31 Context: Bucket flush, response code: 200 dump: 
2021/12/08 21:37:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 21:37:36 Deploying app: TestOnDeleteSrcMutationsWithTimer. Item count src bucket: 0 dst bucket: 0
2021/12/08 21:37:37 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnDeleteSrcMutationsWithTimer' in metakv",
  "warnings": null
 }
}
2021/12/08 21:37:37 Starting bucket ops to source bucket
2021/12/08 21:38:16 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/08 21:38:16 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 21:38:16 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 21:38:21 Waiting for app: TestOnDeleteSrcMutationsWithTimer to get un-deployed
2021/12/08 21:38:26 Waiting for app: TestOnDeleteSrcMutationsWithTimer to get un-deployed
2021/12/08 21:38:31 Waiting for app: TestOnDeleteSrcMutationsWithTimer to get un-deployed
2021/12/08 21:38:36 Waiting for app: TestOnDeleteSrcMutationsWithTimer to get un-deployed
2021/12/08 21:38:36 App: TestOnDeleteSrcMutationsWithTimer got un-deployed
2021/12/08 21:38:36 No eventing-con process running
2021/12/08 21:38:36 Function: TestOnDeleteSrcMutationsWithTimer delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnDeleteSrcMutationsWithTimer deleting in the background"
}
2021/12/08 21:38:38 Context: Bucket flush, response code: 200 dump: 
2021/12/08 21:38:48 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/08 21:38:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 21:38:49 Context: Bucket flush, response code: 200 dump: 
2021/12/08 21:38:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnDeleteSrcMutationsWithTimer (101.31s)
=== RUN   TestInterHandlerRecursion
2021/12/08 21:39:04 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/08 21:39:04 App: TestInterHandlerRecursion_source_bucket_update_op got un-deployed
2021/12/08 21:39:04 No eventing-con process running
2021/12/08 21:39:04 Function: TestInterHandlerRecursion_source_bucket_update_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestInterHandlerRecursion_source_bucket_update_op not found"
 }
}
2021/12/08 21:39:06 Context: Bucket flush, response code: 200 dump: 
2021/12/08 21:39:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 21:39:13 Context: Bucket flush, response code: 200 dump: 
2021/12/08 21:39:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 21:39:23 Waiting for app: TestInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/08 21:39:23 App: TestInterHandlerRecursion_src_bucket_op_on_delete got un-deployed
2021/12/08 21:39:23 No eventing-con process running
2021/12/08 21:39:23 Function: TestInterHandlerRecursion_src_bucket_op_on_delete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestInterHandlerRecursion_src_bucket_op_on_delete not found"
 }
}
2021/12/08 21:39:24 Context: Bucket flush, response code: 200 dump: 
2021/12/08 21:39:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 21:39:31 Context: Bucket flush, response code: 200 dump: 
2021/12/08 21:39:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 21:39:36 Deploying app: TestInterHandlerRecursion_source_bucket_update_op. Item count src bucket: 0 dst bucket: 0
2021/12/08 21:39:37 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterHandlerRecursion_source_bucket_update_op' in metakv",
  "warnings": null
 }
}
2021/12/08 21:39:37 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterHandlerRecursion_source_bucket_update_op' in metakv",
  "warnings": null
 }
} err 
2021/12/08 21:39:42 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/08 21:39:47 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/08 21:39:52 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/08 21:39:57 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/08 21:39:57 App: TestInterHandlerRecursion_source_bucket_update_op got deployed
2021/12/08 21:39:57 Deploying app: TestInterHandlerRecursion_src_bucket_op_on_delete. Item count src bucket: 0 dst bucket: 0
2021/12/08 21:39:57 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_INTER_FUNCTION_RECURSION",
 "code": 50,
 "description": "Inter function recursion error, only one function is allowed to do source bucket mutation/delete on a bucket",
 "attributes": null,
 "runtime_info": {
  "code": 50,
  "info": "Inter handler recursion error"
 }
}
2021/12/08 21:39:57 response body {
 "name": "ERR_INTER_FUNCTION_RECURSION",
 "code": 50,
 "description": "Inter function recursion error, only one function is allowed to do source bucket mutation/delete on a bucket",
 "attributes": null,
 "runtime_info": {
  "code": 50,
  "info": "Inter handler recursion error"
 }
} err 
2021/12/08 21:39:57 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 21:40:02 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/08 21:40:07 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/08 21:40:12 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/08 21:40:17 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/08 21:40:17 App: TestInterHandlerRecursion_source_bucket_update_op got un-deployed
2021/12/08 21:40:17 No eventing-con process running
2021/12/08 21:40:17 Function: TestInterHandlerRecursion_source_bucket_update_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestInterHandlerRecursion_source_bucket_update_op deleting in the background"
}
2021/12/08 21:40:18 Context: Bucket flush, response code: 200 dump: 
2021/12/08 21:40:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 21:40:25 Context: Bucket flush, response code: 200 dump: 
2021/12/08 21:40:30 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 21:40:35 Waiting for app: TestInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/08 21:40:35 App: TestInterHandlerRecursion_src_bucket_op_on_delete got un-deployed
2021/12/08 21:40:35 No eventing-con process running
2021/12/08 21:40:35 Function: TestInterHandlerRecursion_src_bucket_op_on_delete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestInterHandlerRecursion_src_bucket_op_on_delete not found"
 }
}
2021/12/08 21:40:37 Context: Bucket flush, response code: 200 dump: 
2021/12/08 21:40:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 21:40:43 Context: Bucket flush, response code: 200 dump: 
2021/12/08 21:40:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestInterHandlerRecursion (113.95s)
=== RUN   TestInterBucketRecursion
2021/12/08 21:40:53 Deploying app: TestInterBucketRecursion_function1. Item count src bucket: 0 dst bucket: 0
2021/12/08 21:40:54 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterBucketRecursion_function1' in metakv",
  "warnings": null
 }
}
2021/12/08 21:40:54 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterBucketRecursion_function1' in metakv",
  "warnings": null
 }
} err 
2021/12/08 21:40:55 Deploying app: TestInterBucketRecursion_function2. Item count src bucket: 0 dst bucket: 0
2021/12/08 21:40:55 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_INTER_BUCKET_RECURSION",
 "code": 51,
 "description": "Inter bucket recursion error, deployment of current handler will cause inter bucket recursion",
 "attributes": null,
 "runtime_info": {
  "code": 51,
  "info": "Inter bucket recursion error; function: TestInterBucketRecursion_function2 causes a cycle involving functions: [TestInterBucketRecursion_function1], hence deployment is disallowed"
 }
}
2021/12/08 21:41:00 Waiting for app: TestInterBucketRecursion_function1 to get deployed
2021/12/08 21:41:05 Waiting for app: TestInterBucketRecursion_function1 to get deployed
2021/12/08 21:41:10 Waiting for app: TestInterBucketRecursion_function1 to get deployed
2021/12/08 21:41:15 Waiting for app: TestInterBucketRecursion_function1 to get deployed
2021/12/08 21:41:15 App: TestInterBucketRecursion_function1 got deployed
2021/12/08 21:41:15 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 21:41:20 Waiting for app: TestInterBucketRecursion_function1 to get un-deployed
2021/12/08 21:41:25 Waiting for app: TestInterBucketRecursion_function1 to get un-deployed
2021/12/08 21:41:30 Waiting for app: TestInterBucketRecursion_function1 to get un-deployed
2021/12/08 21:41:35 Waiting for app: TestInterBucketRecursion_function1 to get un-deployed
2021/12/08 21:41:35 App: TestInterBucketRecursion_function1 got un-deployed
2021/12/08 21:41:35 No eventing-con process running
2021/12/08 21:41:35 Function: TestInterBucketRecursion_function1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestInterBucketRecursion_function1 deleting in the background"
}
2021/12/08 21:41:36 Context: Bucket flush, response code: 200 dump: 
2021/12/08 21:41:41 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 21:41:43 Context: Bucket flush, response code: 200 dump: 
2021/12/08 21:41:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 21:41:53 Waiting for app: TestInterBucketRecursion_function2 to get un-deployed
2021/12/08 21:41:53 App: TestInterBucketRecursion_function2 got un-deployed
2021/12/08 21:41:53 No eventing-con process running
2021/12/08 21:41:53 Function: TestInterBucketRecursion_function2 delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestInterBucketRecursion_function2 not found"
 }
}
2021/12/08 21:41:54 Context: Bucket flush, response code: 200 dump: 
2021/12/08 21:41:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 21:42:01 Context: Bucket flush, response code: 200 dump: 
2021/12/08 21:42:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestInterBucketRecursion (77.58s)
=== RUN   TestLargeHandler
2021/12/08 21:42:06 Changed force_compress value to false
2021/12/08 21:42:06 Deploying app: TestLargeHandler_function. Item count src bucket: 0 dst bucket: 0
2021/12/08 21:42:06 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_APPCODE_SIZE",
 "code": 39,
 "description": "Function Code size is more than the configured limit",
 "attributes": null,
 "runtime_info": {
  "code": 39,
  "info": "Function: TestLargeHandler_function handler Code size is more than 131072. Code Size: 131768"
 }
}
2021/12/08 21:42:06 Changed force_compress value to true
2021/12/08 21:42:06 Deploying app: TestLargeHandler_function. Item count src bucket: 0 dst bucket: 0
2021/12/08 21:42:08 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestLargeHandler_function' in metakv",
  "warnings": null
 }
}
2021/12/08 21:42:13 Waiting for app: TestLargeHandler_function to get deployed
2021/12/08 21:42:18 Waiting for app: TestLargeHandler_function to get deployed
2021/12/08 21:42:23 Waiting for app: TestLargeHandler_function to get deployed
2021/12/08 21:42:28 Waiting for app: TestLargeHandler_function to get deployed
2021/12/08 21:42:28 App: TestLargeHandler_function got deployed
2021/12/08 21:42:28 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 21:42:33 Waiting for app: TestLargeHandler_function to get un-deployed
2021/12/08 21:42:38 Waiting for app: TestLargeHandler_function to get un-deployed
2021/12/08 21:42:43 Waiting for app: TestLargeHandler_function to get un-deployed
2021/12/08 21:42:48 Waiting for app: TestLargeHandler_function to get un-deployed
2021/12/08 21:42:48 App: TestLargeHandler_function got un-deployed
2021/12/08 21:42:48 No eventing-con process running
2021/12/08 21:42:48 Function: TestLargeHandler_function delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestLargeHandler_function deleting in the background"
}
2021/12/08 21:42:50 Context: Bucket flush, response code: 200 dump: 
2021/12/08 21:42:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 21:42:56 Context: Bucket flush, response code: 200 dump: 
2021/12/08 21:43:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestLargeHandler (55.43s)
=== RUN   TestAllowInterHandlerRecursion
2021/12/08 21:43:11 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/08 21:43:11 App: TestAllowInterHandlerRecursion_source_bucket_update_op got un-deployed
2021/12/08 21:43:11 No eventing-con process running
2021/12/08 21:43:11 Function: TestAllowInterHandlerRecursion_source_bucket_update_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestAllowInterHandlerRecursion_source_bucket_update_op not found"
 }
}
2021/12/08 21:43:13 Context: Bucket flush, response code: 200 dump: 
2021/12/08 21:43:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 21:43:19 Context: Bucket flush, response code: 200 dump: 
2021/12/08 21:43:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 21:43:29 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/08 21:43:29 App: TestAllowInterHandlerRecursion_src_bucket_op_on_delete got un-deployed
2021/12/08 21:43:30 No eventing-con process running
2021/12/08 21:43:30 Function: TestAllowInterHandlerRecursion_src_bucket_op_on_delete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestAllowInterHandlerRecursion_src_bucket_op_on_delete not found"
 }
}
2021/12/08 21:43:31 Context: Bucket flush, response code: 200 dump: 
2021/12/08 21:43:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 21:43:38 Context: Bucket flush, response code: 200 dump: 
2021/12/08 21:43:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 21:43:43 Deploying app: TestAllowInterHandlerRecursion_source_bucket_update_op. Item count src bucket: 0 dst bucket: 0
2021/12/08 21:43:44 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterHandlerRecursion_source_bucket_update_op' in metakv",
  "warnings": null
 }
}
2021/12/08 21:43:44 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterHandlerRecursion_source_bucket_update_op' in metakv",
  "warnings": null
 }
} err 
2021/12/08 21:43:49 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/08 21:43:54 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/08 21:43:59 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/08 21:44:04 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/08 21:44:04 App: TestAllowInterHandlerRecursion_source_bucket_update_op got deployed
2021/12/08 21:44:04 Deploying app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete. Item count src bucket: 0 dst bucket: 0
2021/12/08 21:44:05 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterHandlerRecursion_src_bucket_op_on_delete' in metakv",
  "warnings": null
 }
}
2021/12/08 21:44:05 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterHandlerRecursion_src_bucket_op_on_delete' in metakv",
  "warnings": null
 }
} err 
2021/12/08 21:44:10 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get deployed
2021/12/08 21:44:15 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get deployed
2021/12/08 21:44:20 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get deployed
2021/12/08 21:44:25 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get deployed
2021/12/08 21:44:25 App: TestAllowInterHandlerRecursion_src_bucket_op_on_delete got deployed
2021/12/08 21:44:25 TestAllowInterHandlerRecursion_src_bucket_op_on_delete is deployed
2021/12/08 21:44:25 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 21:44:30 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/08 21:44:35 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/08 21:44:40 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/08 21:44:45 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/08 21:44:45 App: TestAllowInterHandlerRecursion_source_bucket_update_op got un-deployed
2021/12/08 21:44:45 No eventing-con process running
2021/12/08 21:44:45 Function: TestAllowInterHandlerRecursion_source_bucket_update_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAllowInterHandlerRecursion_source_bucket_update_op deleting in the background"
}
2021/12/08 21:44:48 Context: Bucket flush, response code: 200 dump: 
2021/12/08 21:44:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 21:44:54 Context: Bucket flush, response code: 200 dump: 
2021/12/08 21:44:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 21:44:59 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 21:45:04 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/08 21:45:10 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/08 21:45:15 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/08 21:45:20 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/08 21:45:20 App: TestAllowInterHandlerRecursion_src_bucket_op_on_delete got un-deployed
2021/12/08 21:45:20 No eventing-con process running
2021/12/08 21:45:20 Function: TestAllowInterHandlerRecursion_src_bucket_op_on_delete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAllowInterHandlerRecursion_src_bucket_op_on_delete deleting in the background"
}
2021/12/08 21:45:21 Context: Bucket flush, response code: 200 dump: 
2021/12/08 21:45:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 21:45:28 Context: Bucket flush, response code: 200 dump: 
2021/12/08 21:45:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAllowInterHandlerRecursion (151.47s)
=== RUN   TestAllowInterBucketRecursion
2021/12/08 21:45:38 Deploying app: TestAllowInterBucketRecursion_function1. Item count src bucket: 0 dst bucket: 0
2021/12/08 21:45:39 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterBucketRecursion_function1' in metakv",
  "warnings": null
 }
}
2021/12/08 21:45:39 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterBucketRecursion_function1' in metakv",
  "warnings": null
 }
} err 
2021/12/08 21:45:44 Waiting for app: TestAllowInterBucketRecursion_function1 to get deployed
2021/12/08 21:45:49 Waiting for app: TestAllowInterBucketRecursion_function1 to get deployed
2021/12/08 21:45:54 Waiting for app: TestAllowInterBucketRecursion_function1 to get deployed
2021/12/08 21:45:59 Waiting for app: TestAllowInterBucketRecursion_function1 to get deployed
2021/12/08 21:45:59 App: TestAllowInterBucketRecursion_function1 got deployed
2021/12/08 21:45:59 Deploying app: TestAllowInterBucketRecursion_function2. Item count src bucket: 0 dst bucket: 0
2021/12/08 21:46:00 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterBucketRecursion_function2' in metakv",
  "warnings": null
 }
}
2021/12/08 21:46:00 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterBucketRecursion_function2' in metakv",
  "warnings": null
 }
} err 
2021/12/08 21:46:05 Waiting for app: TestAllowInterBucketRecursion_function2 to get deployed
2021/12/08 21:46:10 Waiting for app: TestAllowInterBucketRecursion_function2 to get deployed
2021/12/08 21:46:15 Waiting for app: TestAllowInterBucketRecursion_function2 to get deployed
2021/12/08 21:46:20 Waiting for app: TestAllowInterBucketRecursion_function2 to get deployed
2021/12/08 21:46:20 App: TestAllowInterBucketRecursion_function2 got deployed
2021/12/08 21:46:20 TestAllowInterBucketRecursion_function2 is deployed
2021/12/08 21:46:20 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 21:46:25 Waiting for app: TestAllowInterBucketRecursion_function1 to get un-deployed
2021/12/08 21:46:30 Waiting for app: TestAllowInterBucketRecursion_function1 to get un-deployed
2021/12/08 21:46:35 Waiting for app: TestAllowInterBucketRecursion_function1 to get un-deployed
2021/12/08 21:46:40 Waiting for app: TestAllowInterBucketRecursion_function1 to get un-deployed
2021/12/08 21:46:40 App: TestAllowInterBucketRecursion_function1 got un-deployed
2021/12/08 21:46:40 No eventing-con process running
2021/12/08 21:46:40 Function: TestAllowInterBucketRecursion_function1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAllowInterBucketRecursion_function1 deleting in the background"
}
2021/12/08 21:46:42 Context: Bucket flush, response code: 200 dump: 
2021/12/08 21:46:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 21:46:50 Context: Bucket flush, response code: 200 dump: 
2021/12/08 21:46:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 21:46:55 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 21:47:00 Waiting for app: TestAllowInterBucketRecursion_function2 to get un-deployed
2021/12/08 21:47:05 Waiting for app: TestAllowInterBucketRecursion_function2 to get un-deployed
2021/12/08 21:47:10 Waiting for app: TestAllowInterBucketRecursion_function2 to get un-deployed
2021/12/08 21:47:15 Waiting for app: TestAllowInterBucketRecursion_function2 to get un-deployed
2021/12/08 21:47:15 App: TestAllowInterBucketRecursion_function2 got un-deployed
2021/12/08 21:47:15 No eventing-con process running
2021/12/08 21:47:15 Function: TestAllowInterBucketRecursion_function2 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAllowInterBucketRecursion_function2 deleting in the background"
}
2021/12/08 21:47:16 Context: Bucket flush, response code: 200 dump: 
2021/12/08 21:47:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 21:47:23 Context: Bucket flush, response code: 200 dump: 
2021/12/08 21:47:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAllowInterBucketRecursion (115.30s)
=== RUN   TestInterBucketRecursion2
2021/12/08 21:47:34 Context: Bucket flush, response code: 200 dump: 
2021/12/08 21:47:41 Context: Bucket flush, response code: 200 dump: 
2021/12/08 21:47:46 Context: Bucket flush, response code: 200 dump: 
2021/12/08 21:47:52 Context: Bucket flush, response code: 200 dump: 
2021/12/08 21:47:57 Deploying app: TestInterBucketRecursion2_function1. Item count src bucket: 0 dst bucket: 0
2021/12/08 21:47:59 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterBucketRecursion2_function1' in metakv",
  "warnings": null
 }
}
2021/12/08 21:48:04 Waiting for app: TestInterBucketRecursion2_function1 to get deployed
2021/12/08 21:48:09 Waiting for app: TestInterBucketRecursion2_function1 to get deployed
2021/12/08 21:48:14 Waiting for app: TestInterBucketRecursion2_function1 to get deployed
2021/12/08 21:48:19 Waiting for app: TestInterBucketRecursion2_function1 to get deployed
2021/12/08 21:48:19 App: TestInterBucketRecursion2_function1 got deployed
2021/12/08 21:48:19 Deploying app: TestInterBucketRecursion2_function2. Item count src bucket: 0 dst bucket: 0
2021/12/08 21:48:20 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterBucketRecursion2_function2' in metakv",
  "warnings": null
 }
}
2021/12/08 21:48:25 Waiting for app: TestInterBucketRecursion2_function2 to get deployed
2021/12/08 21:48:30 Waiting for app: TestInterBucketRecursion2_function2 to get deployed
2021/12/08 21:48:35 Waiting for app: TestInterBucketRecursion2_function2 to get deployed
2021/12/08 21:48:40 Waiting for app: TestInterBucketRecursion2_function2 to get deployed
2021/12/08 21:48:40 App: TestInterBucketRecursion2_function2 got deployed
2021/12/08 21:48:40 Deploying app: TestInterBucketRecursion2_function3. Item count src bucket: 0 dst bucket: 0
2021/12/08 21:48:41 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterBucketRecursion2_function3' in metakv",
  "warnings": null
 }
}
2021/12/08 21:48:46 Waiting for app: TestInterBucketRecursion2_function3 to get deployed
2021/12/08 21:48:51 Waiting for app: TestInterBucketRecursion2_function3 to get deployed
2021/12/08 21:48:56 Waiting for app: TestInterBucketRecursion2_function3 to get deployed
2021/12/08 21:49:01 Waiting for app: TestInterBucketRecursion2_function3 to get deployed
2021/12/08 21:49:01 App: TestInterBucketRecursion2_function3 got deployed
2021/12/08 21:49:01 Deploying app: TestInterBucketRecursion2_function4. Item count src bucket: 0 dst bucket: 0
2021/12/08 21:49:01 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_INTER_BUCKET_RECURSION",
 "code": 51,
 "description": "Inter bucket recursion error, deployment of current handler will cause inter bucket recursion",
 "attributes": null,
 "runtime_info": {
  "code": 51,
  "info": "Inter bucket recursion error; function: TestInterBucketRecursion2_function4 causes a cycle involving functions: [TestInterBucketRecursion2_function1 TestInterBucketRecursion2_function2 TestInterBucketRecursion2_function3], hence deployment is disallowed"
 }
}
2021/12/08 21:49:01 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 21:49:06 Waiting for app: TestInterBucketRecursion2_function1 to get un-deployed
2021/12/08 21:49:11 Waiting for app: TestInterBucketRecursion2_function1 to get un-deployed
2021/12/08 21:49:16 Waiting for app: TestInterBucketRecursion2_function1 to get un-deployed
2021/12/08 21:49:21 Waiting for app: TestInterBucketRecursion2_function1 to get un-deployed
2021/12/08 21:49:21 App: TestInterBucketRecursion2_function1 got un-deployed
2021/12/08 21:49:21 No eventing-con process running
2021/12/08 21:49:21 Function: TestInterBucketRecursion2_function1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestInterBucketRecursion2_function1 deleting in the background"
}
2021/12/08 21:49:23 Context: Bucket flush, response code: 200 dump: 
2021/12/08 21:49:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 21:49:29 Context: Bucket flush, response code: 200 dump: 
2021/12/08 21:49:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 21:49:35 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 21:49:40 Waiting for app: TestInterBucketRecursion2_function2 to get un-deployed
2021/12/08 21:49:45 Waiting for app: TestInterBucketRecursion2_function2 to get un-deployed
2021/12/08 21:49:50 Waiting for app: TestInterBucketRecursion2_function2 to get un-deployed
2021/12/08 21:49:55 Waiting for app: TestInterBucketRecursion2_function2 to get un-deployed
2021/12/08 21:49:55 App: TestInterBucketRecursion2_function2 got un-deployed
2021/12/08 21:49:55 No eventing-con process running
2021/12/08 21:49:55 Function: TestInterBucketRecursion2_function2 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestInterBucketRecursion2_function2 deleting in the background"
}
2021/12/08 21:49:56 Context: Bucket flush, response code: 200 dump: 
2021/12/08 21:50:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 21:50:03 Context: Bucket flush, response code: 200 dump: 
2021/12/08 21:50:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 21:50:08 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 21:50:13 Waiting for app: TestInterBucketRecursion2_function3 to get un-deployed
2021/12/08 21:50:18 Waiting for app: TestInterBucketRecursion2_function3 to get un-deployed
2021/12/08 21:50:23 Waiting for app: TestInterBucketRecursion2_function3 to get un-deployed
2021/12/08 21:50:28 Waiting for app: TestInterBucketRecursion2_function3 to get un-deployed
2021/12/08 21:50:28 App: TestInterBucketRecursion2_function3 got un-deployed
2021/12/08 21:50:28 No eventing-con process running
2021/12/08 21:50:28 Function: TestInterBucketRecursion2_function3 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestInterBucketRecursion2_function3 deleting in the background"
}
2021/12/08 21:50:30 Context: Bucket flush, response code: 200 dump: 
2021/12/08 21:50:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 21:50:36 Context: Bucket flush, response code: 200 dump: 
2021/12/08 21:50:41 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 21:50:46 Waiting for app: TestInterBucketRecursion2_function4 to get un-deployed
2021/12/08 21:50:46 App: TestInterBucketRecursion2_function4 got un-deployed
2021/12/08 21:50:46 No eventing-con process running
2021/12/08 21:50:46 Function: TestInterBucketRecursion2_function4 delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestInterBucketRecursion2_function4 not found"
 }
}
2021/12/08 21:50:48 Context: Bucket flush, response code: 200 dump: 
2021/12/08 21:50:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 21:50:55 Context: Bucket flush, response code: 200 dump: 
2021/12/08 21:51:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestInterBucketRecursion2 (213.46s)
=== RUN   TestBucketDeleteWithRebOut
2021/12/08 21:51:07 Removing node(s): [127.0.0.1:19001 127.0.0.1:19002 127.0.0.1:19003] from the cluster
2021/12/08 21:51:12 Rebalance progress: 100

VB distribution across eventing nodes and their workers::

2021/12/08 21:51:15 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:55} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/08 21:51:30 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:55} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/08 21:51:45 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:55} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/08 21:52:00 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:55} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/08 21:52:15 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:55} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/08 21:52:30 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:55} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/08 21:52:45 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:55} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/08 21:53:00 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:55} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/08 21:53:15 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:55} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/08 21:53:30 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:55} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/08 21:53:45 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:55} {Name:#primary Bucket:hello-world Completion:100}]

DCP Stream statuses:

vbucket curr owner:


2021/12/08 21:54:00 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 21:54:00 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/08 21:54:00 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/08 21:54:05 Rebalance progress: 100

VB distribution across eventing nodes and their workers::


DCP Stream statuses:

vbucket curr owner:


2021/12/08 21:54:05 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 21:54:05 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/08 21:54:05 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/08 21:54:10 Rebalance progress: 100

VB distribution across eventing nodes and their workers::


DCP Stream statuses:

vbucket curr owner:


2021/12/08 21:54:10 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 21:54:15 clusterSetup.go:397: Waiting for app: TestBucketDeleteWithRebOut to get un-deployed
2021/12/08 21:54:15 clusterSetup.go:405: App: TestBucketDeleteWithRebOut got un-deployed
2021/12/08 21:54:16 eventingSetup.go:46: No eventing-con process running
2021/12/08 21:54:16 eventingSetup.go:418: Function: TestBucketDeleteWithRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestBucketDeleteWithRebOut not found"
 }
}
2021/12/08 21:54:17 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/08 21:54:22 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 21:54:23 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/08 21:54:28 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 21:54:33 eventingSetup.go:125: Deploying app: TestBucketDeleteWithRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/08 21:54:35 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestBucketDeleteWithRebOut' in metakv",
  "warnings": null
 }
}
2021/12/08 21:54:40 clusterSetup.go:346: Waiting for app: TestBucketDeleteWithRebOut to get deployed
2021/12/08 21:54:45 clusterSetup.go:346: Waiting for app: TestBucketDeleteWithRebOut to get deployed
2021/12/08 21:54:50 clusterSetup.go:346: Waiting for app: TestBucketDeleteWithRebOut to get deployed
2021/12/08 21:54:55 clusterSetup.go:346: Waiting for app: TestBucketDeleteWithRebOut to get deployed
2021/12/08 21:54:55 clusterSetup.go:354: App: TestBucketDeleteWithRebOut got deployed

VB distribution across eventing nodes and their workers::


DCP Stream statuses:
	stream status: running
	len: 1024
	vb list dump: [0-1023]

vbucket curr owner:
Producer node: 127.0.0.1:9302	Node UUID: 625a7c391dce83661389b9255eef38f4
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [114-227]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 114
	v: [228-341]
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [0-113]
Producer node: 172.31.2.134:9300	Node UUID: 7d48dccf03cc30ee41a2c09be793acce
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [683-796]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [797-910]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 113
	v: [911-1023]
Producer node: 127.0.0.1:9303	Node UUID: fea44f1e380744ec9b51e95c0d01a6dc
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [342-455]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [456-569]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 113
	v: [570-682]


2021/12/08 21:54:55 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 21:54:55 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/08 21:55:00 Rebalance progress: 29.16666666666666
2021/12/08 21:55:05 Rebalance progress: 29.16666666666666
2021/12/08 21:55:10 Rebalance progress: 100

VB distribution across eventing nodes and their workers::


DCP Stream statuses:
	stream status: running
	len: 1024
	vb list dump: [0-1023]

vbucket curr owner:
Producer node: 127.0.0.1:9303	Node UUID: fea44f1e380744ec9b51e95c0d01a6dc
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [342-455]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [456-569]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 113
	v: [570-682]
Producer node: 127.0.0.1:9302	Node UUID: 625a7c391dce83661389b9255eef38f4
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [0-113]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [114-227]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 114
	v: [228-341]
Producer node: 172.31.2.134:9300	Node UUID: 7d48dccf03cc30ee41a2c09be793acce
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 113
	v: [911-1023]
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [683-796]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [797-910]


2021/12/08 21:55:10 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 21:55:10 handler_test.go:1674: Deleting source bucket: default
2021/12/08 21:55:11 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/08 21:55:12 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/08 21:55:17 Rebalance progress: 29.16666666666666
2021/12/08 21:55:22 Rebalance progress: 29.16666666666666
2021/12/08 21:55:27 Rebalance progress: 100

VB distribution across eventing nodes and their workers::


DCP Stream statuses:
	stream status: running
	len: 895
	vb list dump: [0-3, 6-10, 12-18, 20-31, 33-39, 41-49, 51-55, 57-61, 64-74, 77-82, 84-94, 96, 98-117, 119-131, 133-150, 152-153, 155, 157-159, 161-166, 168, 170-172, 174, 176-188, 190-213, 215-223, 225-239, 241-263, 265, 267-269, 271-278, 280-286, 288, 290-302, 304, 306-309, 311-313, 316-339, 341-342, 344-349, 351-352, 354-360, 362, 364, 366-372, 374-376, 378-403, 405-419, 421-426, 428-440, 444-449, 451-456, 458, 460-465, 467-468, 470-473, 478-486, 490-492, 494-495, 498, 500-504, 507-576, 578-580, 582-592, 594, 596-603, 605-636, 639-648, 650-653, 655-660, 663-675, 677-688, 690-696, 698-708, 710-714, 716-725, 727-731, 733-734, 736-748, 750-760, 762-763, 765-770, 772-778, 780-781, 783-788, 790-793, 795-806, 809-810, 812-830, 832-838, 840-847, 849-850, 852-854, 856, 858-866, 869, 872-877, 879-888, 890-892, 894-895, 897-910, 913-924, 926-933, 935-937, 939-970, 972, 974-976, 978-993, 995-998, 1000-1013, 1015-1022]
	stream status: stopped
	len: 129
	vb list dump: [4-5, 11, 19, 32, 40, 50, 56, 62-63, 75-76, 83, 95, 97, 118, 132, 151, 154, 156, 160, 167, 169, 173, 175, 189, 214, 224, 240, 264, 266, 270, 279, 287, 289, 303, 305, 310, 314-315, 340, 343, 350, 353, 361, 363, 365, 373, 377, 404, 420, 427, 441-443, 450, 457, 459, 466, 469, 474-477, 487-489, 493, 496-497, 499, 505-506, 577, 581, 593, 595, 604, 637-638, 649, 654, 661-662, 676, 689, 697, 709, 715, 726, 732, 735, 749, 761, 764, 771, 779, 782, 789, 794, 807-808, 811, 831, 839, 848, 851, 855, 857, 867-868, 870-871, 878, 889, 893, 896, 911-912, 925, 934, 938, 971, 973, 977, 994, 999, 1014, 1023]

vbucket curr owner:
Producer node: 127.0.0.1:9302	Node UUID: 625a7c391dce83661389b9255eef38f4
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 99
	v: [0-3, 6-10, 12-18, 20-31, 33-39, 41-49, 51-55, 57-61, 64-74, 77-82, 84-94, 96, 98-113]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 101
	v: [114-117, 119-131, 133-150, 152-153, 155, 157-159, 161-166, 168, 170-172, 174, 176-188, 190-213, 215-223, 225-227]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 101
	v: [228-239, 241-263, 265, 267-269, 271-278, 280-286, 288, 290-302, 304, 306-309, 311-313, 316-339, 341]
Producer node: 172.31.2.134:9300	Node UUID: 7d48dccf03cc30ee41a2c09be793acce
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 101
	v: [913-924, 926-933, 935-937, 939-970, 972, 974-976, 978-993, 995-998, 1000-1013, 1015-1022]
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 99
	v: [683-688, 690-696, 698-708, 710-714, 716-725, 727-731, 733-734, 736-748, 750-760, 762-763, 765-770, 772-778, 780-781, 783-788, 790-793, 795-796]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 97
	v: [797-806, 809-810, 812-830, 832-838, 840-847, 849-850, 852-854, 856, 858-866, 869, 872-877, 879-888, 890-892, 894-895, 897-910]
Producer node: 127.0.0.1:9303	Node UUID: fea44f1e380744ec9b51e95c0d01a6dc
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 97
	v: [456, 458, 460-465, 467-468, 470-473, 478-486, 490-492, 494-495, 498, 500-504, 507-569]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 101
	v: [570-576, 578-580, 582-592, 594, 596-603, 605-636, 639-648, 650-653, 655-660, 663-675, 677-682]
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 99
	v: [342, 344-349, 351-352, 354-360, 362, 364, 366-372, 374-376, 378-403, 405-419, 421-426, 428-440, 444-449, 451-455]


2021/12/08 21:55:27 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 21:55:32 eventingSetup.go:719: Waiting for function: TestBucketDeleteWithRebOut status to change to undeployed
2021/12/08 21:55:32 eventingSetup.go:727: Function: TestBucketDeleteWithRebOut status changed to undeployed
2021/12/08 21:55:37 clusterSetup.go:397: Waiting for app: TestBucketDeleteWithRebOut to get un-deployed
2021/12/08 21:55:37 clusterSetup.go:405: App: TestBucketDeleteWithRebOut got un-deployed
2021/12/08 21:55:37 eventingSetup.go:46: No eventing-con process running
2021/12/08 21:55:38 eventingSetup.go:418: Function: TestBucketDeleteWithRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestBucketDeleteWithRebOut deleting in the background"
}
2021/12/08 21:55:39 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/08 21:55:44 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 21:55:46 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/08 21:55:51 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 21:55:51 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv,index,n1ql to the cluster
2021/12/08 21:55:51 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/08 21:55:56 Rebalance progress: 100
--- PASS: TestBucketDeleteWithRebOut (297.15s)
=== RUN   TestN1QLRecursion
2021/12/08 21:55:59 Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/08 21:56:00 Error in adding nodes : , response: ["Prepare join failed. Node is already part of cluster."]
2021/12/08 21:56:00 Removing node(s): [] from the cluster
2021/12/08 21:56:05 Rebalance progress: 100
2021/12/08 21:56:11 Context: Bucket flush, response code: 200 dump: 
2021/12/08 21:56:17 Context: Bucket flush, response code: 200 dump: 
2021/12/08 21:56:23 Context: Bucket flush, response code: 200 dump: 
2021/12/08 21:56:28 Deploying app: TestN1QLRecursion_function1. Item count src bucket: 0 dst bucket: 0
2021/12/08 21:56:30 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLRecursion_function1' in metakv",
  "warnings": null
 }
}
2021/12/08 21:56:30 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLRecursion_function1' in metakv",
  "warnings": null
 }
} err 
2021/12/08 21:56:35 Waiting for app: TestN1QLRecursion_function1 to get deployed
2021/12/08 21:56:40 Waiting for app: TestN1QLRecursion_function1 to get deployed
2021/12/08 21:56:45 Waiting for app: TestN1QLRecursion_function1 to get deployed
2021/12/08 21:56:50 Waiting for app: TestN1QLRecursion_function1 to get deployed
2021/12/08 21:56:50 App: TestN1QLRecursion_function1 got deployed
2021/12/08 21:56:50 TestN1QLRecursion_function1 is deployed
2021/12/08 21:56:50 Deploying app: TestN1QLRecursion_function3. Item count src bucket: 0 dst bucket: 0
2021/12/08 21:56:51 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLRecursion_function3' in metakv",
  "warnings": null
 }
}
2021/12/08 21:56:51 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLRecursion_function3' in metakv",
  "warnings": null
 }
} err 
2021/12/08 21:56:56 Waiting for app: TestN1QLRecursion_function3 to get deployed
2021/12/08 21:57:01 Waiting for app: TestN1QLRecursion_function3 to get deployed
2021/12/08 21:57:06 Waiting for app: TestN1QLRecursion_function3 to get deployed
2021/12/08 21:57:11 Waiting for app: TestN1QLRecursion_function3 to get deployed
2021/12/08 21:57:11 App: TestN1QLRecursion_function3 got deployed
2021/12/08 21:57:11 TestN1QLRecursion_function3 is deployed
2021/12/08 21:57:11 Deploying app: TestN1QLRecursion_function2. Item count src bucket: 0 dst bucket: 0
2021/12/08 21:57:11 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_INTER_BUCKET_RECURSION",
 "code": 51,
 "description": "Inter bucket recursion error, deployment of current handler will cause inter bucket recursion",
 "attributes": null,
 "runtime_info": {
  "code": 51,
  "info": "Inter bucket recursion error; function: TestN1QLRecursion_function2 causes a cycle involving functions: [TestN1QLRecursion_function3 TestN1QLRecursion_function1], hence deployment is disallowed"
 }
}
2021/12/08 21:57:11 response body {
 "name": "ERR_INTER_BUCKET_RECURSION",
 "code": 51,
 "description": "Inter bucket recursion error, deployment of current handler will cause inter bucket recursion",
 "attributes": null,
 "runtime_info": {
  "code": 51,
  "info": "Inter bucket recursion error; function: TestN1QLRecursion_function2 causes a cycle involving functions: [TestN1QLRecursion_function3 TestN1QLRecursion_function1], hence deployment is disallowed"
 }
} err 
2021/12/08 21:57:11 Success: TestN1QLRecursion_function2 is not deployed
2021/12/08 21:57:11 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 21:57:16 Waiting for app: TestN1QLRecursion_function1 to get un-deployed
2021/12/08 21:57:21 Waiting for app: TestN1QLRecursion_function1 to get un-deployed
2021/12/08 21:57:26 Waiting for app: TestN1QLRecursion_function1 to get un-deployed
2021/12/08 21:57:31 Waiting for app: TestN1QLRecursion_function1 to get un-deployed
2021/12/08 21:57:31 App: TestN1QLRecursion_function1 got un-deployed
2021/12/08 21:57:31 No eventing-con process running
2021/12/08 21:57:31 Function: TestN1QLRecursion_function1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLRecursion_function1 deleting in the background"
}
2021/12/08 21:57:33 Context: Bucket flush, response code: 200 dump: 
2021/12/08 21:57:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 21:57:40 Context: Bucket flush, response code: 200 dump: 
2021/12/08 21:57:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 21:57:45 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 21:57:50 Waiting for app: TestN1QLRecursion_function3 to get un-deployed
2021/12/08 21:57:55 Waiting for app: TestN1QLRecursion_function3 to get un-deployed
2021/12/08 21:58:00 Waiting for app: TestN1QLRecursion_function3 to get un-deployed
2021/12/08 21:58:05 Waiting for app: TestN1QLRecursion_function3 to get un-deployed
2021/12/08 21:58:05 App: TestN1QLRecursion_function3 got un-deployed
2021/12/08 21:58:05 No eventing-con process running
2021/12/08 21:58:05 Function: TestN1QLRecursion_function3 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLRecursion_function3 deleting in the background"
}
2021/12/08 21:58:07 Context: Bucket flush, response code: 200 dump: 
2021/12/08 21:58:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 21:58:14 Context: Bucket flush, response code: 200 dump: 
2021/12/08 21:58:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 21:58:20 Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/08 21:58:25 Rebalance progress: 100
--- PASS: TestN1QLRecursion (146.44s)
=== RUN   TestN1QLAllowRecursion
2021/12/08 21:58:25 Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/08 21:58:26 Error in adding nodes : , response: ["Prepare join failed. Node is already part of cluster."]
2021/12/08 21:58:26 Removing node(s): [] from the cluster
2021/12/08 21:58:31 Rebalance progress: 100
2021/12/08 21:58:36 Context: Bucket flush, response code: 200 dump: 
2021/12/08 21:58:42 Context: Bucket flush, response code: 200 dump: 
2021/12/08 21:58:48 Context: Bucket flush, response code: 200 dump: 
2021/12/08 21:58:53 Deploying app: TestN1QLAllowRecursion_function1. Item count src bucket: 0 dst bucket: 0
2021/12/08 21:58:54 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function1' in metakv",
  "warnings": null
 }
}
2021/12/08 21:58:54 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function1' in metakv",
  "warnings": null
 }
} err 
2021/12/08 21:58:59 Waiting for app: TestN1QLAllowRecursion_function1 to get deployed
2021/12/08 21:59:04 Waiting for app: TestN1QLAllowRecursion_function1 to get deployed
2021/12/08 21:59:09 Waiting for app: TestN1QLAllowRecursion_function1 to get deployed
2021/12/08 21:59:14 Waiting for app: TestN1QLAllowRecursion_function1 to get deployed
2021/12/08 21:59:14 App: TestN1QLAllowRecursion_function1 got deployed
2021/12/08 21:59:14 TestN1QLAllowRecursion_function1 is deployed
2021/12/08 21:59:14 Deploying app: TestN1QLAllowRecursion_function3. Item count src bucket: 0 dst bucket: 0
2021/12/08 21:59:15 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function3' in metakv",
  "warnings": null
 }
}
2021/12/08 21:59:15 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function3' in metakv",
  "warnings": null
 }
} err 
2021/12/08 21:59:20 Waiting for app: TestN1QLAllowRecursion_function3 to get deployed
2021/12/08 21:59:25 Waiting for app: TestN1QLAllowRecursion_function3 to get deployed
2021/12/08 21:59:30 Waiting for app: TestN1QLAllowRecursion_function3 to get deployed
2021/12/08 21:59:35 Waiting for app: TestN1QLAllowRecursion_function3 to get deployed
2021/12/08 21:59:35 App: TestN1QLAllowRecursion_function3 got deployed
2021/12/08 21:59:35 TestN1QLAllowRecursion_function3 is deployed
2021/12/08 21:59:35 Deploying app: TestN1QLAllowRecursion_function2. Item count src bucket: 0 dst bucket: 0
2021/12/08 21:59:37 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function2' in metakv",
  "warnings": null
 }
}
2021/12/08 21:59:37 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function2' in metakv",
  "warnings": null
 }
} err 
2021/12/08 21:59:42 Waiting for app: TestN1QLAllowRecursion_function2 to get deployed
2021/12/08 21:59:47 Waiting for app: TestN1QLAllowRecursion_function2 to get deployed
2021/12/08 21:59:52 Waiting for app: TestN1QLAllowRecursion_function2 to get deployed
2021/12/08 21:59:57 Waiting for app: TestN1QLAllowRecursion_function2 to get deployed
2021/12/08 21:59:57 App: TestN1QLAllowRecursion_function2 got deployed
2021/12/08 21:59:57 Success: TestN1QLAllowRecursion_function2 is deployed after allow_interbucket_recursion
2021/12/08 21:59:57 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 22:00:02 Waiting for app: TestN1QLAllowRecursion_function1 to get un-deployed
2021/12/08 22:00:07 Waiting for app: TestN1QLAllowRecursion_function1 to get un-deployed
2021/12/08 22:00:12 Waiting for app: TestN1QLAllowRecursion_function1 to get un-deployed
2021/12/08 22:00:17 Waiting for app: TestN1QLAllowRecursion_function1 to get un-deployed
2021/12/08 22:00:17 App: TestN1QLAllowRecursion_function1 got un-deployed
2021/12/08 22:00:17 No eventing-con process running
2021/12/08 22:00:17 Function: TestN1QLAllowRecursion_function1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLAllowRecursion_function1 deleting in the background"
}
2021/12/08 22:00:19 Context: Bucket flush, response code: 200 dump: 
2021/12/08 22:00:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 22:00:26 Context: Bucket flush, response code: 200 dump: 
2021/12/08 22:00:31 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 22:00:31 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 22:00:36 Waiting for app: TestN1QLAllowRecursion_function2 to get un-deployed
2021/12/08 22:00:41 Waiting for app: TestN1QLAllowRecursion_function2 to get un-deployed
2021/12/08 22:00:46 Waiting for app: TestN1QLAllowRecursion_function2 to get un-deployed
2021/12/08 22:00:51 Waiting for app: TestN1QLAllowRecursion_function2 to get un-deployed
2021/12/08 22:00:51 App: TestN1QLAllowRecursion_function2 got un-deployed
2021/12/08 22:00:51 No eventing-con process running
2021/12/08 22:00:51 Function: TestN1QLAllowRecursion_function2 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLAllowRecursion_function2 deleting in the background"
}
2021/12/08 22:00:53 Context: Bucket flush, response code: 200 dump: 
2021/12/08 22:00:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 22:01:00 Context: Bucket flush, response code: 200 dump: 
2021/12/08 22:01:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 22:01:05 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 22:01:10 Waiting for app: TestN1QLAllowRecursion_function3 to get un-deployed
2021/12/08 22:01:15 Waiting for app: TestN1QLAllowRecursion_function3 to get un-deployed
2021/12/08 22:01:20 Waiting for app: TestN1QLAllowRecursion_function3 to get un-deployed
2021/12/08 22:01:25 Waiting for app: TestN1QLAllowRecursion_function3 to get un-deployed
2021/12/08 22:01:25 App: TestN1QLAllowRecursion_function3 got un-deployed
2021/12/08 22:01:25 No eventing-con process running
2021/12/08 22:01:25 Function: TestN1QLAllowRecursion_function3 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLAllowRecursion_function3 deleting in the background"
}
2021/12/08 22:01:27 Context: Bucket flush, response code: 200 dump: 
2021/12/08 22:01:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 22:01:34 Context: Bucket flush, response code: 200 dump: 
2021/12/08 22:01:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 22:01:40 Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/08 22:01:45 Rebalance progress: 100
--- PASS: TestN1QLAllowRecursion (200.02s)
=== RUN   TestOnDeleteExpiryBucketOp
2021/12/08 22:01:45 Sleeping for some time
2021/12/08 22:01:55 Waiting for app: TestOnDeleteExpiryBucketOp to get un-deployed
2021/12/08 22:01:55 App: TestOnDeleteExpiryBucketOp got un-deployed
2021/12/08 22:01:55 No eventing-con process running
2021/12/08 22:01:55 Function: TestOnDeleteExpiryBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnDeleteExpiryBucketOp not found"
 }
}
2021/12/08 22:01:57 Context: Bucket flush, response code: 200 dump: 
2021/12/08 22:02:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 22:02:04 Context: Bucket flush, response code: 200 dump: 
2021/12/08 22:02:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 22:02:09 Deploying app: TestOnDeleteExpiryBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/08 22:02:10 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnDeleteExpiryBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/08 22:02:10 Starting bucket ops to source bucket
2021/12/08 22:02:59 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 22:03:04 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 22:03:04 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/08 22:03:04 Starting bucket ops to source bucket
2021/12/08 22:03:05 Starting bucket ops to source bucket
2021/12/08 22:04:11 Waiting for dst bucket item count to get to: 7000 curr count: 5000
2021/12/08 22:04:11 src & dst bucket item count matched up. src bucket count: 7000 dst bucket count: 7000
2021/12/08 22:04:11 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 22:04:11 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 22:04:16 Waiting for app: TestOnDeleteExpiryBucketOp to get un-deployed
2021/12/08 22:04:21 Waiting for app: TestOnDeleteExpiryBucketOp to get un-deployed
2021/12/08 22:04:26 Waiting for app: TestOnDeleteExpiryBucketOp to get un-deployed
2021/12/08 22:04:31 Waiting for app: TestOnDeleteExpiryBucketOp to get un-deployed
2021/12/08 22:04:31 App: TestOnDeleteExpiryBucketOp got un-deployed
2021/12/08 22:04:31 No eventing-con process running
2021/12/08 22:04:31 Function: TestOnDeleteExpiryBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnDeleteExpiryBucketOp deleting in the background"
}
2021/12/08 22:04:33 Context: Bucket flush, response code: 200 dump: 
2021/12/08 22:04:43 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/08 22:04:48 Waiting for dst bucket item count to get to: 0 curr count: 501
2021/12/08 22:04:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 22:04:50 Context: Bucket flush, response code: 200 dump: 
2021/12/08 22:05:00 Waiting for dst bucket item count to get to: 0 curr count: 4422
2021/12/08 22:05:05 Waiting for dst bucket item count to get to: 0 curr count: 4422
2021/12/08 22:05:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnDeleteExpiryBucketOp (199.84s)
=== RUN   TestTimerOverWriteSameReference
2021/12/08 22:05:10 Waiting for app: TestTimerOverWriteSameReference to get un-deployed
2021/12/08 22:05:10 App: TestTimerOverWriteSameReference got un-deployed
2021/12/08 22:05:10 No eventing-con process running
2021/12/08 22:05:10 Function: TestTimerOverWriteSameReference delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestTimerOverWriteSameReference not found"
 }
}
2021/12/08 22:05:12 Context: Bucket flush, response code: 200 dump: 
2021/12/08 22:05:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 22:05:19 Context: Bucket flush, response code: 200 dump: 
2021/12/08 22:05:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 22:05:24 Deploying app: TestTimerOverWriteSameReference. Item count src bucket: 0 dst bucket: 0
2021/12/08 22:05:25 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestTimerOverWriteSameReference' in metakv",
  "warnings": null
 }
}
2021/12/08 22:05:30 Waiting for app: TestTimerOverWriteSameReference to get deployed
2021/12/08 22:05:35 Waiting for app: TestTimerOverWriteSameReference to get deployed
2021/12/08 22:05:40 Waiting for app: TestTimerOverWriteSameReference to get deployed
2021/12/08 22:05:45 Waiting for app: TestTimerOverWriteSameReference to get deployed
2021/12/08 22:05:45 App: TestTimerOverWriteSameReference got deployed
2021/12/08 22:05:55 Starting bucket ops to source bucket
2021/12/08 22:06:56 src & dst bucket item count matched up. src bucket count: 1 dst bucket count: 1
2021/12/08 22:07:06 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 22:07:06 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 22:07:11 Waiting for app: TestTimerOverWriteSameReference to get un-deployed
2021/12/08 22:07:16 Waiting for app: TestTimerOverWriteSameReference to get un-deployed
2021/12/08 22:07:21 Waiting for app: TestTimerOverWriteSameReference to get un-deployed
2021/12/08 22:07:26 Waiting for app: TestTimerOverWriteSameReference to get un-deployed
2021/12/08 22:07:26 App: TestTimerOverWriteSameReference got un-deployed
2021/12/08 22:07:26 No eventing-con process running
2021/12/08 22:07:26 Function: TestTimerOverWriteSameReference delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestTimerOverWriteSameReference deleting in the background"
}
2021/12/08 22:07:28 Context: Bucket flush, response code: 200 dump: 
2021/12/08 22:07:38 Waiting for dst bucket item count to get to: 0 curr count: 623
2021/12/08 22:07:43 Waiting for dst bucket item count to get to: 0 curr count: 123
2021/12/08 22:07:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 22:07:45 Context: Bucket flush, response code: 200 dump: 
2021/12/08 22:07:55 Waiting for dst bucket item count to get to: 0 curr count: 1
2021/12/08 22:07:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestTimerOverWriteSameReference (169.83s)
=== RUN   TestCancelTimerBucketop
2021/12/08 22:08:00 Waiting for app: TestCancelTimerBucketop to get un-deployed
2021/12/08 22:08:00 App: TestCancelTimerBucketop got un-deployed
2021/12/08 22:08:00 No eventing-con process running
2021/12/08 22:08:00 Function: TestCancelTimerBucketop delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestCancelTimerBucketop not found"
 }
}
2021/12/08 22:08:02 Context: Bucket flush, response code: 200 dump: 
2021/12/08 22:08:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 22:08:09 Context: Bucket flush, response code: 200 dump: 
2021/12/08 22:08:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 22:08:14 Deploying app: TestCancelTimerBucketop. Item count src bucket: 0 dst bucket: 0
2021/12/08 22:08:15 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCancelTimerBucketop' in metakv",
  "warnings": null
 }
}
2021/12/08 22:08:20 Waiting for app: TestCancelTimerBucketop to get deployed
2021/12/08 22:08:25 Waiting for app: TestCancelTimerBucketop to get deployed
2021/12/08 22:08:30 Waiting for app: TestCancelTimerBucketop to get deployed
2021/12/08 22:08:35 Waiting for app: TestCancelTimerBucketop to get deployed
2021/12/08 22:08:35 App: TestCancelTimerBucketop got deployed
2021/12/08 22:08:35 Starting bucket ops to source bucket
2021/12/08 22:08:42 Starting bucket ops to source bucket
2021/12/08 22:09:44 src & dst bucket item count matched up. src bucket count: 1000 dst bucket count: 1000
2021/12/08 22:09:44 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 22:09:44 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 22:09:49 Waiting for app: TestCancelTimerBucketop to get un-deployed
2021/12/08 22:09:54 Waiting for app: TestCancelTimerBucketop to get un-deployed
2021/12/08 22:09:59 Waiting for app: TestCancelTimerBucketop to get un-deployed
2021/12/08 22:10:04 Waiting for app: TestCancelTimerBucketop to get un-deployed
2021/12/08 22:10:04 App: TestCancelTimerBucketop got un-deployed
2021/12/08 22:10:04 No eventing-con process running
2021/12/08 22:10:04 Function: TestCancelTimerBucketop delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCancelTimerBucketop deleting in the background"
}
2021/12/08 22:10:06 Context: Bucket flush, response code: 200 dump: 
2021/12/08 22:10:16 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/08 22:10:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 22:10:18 Context: Bucket flush, response code: 200 dump: 
2021/12/08 22:10:28 Waiting for dst bucket item count to get to: 0 curr count: 765
2021/12/08 22:10:33 Waiting for dst bucket item count to get to: 0 curr count: 265
2021/12/08 22:10:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCancelTimerBucketop (158.33s)
=== RUN   TestJSExpiryDate
2021/12/08 22:10:38 Starting bucket ops to source bucket
2021/12/08 22:10:56 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/08 22:10:56 Deploying app: TestJSExpiryDate. Item count src bucket: 5000 dst bucket: 0
2021/12/08 22:10:57 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestJSExpiryDate' in metakv",
  "warnings": null
 }
}
2021/12/08 22:11:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 22:11:28 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 22:11:28 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 22:11:33 Waiting for app: TestJSExpiryDate to get un-deployed
2021/12/08 22:11:38 Waiting for app: TestJSExpiryDate to get un-deployed
2021/12/08 22:11:43 Waiting for app: TestJSExpiryDate to get un-deployed
2021/12/08 22:11:48 Waiting for app: TestJSExpiryDate to get un-deployed
2021/12/08 22:11:48 App: TestJSExpiryDate got un-deployed
2021/12/08 22:11:48 No eventing-con process running
2021/12/08 22:11:48 Function: TestJSExpiryDate delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestJSExpiryDate deleting in the background"
}
2021/12/08 22:11:49 Context: Bucket flush, response code: 200 dump: 
2021/12/08 22:11:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 22:11:57 Context: Bucket flush, response code: 200 dump: 
2021/12/08 22:12:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestJSExpiryDate (88.74s)
=== RUN   TestBinaryDoc
2021/12/08 22:12:07 Starting bucket ops to source bucket
2021/12/08 22:12:07 Deploying app: TestBinaryDoc. Item count src bucket: 0 dst bucket: 0
2021/12/08 22:12:08 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestBinaryDoc' in metakv",
  "warnings": null
 }
}
2021/12/08 22:12:08 Starting bucket ops to source bucket
2021/12/08 22:12:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 22:12:08 Deleting source bucket docs now
2021/12/08 22:12:08 Starting bucket ops to source bucket
2021/12/08 22:12:46 Waiting for dst bucket item count to get to: 32 curr count: 24
2021/12/08 22:12:51 Waiting for dst bucket item count to get to: 32 curr count: 29
2021/12/08 22:12:56 Waiting for dst bucket item count to get to: 32 curr count: 31
2021/12/08 22:12:56 src & dst bucket item count matched up. src bucket count: 32 dst bucket count: 32
2021/12/08 22:12:56 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 22:12:56 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 22:13:01 Waiting for app: TestBinaryDoc to get un-deployed
2021/12/08 22:13:06 Waiting for app: TestBinaryDoc to get un-deployed
2021/12/08 22:13:11 Waiting for app: TestBinaryDoc to get un-deployed
2021/12/08 22:13:16 Waiting for app: TestBinaryDoc to get un-deployed
2021/12/08 22:13:16 App: TestBinaryDoc got un-deployed
2021/12/08 22:13:16 No eventing-con process running
2021/12/08 22:13:16 Function: TestBinaryDoc delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestBinaryDoc deleting in the background"
}
2021/12/08 22:13:18 Context: Bucket flush, response code: 200 dump: 
2021/12/08 22:13:28 Waiting for dst bucket item count to get to: 0 curr count: 18
2021/12/08 22:13:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 22:13:29 Context: Bucket flush, response code: 200 dump: 
2021/12/08 22:13:39 Waiting for dst bucket item count to get to: 0 curr count: 32
2021/12/08 22:13:44 Waiting for dst bucket item count to get to: 0 curr count: 16
2021/12/08 22:13:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestBinaryDoc (102.39s)
=== RUN   TestConstantBindings
2021/12/08 22:13:44 Deploying app: TestConstantBindings. Item count src bucket: 0 dst bucket: 0
2021/12/08 22:13:45 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestConstantBindings' in metakv",
  "warnings": null
 }
}
2021/12/08 22:13:45 Starting bucket ops to source bucket
2021/12/08 22:13:51 Waiting for dst bucket item count to get to: 6000 curr count: 0
2021/12/08 22:13:56 Waiting for dst bucket item count to get to: 6000 curr count: 0
2021/12/08 22:14:01 Waiting for dst bucket item count to get to: 6000 curr count: 0
2021/12/08 22:14:06 Waiting for dst bucket item count to get to: 6000 curr count: 2655
2021/12/08 22:14:11 Waiting for dst bucket item count to get to: 6000 curr count: 2655
2021/12/08 22:14:11 src & dst bucket item count matched up. src bucket count: 6000 dst bucket count: 6000
2021/12/08 22:14:11 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 22:14:11 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 22:14:16 Waiting for app: TestConstantBindings to get un-deployed
2021/12/08 22:14:21 Waiting for app: TestConstantBindings to get un-deployed
2021/12/08 22:14:26 Waiting for app: TestConstantBindings to get un-deployed
2021/12/08 22:14:31 Waiting for app: TestConstantBindings to get un-deployed
2021/12/08 22:14:31 App: TestConstantBindings got un-deployed
2021/12/08 22:14:31 No eventing-con process running
2021/12/08 22:14:31 Function: TestConstantBindings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestConstantBindings deleting in the background"
}
2021/12/08 22:14:33 Context: Bucket flush, response code: 200 dump: 
2021/12/08 22:14:43 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/08 22:14:48 Waiting for dst bucket item count to get to: 0 curr count: 500
2021/12/08 22:14:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 22:14:50 Context: Bucket flush, response code: 200 dump: 
2021/12/08 22:15:00 Waiting for dst bucket item count to get to: 0 curr count: 3645
2021/12/08 22:15:05 Waiting for dst bucket item count to get to: 0 curr count: 3645
2021/12/08 22:15:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestConstantBindings (80.66s)
=== RUN   TestCodeInsight
2021/12/08 22:15:10 Waiting for app: TestCodeInsight to get un-deployed
2021/12/08 22:15:10 App: TestCodeInsight got un-deployed
2021/12/08 22:15:10 No eventing-con process running
2021/12/08 22:15:10 Function: TestCodeInsight delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestCodeInsight not found"
 }
}
2021/12/08 22:15:12 Context: Bucket flush, response code: 200 dump: 
2021/12/08 22:15:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 22:15:19 Context: Bucket flush, response code: 200 dump: 
2021/12/08 22:15:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 22:15:24 Starting bucket ops to source bucket
2021/12/08 22:15:24 Deploying app: TestCodeInsight. Item count src bucket: 0 dst bucket: 0
2021/12/08 22:15:25 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCodeInsight' in metakv",
  "warnings": null
 }
}
2021/12/08 22:15:30 Waiting for app: TestCodeInsight to get deployed
2021/12/08 22:15:35 Waiting for app: TestCodeInsight to get deployed
2021/12/08 22:15:40 Waiting for app: TestCodeInsight to get deployed
2021/12/08 22:15:45 Waiting for app: TestCodeInsight to get deployed
2021/12/08 22:15:45 App: TestCodeInsight got deployed
2021/12/08 22:15:50 Waiting for dst bucket item count to get to: 1000 curr count: 574
2021/12/08 22:15:50 src & dst bucket item count matched up. src bucket count: 1000 dst bucket count: 1000
2021/12/08 22:15:50 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 22:15:55 Waiting for app: TestCodeInsight to get un-deployed
2021/12/08 22:16:01 Waiting for app: TestCodeInsight to get un-deployed
2021/12/08 22:16:06 Waiting for app: TestCodeInsight to get un-deployed
2021/12/08 22:16:11 Waiting for app: TestCodeInsight to get un-deployed
2021/12/08 22:16:11 App: TestCodeInsight got un-deployed
2021/12/08 22:16:11 No eventing-con process running
2021/12/08 22:16:11 Function: TestCodeInsight delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCodeInsight deleting in the background"
}
2021/12/08 22:16:12 Context: Bucket flush, response code: 200 dump: 
2021/12/08 22:16:22 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/08 22:16:27 Waiting for dst bucket item count to get to: 0 curr count: 500
2021/12/08 22:16:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 22:16:29 Context: Bucket flush, response code: 200 dump: 
2021/12/08 22:16:39 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/08 22:16:44 Waiting for dst bucket item count to get to: 0 curr count: 500
2021/12/08 22:16:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCodeInsight (99.17s)
=== RUN   TestParserTransform
--- PASS: TestParserTransform (0.05s)
=== RUN   TestParserDetect
--- PASS: TestParserDetect (0.00s)
PASS
ok  	github.com/couchbase/eventing/tests/functional_tests	12593.475s

Functional tests took 3 hours, 29 minutes and 57 seconds.

N1ql tests

2021/12/08 22:16:53 Started n1ql
Post "http://127.0.0.1:9000/pools/default": EOF
Quota setup Post "http://127.0.0.1:9000/pools/default": EOF
2021/12/08 22:17:04 Adding node: https://127.0.0.1:19001 with role: kv,index,n1ql to the cluster
2021/12/08 22:17:04 Error in adding nodes : , response: ["Prepare join failed. Node is already part of cluster."]
2021/12/08 22:17:04 Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/08 22:17:04 Removing node(s): [] from the cluster
2021/12/08 22:17:09 Rebalance progress: 100
=== RUN   TestRecursiveMutationN1QL
2021/12/08 22:17:14 Waiting for app: TestRecursiveMutationN1QL to get un-deployed
2021/12/08 22:17:14 App: TestRecursiveMutationN1QL got un-deployed
2021/12/08 22:17:14 No eventing-con process running
2021/12/08 22:17:14 Function: TestRecursiveMutationN1QL delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestRecursiveMutationN1QL not found"
 }
}
2021/12/08 22:17:16 Context: Bucket flush, response code: 200 dump: 
2021/12/08 22:17:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 22:17:23 Context: Bucket flush, response code: 200 dump: 
2021/12/08 22:17:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 22:17:28 Deploying app: TestRecursiveMutationN1QL. Item count src bucket: 0 dst bucket: 0
2021/12/08 22:17:28 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_INTER_BUCKET_RECURSION",
 "code": 51,
 "description": "Inter bucket recursion error, deployment of current handler will cause inter bucket recursion",
 "attributes": null,
 "runtime_info": {
  "code": 51,
  "info": "Function: TestRecursiveMutationN1QL N1QL DML to source keyspace default:default"
 }
}
--- PASS: TestRecursiveMutationN1QL (18.65s)
=== RUN   TestFlexReset1
2021/12/08 22:17:33 Waiting for app: TestFlexReset1 to get un-deployed
2021/12/08 22:17:33 App: TestFlexReset1 got un-deployed
2021/12/08 22:17:33 No eventing-con process running
2021/12/08 22:17:33 Function: TestFlexReset1 delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestFlexReset1 not found"
 }
}
2021/12/08 22:17:35 Context: Bucket flush, response code: 200 dump: 
2021/12/08 22:17:40 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 22:17:41 Context: Bucket flush, response code: 200 dump: 
2021/12/08 22:17:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 22:17:46 Starting bucket ops to source bucket
2021/12/08 22:17:47 Deploying app: TestFlexReset1. Item count src bucket: 0 dst bucket: 0
2021/12/08 22:17:48 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestFlexReset1' in metakv",
  "warnings": null
 }
}
2021/12/08 22:17:53 Waiting for app: TestFlexReset1 to get deployed
2021/12/08 22:17:58 Waiting for app: TestFlexReset1 to get deployed
2021/12/08 22:18:03 Waiting for app: TestFlexReset1 to get deployed
2021/12/08 22:18:08 Waiting for app: TestFlexReset1 to get deployed
2021/12/08 22:18:08 App: TestFlexReset1 got deployed
2021/12/08 22:18:13 Waiting for dst bucket item count to get to: 1000 curr count: 2
2021/12/08 22:18:13 src & dst bucket item count matched up. src bucket count: 1000 dst bucket count: 1000
2021/12/08 22:18:13 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 22:18:13 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 22:18:18 Waiting for app: TestFlexReset1 to get un-deployed
2021/12/08 22:18:23 Waiting for app: TestFlexReset1 to get un-deployed
2021/12/08 22:18:28 Waiting for app: TestFlexReset1 to get un-deployed
2021/12/08 22:18:33 Waiting for app: TestFlexReset1 to get un-deployed
2021/12/08 22:18:33 App: TestFlexReset1 got un-deployed
2021/12/08 22:18:33 No eventing-con process running
2021/12/08 22:18:33 Function: TestFlexReset1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestFlexReset1 deleting in the background"
}
2021/12/08 22:18:35 Context: Bucket flush, response code: 200 dump: 
2021/12/08 22:18:45 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/08 22:18:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 22:18:47 Context: Bucket flush, response code: 200 dump: 
2021/12/08 22:18:57 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/08 22:18:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestFlexReset1 (89.28s)
=== RUN   TestFlexReset2
2021/12/08 22:19:02 Waiting for app: TestFlexReset2 to get un-deployed
2021/12/08 22:19:02 App: TestFlexReset2 got un-deployed
2021/12/08 22:19:02 No eventing-con process running
2021/12/08 22:19:02 Function: TestFlexReset2 delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestFlexReset2 not found"
 }
}
2021/12/08 22:19:04 Context: Bucket flush, response code: 200 dump: 
2021/12/08 22:19:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 22:19:11 Context: Bucket flush, response code: 200 dump: 
2021/12/08 22:19:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 22:19:16 Starting bucket ops to source bucket
2021/12/08 22:19:16 Deploying app: TestFlexReset2. Item count src bucket: 0 dst bucket: 0
2021/12/08 22:19:17 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestFlexReset2' in metakv",
  "warnings": null
 }
}
2021/12/08 22:19:22 Waiting for app: TestFlexReset2 to get deployed
2021/12/08 22:19:28 Waiting for app: TestFlexReset2 to get deployed
2021/12/08 22:19:33 Waiting for app: TestFlexReset2 to get deployed
2021/12/08 22:19:38 Waiting for app: TestFlexReset2 to get deployed
2021/12/08 22:19:38 App: TestFlexReset2 got deployed
2021/12/08 22:19:43 Waiting for dst bucket item count to get to: 1000 curr count: 1
2021/12/08 22:19:43 src & dst bucket item count matched up. src bucket count: 1000 dst bucket count: 1000
2021/12/08 22:19:43 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 22:19:43 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 22:19:48 Waiting for app: TestFlexReset2 to get un-deployed
2021/12/08 22:19:53 Waiting for app: TestFlexReset2 to get un-deployed
2021/12/08 22:19:58 Waiting for app: TestFlexReset2 to get un-deployed
2021/12/08 22:20:03 Waiting for app: TestFlexReset2 to get un-deployed
2021/12/08 22:20:03 App: TestFlexReset2 got un-deployed
2021/12/08 22:20:03 No eventing-con process running
2021/12/08 22:20:03 Function: TestFlexReset2 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestFlexReset2 deleting in the background"
}
2021/12/08 22:20:04 Context: Bucket flush, response code: 200 dump: 
2021/12/08 22:20:14 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/08 22:20:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 22:20:16 Context: Bucket flush, response code: 200 dump: 
2021/12/08 22:20:26 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/08 22:20:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestFlexReset2 (89.25s)
=== RUN   TestN1QLLabelledBreak
2021/12/08 22:20:31 Waiting for app: TestN1QLLabelledBreak to get un-deployed
2021/12/08 22:20:31 App: TestN1QLLabelledBreak got un-deployed
2021/12/08 22:20:31 No eventing-con process running
2021/12/08 22:20:31 Function: TestN1QLLabelledBreak delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLLabelledBreak not found"
 }
}
2021/12/08 22:20:33 Context: Bucket flush, response code: 200 dump: 
2021/12/08 22:20:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 22:20:40 Context: Bucket flush, response code: 200 dump: 
2021/12/08 22:20:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 22:20:45 Deploying app: TestN1QLLabelledBreak. Item count src bucket: 0 dst bucket: 0
2021/12/08 22:20:46 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLLabelledBreak' in metakv",
  "warnings": null
 }
}
2021/12/08 22:20:51 Waiting for app: TestN1QLLabelledBreak to get deployed
2021/12/08 22:20:56 Waiting for app: TestN1QLLabelledBreak to get deployed
2021/12/08 22:21:01 Waiting for app: TestN1QLLabelledBreak to get deployed
2021/12/08 22:21:06 Waiting for app: TestN1QLLabelledBreak to get deployed
2021/12/08 22:21:06 App: TestN1QLLabelledBreak got deployed
2021/12/08 22:21:06 Starting bucket ops to source bucket
2021/12/08 22:21:15 Waiting for dst bucket item count to get to: 10000 curr count: 0
2021/12/08 22:21:20 Waiting for dst bucket item count to get to: 10000 curr count: 283
2021/12/08 22:21:25 Waiting for dst bucket item count to get to: 10000 curr count: 283
2021/12/08 22:21:31 Waiting for dst bucket item count to get to: 10000 curr count: 3737
2021/12/08 22:21:36 Waiting for dst bucket item count to get to: 10000 curr count: 3737
2021/12/08 22:21:41 Waiting for dst bucket item count to get to: 10000 curr count: 7833
2021/12/08 22:21:46 Waiting for dst bucket item count to get to: 10000 curr count: 7833
2021/12/08 22:21:46 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/08 22:21:46 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 22:21:46 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 22:21:51 Waiting for app: TestN1QLLabelledBreak to get un-deployed
2021/12/08 22:21:56 Waiting for app: TestN1QLLabelledBreak to get un-deployed
2021/12/08 22:22:01 Waiting for app: TestN1QLLabelledBreak to get un-deployed
2021/12/08 22:22:06 Waiting for app: TestN1QLLabelledBreak to get un-deployed
2021/12/08 22:22:06 App: TestN1QLLabelledBreak got un-deployed
2021/12/08 22:22:06 No eventing-con process running
2021/12/08 22:22:06 Function: TestN1QLLabelledBreak delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLLabelledBreak deleting in the background"
}
2021/12/08 22:22:08 Context: Bucket flush, response code: 200 dump: 
2021/12/08 22:22:18 Waiting for dst bucket item count to get to: 0 curr count: 2647
2021/12/08 22:22:23 Waiting for dst bucket item count to get to: 0 curr count: 147
2021/12/08 22:22:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 22:22:24 Context: Bucket flush, response code: 200 dump: 
2021/12/08 22:22:34 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/08 22:22:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLLabelledBreak (128.19s)
=== RUN   TestN1QLUnlabelledBreak
2021/12/08 22:22:39 Waiting for app: TestN1QLUnlabelledBreak to get un-deployed
2021/12/08 22:22:40 App: TestN1QLUnlabelledBreak got un-deployed
2021/12/08 22:22:40 No eventing-con process running
2021/12/08 22:22:40 Function: TestN1QLUnlabelledBreak delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLUnlabelledBreak not found"
 }
}
2021/12/08 22:22:41 Context: Bucket flush, response code: 200 dump: 
2021/12/08 22:22:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 22:22:48 Context: Bucket flush, response code: 200 dump: 
2021/12/08 22:22:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 22:22:53 Deploying app: TestN1QLUnlabelledBreak. Item count src bucket: 0 dst bucket: 0
2021/12/08 22:22:54 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLUnlabelledBreak' in metakv",
  "warnings": null
 }
}
2021/12/08 22:22:59 Waiting for app: TestN1QLUnlabelledBreak to get deployed
2021/12/08 22:23:04 Waiting for app: TestN1QLUnlabelledBreak to get deployed
2021/12/08 22:23:09 Waiting for app: TestN1QLUnlabelledBreak to get deployed
2021/12/08 22:23:14 Waiting for app: TestN1QLUnlabelledBreak to get deployed
2021/12/08 22:23:14 App: TestN1QLUnlabelledBreak got deployed
2021/12/08 22:23:14 Starting bucket ops to source bucket
2021/12/08 22:23:24 Waiting for dst bucket item count to get to: 10000 curr count: 0
2021/12/08 22:23:29 Waiting for dst bucket item count to get to: 10000 curr count: 328
2021/12/08 22:23:34 Waiting for dst bucket item count to get to: 10000 curr count: 991
2021/12/08 22:23:39 Waiting for dst bucket item count to get to: 10000 curr count: 3623
2021/12/08 22:23:44 Waiting for dst bucket item count to get to: 10000 curr count: 6562
2021/12/08 22:23:49 Waiting for dst bucket item count to get to: 10000 curr count: 8602
2021/12/08 22:23:49 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/08 22:23:49 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 22:23:49 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 22:23:54 Waiting for app: TestN1QLUnlabelledBreak to get un-deployed
2021/12/08 22:23:59 Waiting for app: TestN1QLUnlabelledBreak to get un-deployed
2021/12/08 22:24:04 Waiting for app: TestN1QLUnlabelledBreak to get un-deployed
2021/12/08 22:24:09 Waiting for app: TestN1QLUnlabelledBreak to get un-deployed
2021/12/08 22:24:09 App: TestN1QLUnlabelledBreak got un-deployed
2021/12/08 22:24:09 No eventing-con process running
2021/12/08 22:24:09 Function: TestN1QLUnlabelledBreak delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLUnlabelledBreak deleting in the background"
}
2021/12/08 22:24:11 Context: Bucket flush, response code: 200 dump: 
2021/12/08 22:24:21 Waiting for dst bucket item count to get to: 0 curr count: 4595
2021/12/08 22:24:26 Waiting for dst bucket item count to get to: 0 curr count: 4595
2021/12/08 22:24:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 22:24:28 Context: Bucket flush, response code: 200 dump: 
2021/12/08 22:24:38 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/08 22:24:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLUnlabelledBreak (123.11s)
=== RUN   TestN1QLThrowStatement
2021/12/08 22:24:43 Waiting for app: TestN1QLThrowStatement to get un-deployed
2021/12/08 22:24:43 App: TestN1QLThrowStatement got un-deployed
2021/12/08 22:24:43 No eventing-con process running
2021/12/08 22:24:43 Function: TestN1QLThrowStatement delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLThrowStatement not found"
 }
}
2021/12/08 22:24:44 Context: Bucket flush, response code: 200 dump: 
2021/12/08 22:24:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 22:24:51 Context: Bucket flush, response code: 200 dump: 
2021/12/08 22:24:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 22:24:56 Deploying app: TestN1QLThrowStatement. Item count src bucket: 0 dst bucket: 0
2021/12/08 22:24:57 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLThrowStatement' in metakv",
  "warnings": null
 }
}
2021/12/08 22:25:02 Waiting for app: TestN1QLThrowStatement to get deployed
2021/12/08 22:25:07 Waiting for app: TestN1QLThrowStatement to get deployed
2021/12/08 22:25:12 Waiting for app: TestN1QLThrowStatement to get deployed
2021/12/08 22:25:17 Waiting for app: TestN1QLThrowStatement to get deployed
2021/12/08 22:25:17 App: TestN1QLThrowStatement got deployed
2021/12/08 22:25:17 Starting bucket ops to source bucket
2021/12/08 22:25:27 Waiting for dst bucket item count to get to: 10000 curr count: 0
2021/12/08 22:25:32 Waiting for dst bucket item count to get to: 10000 curr count: 129
2021/12/08 22:25:37 Waiting for dst bucket item count to get to: 10000 curr count: 129
2021/12/08 22:25:42 Waiting for dst bucket item count to get to: 10000 curr count: 3213
2021/12/08 22:25:47 Waiting for dst bucket item count to get to: 10000 curr count: 3213
2021/12/08 22:25:52 Waiting for dst bucket item count to get to: 10000 curr count: 7282
2021/12/08 22:25:57 Waiting for dst bucket item count to get to: 10000 curr count: 7282
2021/12/08 22:25:57 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/08 22:25:57 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 22:25:57 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 22:26:02 Waiting for app: TestN1QLThrowStatement to get un-deployed
2021/12/08 22:26:07 Waiting for app: TestN1QLThrowStatement to get un-deployed
2021/12/08 22:26:12 Waiting for app: TestN1QLThrowStatement to get un-deployed
2021/12/08 22:26:17 Waiting for app: TestN1QLThrowStatement to get un-deployed
2021/12/08 22:26:17 App: TestN1QLThrowStatement got un-deployed
2021/12/08 22:26:17 No eventing-con process running
2021/12/08 22:26:17 Function: TestN1QLThrowStatement delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLThrowStatement deleting in the background"
}
2021/12/08 22:26:19 Context: Bucket flush, response code: 200 dump: 
2021/12/08 22:26:29 Waiting for dst bucket item count to get to: 0 curr count: 4827
2021/12/08 22:26:34 Waiting for dst bucket item count to get to: 0 curr count: 2327
2021/12/08 22:26:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 22:26:36 Context: Bucket flush, response code: 200 dump: 
2021/12/08 22:26:46 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/08 22:26:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLThrowStatement (128.22s)
=== RUN   TestN1QLNestedForLoop
2021/12/08 22:26:51 Waiting for app: TestN1QLNestedForLoop to get un-deployed
2021/12/08 22:26:51 App: TestN1QLNestedForLoop got un-deployed
2021/12/08 22:26:51 No eventing-con process running
2021/12/08 22:26:51 Function: TestN1QLNestedForLoop delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLNestedForLoop not found"
 }
}
2021/12/08 22:26:53 Context: Bucket flush, response code: 200 dump: 
2021/12/08 22:26:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 22:26:59 Context: Bucket flush, response code: 200 dump: 
2021/12/08 22:27:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 22:27:04 Deploying app: TestN1QLNestedForLoop. Item count src bucket: 0 dst bucket: 0
2021/12/08 22:27:06 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLNestedForLoop' in metakv",
  "warnings": null
 }
}
2021/12/08 22:27:11 Waiting for app: TestN1QLNestedForLoop to get deployed
2021/12/08 22:27:16 Waiting for app: TestN1QLNestedForLoop to get deployed
2021/12/08 22:27:21 Waiting for app: TestN1QLNestedForLoop to get deployed
2021/12/08 22:27:26 Waiting for app: TestN1QLNestedForLoop to get deployed
2021/12/08 22:27:26 App: TestN1QLNestedForLoop got deployed
2021/12/08 22:27:26 Starting bucket ops to source bucket
2021/12/08 22:27:35 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 22:27:40 Waiting for dst bucket item count to get to: 5000 curr count: 176
2021/12/08 22:27:45 Waiting for dst bucket item count to get to: 5000 curr count: 176
2021/12/08 22:27:50 Waiting for dst bucket item count to get to: 5000 curr count: 2283
2021/12/08 22:27:55 Waiting for dst bucket item count to get to: 5000 curr count: 2283
2021/12/08 22:28:00 Waiting for dst bucket item count to get to: 5000 curr count: 4709
2021/12/08 22:28:05 Waiting for dst bucket item count to get to: 5000 curr count: 4709
2021/12/08 22:28:05 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/08 22:28:05 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 22:28:05 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 22:28:10 Waiting for app: TestN1QLNestedForLoop to get un-deployed
2021/12/08 22:28:15 Waiting for app: TestN1QLNestedForLoop to get un-deployed
2021/12/08 22:28:20 Waiting for app: TestN1QLNestedForLoop to get un-deployed
2021/12/08 22:28:25 Waiting for app: TestN1QLNestedForLoop to get un-deployed
2021/12/08 22:28:25 App: TestN1QLNestedForLoop got un-deployed
2021/12/08 22:28:25 No eventing-con process running
2021/12/08 22:28:25 Function: TestN1QLNestedForLoop delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLNestedForLoop deleting in the background"
}
2021/12/08 22:28:27 Context: Bucket flush, response code: 200 dump: 
2021/12/08 22:28:37 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/08 22:28:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 22:28:39 Context: Bucket flush, response code: 200 dump: 
2021/12/08 22:28:49 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/08 22:28:54 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/08 22:28:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLNestedForLoop (128.13s)
=== RUN   TestN1QLAutoClose
2021/12/08 22:28:59 Waiting for app: TestN1QLAutoClose to get un-deployed
2021/12/08 22:28:59 App: TestN1QLAutoClose got un-deployed
2021/12/08 22:28:59 No eventing-con process running
2021/12/08 22:28:59 Function: TestN1QLAutoClose delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLAutoClose not found"
 }
}
2021/12/08 22:29:01 Context: Bucket flush, response code: 200 dump: 
2021/12/08 22:29:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 22:29:07 Context: Bucket flush, response code: 200 dump: 
2021/12/08 22:29:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 22:29:12 Deploying app: TestN1QLAutoClose. Item count src bucket: 0 dst bucket: 0
2021/12/08 22:29:14 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAutoClose' in metakv",
  "warnings": null
 }
}
2021/12/08 22:29:19 Waiting for app: TestN1QLAutoClose to get deployed
2021/12/08 22:29:24 Waiting for app: TestN1QLAutoClose to get deployed
2021/12/08 22:29:29 Waiting for app: TestN1QLAutoClose to get deployed
2021/12/08 22:29:34 Waiting for app: TestN1QLAutoClose to get deployed
2021/12/08 22:29:34 App: TestN1QLAutoClose got deployed
2021/12/08 22:29:34 Starting bucket ops to source bucket
2021/12/08 22:29:39 Waiting for dst bucket item count to get to: 500 curr count: 0
2021/12/08 22:29:44 Waiting for dst bucket item count to get to: 500 curr count: 4
2021/12/08 22:29:49 Waiting for dst bucket item count to get to: 500 curr count: 8
2021/12/08 22:29:54 Waiting for dst bucket item count to get to: 500 curr count: 17
2021/12/08 22:29:59 Waiting for dst bucket item count to get to: 500 curr count: 30
2021/12/08 22:30:04 Waiting for dst bucket item count to get to: 500 curr count: 41
2021/12/08 22:30:09 Waiting for dst bucket item count to get to: 500 curr count: 52
2021/12/08 22:30:14 Waiting for dst bucket item count to get to: 500 curr count: 61
2021/12/08 22:30:19 Waiting for dst bucket item count to get to: 500 curr count: 72
2021/12/08 22:30:24 Waiting for dst bucket item count to get to: 500 curr count: 83
2021/12/08 22:30:29 Waiting for dst bucket item count to get to: 500 curr count: 93
2021/12/08 22:30:34 Waiting for dst bucket item count to get to: 500 curr count: 102
2021/12/08 22:30:39 Waiting for dst bucket item count to get to: 500 curr count: 102
2021/12/08 22:30:44 Waiting for dst bucket item count to get to: 500 curr count: 113
2021/12/08 22:30:49 Waiting for dst bucket item count to get to: 500 curr count: 123
2021/12/08 22:30:54 Waiting for dst bucket item count to get to: 500 curr count: 135
2021/12/08 22:30:59 Waiting for dst bucket item count to get to: 500 curr count: 157
2021/12/08 22:31:04 Waiting for dst bucket item count to get to: 500 curr count: 157
2021/12/08 22:31:09 Waiting for dst bucket item count to get to: 500 curr count: 177
2021/12/08 22:31:14 Waiting for dst bucket item count to get to: 500 curr count: 177
2021/12/08 22:31:19 Waiting for dst bucket item count to get to: 500 curr count: 199
2021/12/08 22:31:25 Waiting for dst bucket item count to get to: 500 curr count: 199
2021/12/08 22:31:30 Waiting for dst bucket item count to get to: 500 curr count: 222
2021/12/08 22:31:35 Waiting for dst bucket item count to get to: 500 curr count: 222
2021/12/08 22:31:40 Waiting for dst bucket item count to get to: 500 curr count: 242
2021/12/08 22:31:45 Waiting for dst bucket item count to get to: 500 curr count: 242
2021/12/08 22:31:50 Waiting for dst bucket item count to get to: 500 curr count: 264
2021/12/08 22:31:55 Waiting for dst bucket item count to get to: 500 curr count: 264
2021/12/08 22:32:00 Waiting for dst bucket item count to get to: 500 curr count: 285
2021/12/08 22:32:05 Waiting for dst bucket item count to get to: 500 curr count: 285
2021/12/08 22:32:10 Waiting for dst bucket item count to get to: 500 curr count: 307
2021/12/08 22:32:15 Waiting for dst bucket item count to get to: 500 curr count: 307
2021/12/08 22:32:20 Waiting for dst bucket item count to get to: 500 curr count: 329
2021/12/08 22:32:25 Waiting for dst bucket item count to get to: 500 curr count: 329
2021/12/08 22:32:30 Waiting for dst bucket item count to get to: 500 curr count: 350
2021/12/08 22:32:35 Waiting for dst bucket item count to get to: 500 curr count: 350
2021/12/08 22:32:40 Waiting for dst bucket item count to get to: 500 curr count: 371
2021/12/08 22:32:45 Waiting for dst bucket item count to get to: 500 curr count: 371
2021/12/08 22:32:50 Waiting for dst bucket item count to get to: 500 curr count: 393
2021/12/08 22:32:55 Waiting for dst bucket item count to get to: 500 curr count: 393
2021/12/08 22:33:00 Waiting for dst bucket item count to get to: 500 curr count: 414
2021/12/08 22:33:05 Waiting for dst bucket item count to get to: 500 curr count: 414
2021/12/08 22:33:10 Waiting for dst bucket item count to get to: 500 curr count: 435
2021/12/08 22:33:15 Waiting for dst bucket item count to get to: 500 curr count: 448
2021/12/08 22:33:20 Waiting for dst bucket item count to get to: 500 curr count: 457
2021/12/08 22:33:25 Waiting for dst bucket item count to get to: 500 curr count: 468
2021/12/08 22:33:30 Waiting for dst bucket item count to get to: 500 curr count: 481
2021/12/08 22:33:35 Waiting for dst bucket item count to get to: 500 curr count: 490
2021/12/08 22:33:40 Waiting for dst bucket item count to get to: 500 curr count: 498
2021/12/08 22:33:40 src & dst bucket item count matched up. src bucket count: 500 dst bucket count: 500
2021/12/08 22:33:40 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 22:33:40 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 22:33:45 Waiting for app: TestN1QLAutoClose to get un-deployed
2021/12/08 22:33:50 Waiting for app: TestN1QLAutoClose to get un-deployed
2021/12/08 22:33:55 Waiting for app: TestN1QLAutoClose to get un-deployed
2021/12/08 22:34:00 Waiting for app: TestN1QLAutoClose to get un-deployed
2021/12/08 22:34:00 App: TestN1QLAutoClose got un-deployed
2021/12/08 22:34:00 No eventing-con process running
2021/12/08 22:34:00 Function: TestN1QLAutoClose delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLAutoClose deleting in the background"
}
2021/12/08 22:34:02 Context: Bucket flush, response code: 200 dump: 
2021/12/08 22:34:12 Waiting for dst bucket item count to get to: 0 curr count: 500
2021/12/08 22:34:17 Waiting for dst bucket item count to get to: 0 curr count: 500
2021/12/08 22:34:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 22:34:19 Context: Bucket flush, response code: 200 dump: 
2021/12/08 22:34:29 Waiting for dst bucket item count to get to: 0 curr count: 418
2021/12/08 22:34:34 Waiting for dst bucket item count to get to: 0 curr count: 168
2021/12/08 22:34:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLAutoClose (339.58s)
=== RUN   TestN1QLPosParams
2021/12/08 22:34:39 Waiting for app: TestN1QLPosParams to get un-deployed
2021/12/08 22:34:39 App: TestN1QLPosParams got un-deployed
2021/12/08 22:34:39 No eventing-con process running
2021/12/08 22:34:39 Function: TestN1QLPosParams delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLPosParams not found"
 }
}
2021/12/08 22:34:40 Context: Bucket flush, response code: 200 dump: 
2021/12/08 22:34:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 22:34:47 Context: Bucket flush, response code: 200 dump: 
2021/12/08 22:34:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 22:34:52 Deploying app: TestN1QLPosParams. Item count src bucket: 0 dst bucket: 0
2021/12/08 22:34:53 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLPosParams' in metakv",
  "warnings": null
 }
}
2021/12/08 22:34:58 Waiting for app: TestN1QLPosParams to get deployed
2021/12/08 22:35:03 Waiting for app: TestN1QLPosParams to get deployed
2021/12/08 22:35:08 Waiting for app: TestN1QLPosParams to get deployed
2021/12/08 22:35:13 Waiting for app: TestN1QLPosParams to get deployed
2021/12/08 22:35:13 App: TestN1QLPosParams got deployed
2021/12/08 22:35:13 Starting bucket ops to source bucket
2021/12/08 22:35:23 Waiting for dst bucket item count to get to: 5000 curr count: 632
2021/12/08 22:35:28 Waiting for dst bucket item count to get to: 5000 curr count: 632
2021/12/08 22:35:33 Waiting for dst bucket item count to get to: 5000 curr count: 1919
2021/12/08 22:35:38 Waiting for dst bucket item count to get to: 5000 curr count: 3787
2021/12/08 22:35:38 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/08 22:35:38 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 22:35:38 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 22:35:43 Waiting for app: TestN1QLPosParams to get un-deployed
2021/12/08 22:35:48 Waiting for app: TestN1QLPosParams to get un-deployed
2021/12/08 22:35:53 Waiting for app: TestN1QLPosParams to get un-deployed
2021/12/08 22:35:58 Waiting for app: TestN1QLPosParams to get un-deployed
2021/12/08 22:35:58 App: TestN1QLPosParams got un-deployed
2021/12/08 22:35:58 No eventing-con process running
2021/12/08 22:35:58 Function: TestN1QLPosParams delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLPosParams deleting in the background"
}
2021/12/08 22:36:00 Context: Bucket flush, response code: 200 dump: 
2021/12/08 22:36:10 Waiting for dst bucket item count to get to: 0 curr count: 3153
2021/12/08 22:36:15 Waiting for dst bucket item count to get to: 0 curr count: 3153
2021/12/08 22:36:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 22:36:17 Context: Bucket flush, response code: 200 dump: 
2021/12/08 22:36:27 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/08 22:36:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLPosParams (113.22s)
=== RUN   TestN1QLExhaustConnPool
2021/12/08 22:36:32 Waiting for app: TestN1QLExhaustConnPool to get un-deployed
2021/12/08 22:36:32 App: TestN1QLExhaustConnPool got un-deployed
2021/12/08 22:36:32 No eventing-con process running
2021/12/08 22:36:32 Function: TestN1QLExhaustConnPool delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLExhaustConnPool not found"
 }
}
2021/12/08 22:36:34 Context: Bucket flush, response code: 200 dump: 
2021/12/08 22:36:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 22:36:40 Context: Bucket flush, response code: 200 dump: 
2021/12/08 22:36:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 22:36:45 Deploying app: TestN1QLExhaustConnPool. Item count src bucket: 0 dst bucket: 0
2021/12/08 22:36:47 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLExhaustConnPool' in metakv",
  "warnings": null
 }
}
2021/12/08 22:36:52 Waiting for app: TestN1QLExhaustConnPool to get deployed
2021/12/08 22:36:57 Waiting for app: TestN1QLExhaustConnPool to get deployed
2021/12/08 22:37:02 Waiting for app: TestN1QLExhaustConnPool to get deployed
2021/12/08 22:37:07 Waiting for app: TestN1QLExhaustConnPool to get deployed
2021/12/08 22:37:07 App: TestN1QLExhaustConnPool got deployed
2021/12/08 22:37:07 Starting bucket ops to source bucket
2021/12/08 22:37:12 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/08 22:37:17 Waiting for dst bucket item count to get to: 100 curr count: 46
2021/12/08 22:37:22 Waiting for dst bucket item count to get to: 100 curr count: 46
2021/12/08 22:37:22 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/08 22:37:22 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 22:37:22 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 22:37:27 Waiting for app: TestN1QLExhaustConnPool to get un-deployed
2021/12/08 22:37:32 Waiting for app: TestN1QLExhaustConnPool to get un-deployed
2021/12/08 22:37:37 Waiting for app: TestN1QLExhaustConnPool to get un-deployed
2021/12/08 22:37:42 Waiting for app: TestN1QLExhaustConnPool to get un-deployed
2021/12/08 22:37:42 App: TestN1QLExhaustConnPool got un-deployed
2021/12/08 22:37:42 No eventing-con process running
2021/12/08 22:37:42 Function: TestN1QLExhaustConnPool delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLExhaustConnPool deleting in the background"
}
2021/12/08 22:37:44 Context: Bucket flush, response code: 200 dump: 
2021/12/08 22:37:54 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/08 22:37:59 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/08 22:37:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 22:38:01 Context: Bucket flush, response code: 200 dump: 
2021/12/08 22:38:11 Waiting for dst bucket item count to get to: 0 curr count: 82
2021/12/08 22:38:16 Waiting for dst bucket item count to get to: 0 curr count: 82
2021/12/08 22:38:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLExhaustConnPool (108.86s)
=== RUN   TestN1QLGraceSufficientTimeOut
2021/12/08 22:38:21 Waiting for app: TestN1QLGraceSufficientTimeOut to get un-deployed
2021/12/08 22:38:21 App: TestN1QLGraceSufficientTimeOut got un-deployed
2021/12/08 22:38:21 No eventing-con process running
2021/12/08 22:38:21 Function: TestN1QLGraceSufficientTimeOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLGraceSufficientTimeOut not found"
 }
}
2021/12/08 22:38:22 Context: Bucket flush, response code: 200 dump: 
2021/12/08 22:38:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 22:38:29 Context: Bucket flush, response code: 200 dump: 
2021/12/08 22:38:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 22:38:34 Starting bucket ops to source bucket
2021/12/08 22:38:51 Starting bucket ops to source bucket
2021/12/08 22:39:09 Deploying app: TestN1QLGraceSufficientTimeOut. Item count src bucket: 30000 dst bucket: 12183
2021/12/08 22:39:10 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLGraceSufficientTimeOut' in metakv",
  "warnings": null
 }
}
2021/12/08 22:39:15 Waiting for app: TestN1QLGraceSufficientTimeOut to get deployed
2021/12/08 22:39:20 Waiting for app: TestN1QLGraceSufficientTimeOut to get deployed
2021/12/08 22:39:25 Waiting for app: TestN1QLGraceSufficientTimeOut to get deployed
2021/12/08 22:39:30 Waiting for app: TestN1QLGraceSufficientTimeOut to get deployed
2021/12/08 22:39:30 App: TestN1QLGraceSufficientTimeOut got deployed
2021/12/08 22:39:30 Starting bucket ops to source bucket
2021/12/08 22:39:35 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/08 22:39:40 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/08 22:39:45 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/08 22:39:50 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/08 22:39:55 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/08 22:40:00 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/08 22:40:05 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/08 22:40:10 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/08 22:40:15 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/08 22:40:20 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/08 22:40:25 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/08 22:40:25 src & dst bucket item count matched up. src bucket count: 30001 dst bucket count: 30001
2021/12/08 22:40:25 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 22:40:25 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 22:40:30 Waiting for app: TestN1QLGraceSufficientTimeOut to get un-deployed
2021/12/08 22:40:35 Waiting for app: TestN1QLGraceSufficientTimeOut to get un-deployed
2021/12/08 22:40:40 Waiting for app: TestN1QLGraceSufficientTimeOut to get un-deployed
2021/12/08 22:40:45 Waiting for app: TestN1QLGraceSufficientTimeOut to get un-deployed
2021/12/08 22:40:45 App: TestN1QLGraceSufficientTimeOut got un-deployed
2021/12/08 22:40:45 No eventing-con process running
2021/12/08 22:40:45 Function: TestN1QLGraceSufficientTimeOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLGraceSufficientTimeOut deleting in the background"
}
2021/12/08 22:40:47 Context: Bucket flush, response code: 200 dump: 
2021/12/08 22:40:57 Waiting for dst bucket item count to get to: 0 curr count: 30000
2021/12/08 22:40:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 22:40:59 Context: Bucket flush, response code: 200 dump: 
2021/12/08 22:41:09 Waiting for dst bucket item count to get to: 0 curr count: 30001
2021/12/08 22:41:14 Waiting for dst bucket item count to get to: 0 curr count: 15001
2021/12/08 22:41:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLGraceSufficientTimeOut (178.37s)
=== RUN   TestN1QLQueryConsistency
2021/12/08 22:41:19 Waiting for app: n1qlquery_consistency to get un-deployed
2021/12/08 22:41:19 App: n1qlquery_consistency got un-deployed
2021/12/08 22:41:19 No eventing-con process running
2021/12/08 22:41:19 Function: n1qlquery_consistency delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: n1qlquery_consistency not found"
 }
}
2021/12/08 22:41:21 Context: Bucket flush, response code: 200 dump: 
2021/12/08 22:41:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 22:41:28 Context: Bucket flush, response code: 200 dump: 
2021/12/08 22:41:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 22:41:33 Starting bucket ops to source bucket
2021/12/08 22:41:36 Deploying app: n1qlquery_consistency. Item count src bucket: 0 dst bucket: 0
2021/12/08 22:41:37 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'n1qlquery_consistency' in metakv",
  "warnings": null
 }
}
2021/12/08 22:41:42 Waiting for app: n1qlquery_consistency to get deployed
2021/12/08 22:41:47 Waiting for app: n1qlquery_consistency to get deployed
2021/12/08 22:41:52 Waiting for app: n1qlquery_consistency to get deployed
2021/12/08 22:41:57 Waiting for app: n1qlquery_consistency to get deployed
2021/12/08 22:41:57 App: n1qlquery_consistency got deployed
2021/12/08 22:42:02 Waiting for dst bucket item count to get to: 5000 curr count: 67
2021/12/08 22:42:07 Waiting for dst bucket item count to get to: 5000 curr count: 2567
2021/12/08 22:42:07 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/08 22:42:07 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 22:42:07 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 22:42:12 Waiting for app: n1qlquery_consistency to get un-deployed
2021/12/08 22:42:17 Waiting for app: n1qlquery_consistency to get un-deployed
2021/12/08 22:42:22 Waiting for app: n1qlquery_consistency to get un-deployed
2021/12/08 22:42:27 Waiting for app: n1qlquery_consistency to get un-deployed
2021/12/08 22:42:27 App: n1qlquery_consistency got un-deployed
2021/12/08 22:42:27 No eventing-con process running
2021/12/08 22:42:27 Function: n1qlquery_consistency delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: n1qlquery_consistency deleting in the background"
}
2021/12/08 22:42:29 Context: Bucket flush, response code: 200 dump: 
2021/12/08 22:42:39 Waiting for dst bucket item count to get to: 0 curr count: 4724
2021/12/08 22:42:44 Waiting for dst bucket item count to get to: 0 curr count: 2224
2021/12/08 22:42:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 22:42:45 Context: Bucket flush, response code: 200 dump: 
2021/12/08 22:42:55 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/08 22:42:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLQueryConsistency (101.52s)
PASS
ok  	github.com/couchbase/eventing/tests/functional_tests	1558.182s

N1ql tests took 0 hours, 26 minutes and 3 seconds.

Starting server
Starting server: attempt 1

Rebalance tests

2021/12/08 22:44:31 Started rebalance
Post "http://127.0.0.1:9000/pools/default": EOF
Quota setup Post "http://127.0.0.1:9000/pools/default": EOF
=== RUN   TestDuplicateWithEventingRebIn
2021/12/08 22:44:49 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
running apps fetch error json: cannot unmarshal number into Go value of type string
2021/12/08 22:44:54 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
2021/12/08 22:44:54 App: TestDuplicateWithEventingRebIn got un-deployed
2021/12/08 22:44:54 No eventing-con process running
2021/12/08 22:44:54 Function: TestDuplicateWithEventingRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithEventingRebIn not found"
 }
}
2021/12/08 22:44:56 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 22:45:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 22:45:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 22:45:03 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 22:45:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 22:45:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 22:45:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 22:45:13 Deploying app: TestDuplicateWithEventingRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/08 22:45:14 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithEventingRebIn' in metakv",
  "warnings": null
 }
}
2021/12/08 22:45:19 Waiting for app: TestDuplicateWithEventingRebIn to get deployed
2021/12/08 22:45:24 Waiting for app: TestDuplicateWithEventingRebIn to get deployed
2021/12/08 22:45:29 Waiting for app: TestDuplicateWithEventingRebIn to get deployed
2021/12/08 22:45:34 Waiting for app: TestDuplicateWithEventingRebIn to get deployed
2021/12/08 22:45:34 App: TestDuplicateWithEventingRebIn got deployed
2021/12/08 22:45:34 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/08 22:45:34 Starting bucket ops to source bucket
2021/12/08 22:45:49 Removing node(s): [] from the cluster
2021/12/08 22:45:54 Rebalance progress: 0
2021/12/08 22:45:59 Rebalance progress: 0
2021/12/08 22:46:04 Rebalance progress: 25
2021/12/08 22:46:09 Rebalance progress: 25
2021/12/08 22:46:14 Rebalance progress: 25
2021/12/08 22:46:19 Rebalance progress: 97.3529411764706
2021/12/08 22:46:24 Rebalance progress: 99.11764705882354
2021/12/08 22:46:29 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 22:46:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 22:46:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 22:46:29 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9302/api/v1/stats": dial tcp 127.0.0.1:9302: connect: connection refused
2021/12/08 22:46:29 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/08 22:46:29 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/08 22:46:34 Rebalance progress: 25
2021/12/08 22:46:39 Rebalance progress: 25
2021/12/08 22:46:44 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 22:46:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 22:46:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 22:46:44 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9302/api/v1/stats": dial tcp 127.0.0.1:9302: connect: connection refused
2021/12/08 22:46:44 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/08 22:46:44 Waiting for src bucket item count to get to: 200000 curr count: 92317
2021/12/08 22:46:49 Waiting for src bucket item count to get to: 200000 curr count: 108719
2021/12/08 22:46:54 Waiting for src bucket item count to get to: 200000 curr count: 125145
2021/12/08 22:46:59 Waiting for src bucket item count to get to: 200000 curr count: 125145
2021/12/08 22:47:04 Waiting for src bucket item count to get to: 200000 curr count: 142265
2021/12/08 22:47:09 Waiting for src bucket item count to get to: 200000 curr count: 142265
2021/12/08 22:47:14 Waiting for src bucket item count to get to: 200000 curr count: 159124
2021/12/08 22:47:19 Waiting for src bucket item count to get to: 200000 curr count: 159124
2021/12/08 22:47:24 Waiting for src bucket item count to get to: 200000 curr count: 175863
2021/12/08 22:47:29 Waiting for src bucket item count to get to: 200000 curr count: 175863
2021/12/08 22:47:34 Waiting for src bucket item count to get to: 200000 curr count: 192345
2021/12/08 22:47:39 Waiting for src bucket item count to get to: 200000 curr count: 192345
2021/12/08 22:47:44 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/08 22:47:44 Post SET ops. Expected item count: 200000 got 200000
2021/12/08 22:47:44 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 22:47:49 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
2021/12/08 22:47:54 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
2021/12/08 22:47:59 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
2021/12/08 22:48:04 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
2021/12/08 22:48:04 App: TestDuplicateWithEventingRebIn got un-deployed
2021/12/08 22:48:04 No eventing-con process running
2021/12/08 22:48:04 Function: TestDuplicateWithEventingRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithEventingRebIn deleting in the background"
}
2021/12/08 22:48:06 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 22:48:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 22:48:16 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/08 22:48:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 22:48:18 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 22:48:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 22:48:28 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/08 22:48:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithEventingRebIn (228.17s)
=== RUN   TestDuplicateWithEventingRebOut
2021/12/08 22:48:33 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/08 22:48:33 Removing node(s): [] from the cluster
2021/12/08 22:48:38 Rebalance progress: 100
2021/12/08 22:48:43 Waiting for app: TestDuplicateWithEventingRebOut to get un-deployed
2021/12/08 22:48:43 App: TestDuplicateWithEventingRebOut got un-deployed
2021/12/08 22:48:43 No eventing-con process running
2021/12/08 22:48:43 Function: TestDuplicateWithEventingRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithEventingRebOut not found"
 }
}
2021/12/08 22:48:45 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 22:48:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 22:48:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 22:48:51 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 22:48:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 22:48:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 22:49:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 22:49:01 Deploying app: TestDuplicateWithEventingRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/08 22:49:02 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithEventingRebOut' in metakv",
  "warnings": null
 }
}
2021/12/08 22:49:07 Waiting for app: TestDuplicateWithEventingRebOut to get deployed
2021/12/08 22:49:12 Waiting for app: TestDuplicateWithEventingRebOut to get deployed
2021/12/08 22:49:17 Waiting for app: TestDuplicateWithEventingRebOut to get deployed
2021/12/08 22:49:22 Waiting for app: TestDuplicateWithEventingRebOut to get deployed
2021/12/08 22:49:22 App: TestDuplicateWithEventingRebOut got deployed
2021/12/08 22:49:22 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/08 22:49:22 Starting bucket ops to source bucket
2021/12/08 22:49:28 Rebalance progress: 25
2021/12/08 22:49:33 Rebalance progress: 25
2021/12/08 22:49:38 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 22:49:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 22:49:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 22:49:38 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9302/api/v1/stats": dial tcp 127.0.0.1:9302: connect: connection refused
2021/12/08 22:49:38 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/08 22:49:38 Waiting for src bucket item count to get to: 200000 curr count: 10969
2021/12/08 22:49:43 Waiting for src bucket item count to get to: 200000 curr count: 27828
2021/12/08 22:49:48 Waiting for src bucket item count to get to: 200000 curr count: 27828
2021/12/08 22:49:53 Waiting for src bucket item count to get to: 200000 curr count: 44768
2021/12/08 22:49:58 Waiting for src bucket item count to get to: 200000 curr count: 44768
2021/12/08 22:50:03 Waiting for src bucket item count to get to: 200000 curr count: 61888
2021/12/08 22:50:08 Waiting for src bucket item count to get to: 200000 curr count: 61888
2021/12/08 22:50:13 Waiting for src bucket item count to get to: 200000 curr count: 61888
2021/12/08 22:50:18 Waiting for src bucket item count to get to: 200000 curr count: 78409
2021/12/08 22:50:23 Waiting for src bucket item count to get to: 200000 curr count: 78409
2021/12/08 22:50:28 Waiting for src bucket item count to get to: 200000 curr count: 95165
2021/12/08 22:50:33 Waiting for src bucket item count to get to: 200000 curr count: 95165
2021/12/08 22:50:38 Waiting for src bucket item count to get to: 200000 curr count: 111937
2021/12/08 22:50:43 Waiting for src bucket item count to get to: 200000 curr count: 111937
2021/12/08 22:50:48 Waiting for src bucket item count to get to: 200000 curr count: 128832
2021/12/08 22:50:53 Waiting for src bucket item count to get to: 200000 curr count: 128832
2021/12/08 22:50:58 Waiting for src bucket item count to get to: 200000 curr count: 145731
2021/12/08 22:51:03 Waiting for src bucket item count to get to: 200000 curr count: 145731
2021/12/08 22:51:08 Waiting for src bucket item count to get to: 200000 curr count: 162699
2021/12/08 22:51:13 Waiting for src bucket item count to get to: 200000 curr count: 162699
2021/12/08 22:51:18 Waiting for src bucket item count to get to: 200000 curr count: 179097
2021/12/08 22:51:23 Waiting for src bucket item count to get to: 200000 curr count: 179097
2021/12/08 22:51:28 Waiting for src bucket item count to get to: 200000 curr count: 195983
2021/12/08 22:51:33 Waiting for src bucket item count to get to: 200000 curr count: 195983
2021/12/08 22:51:38 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/08 22:51:38 Post SET ops. Expected item count: 200000 got 200000
2021/12/08 22:51:38 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 22:51:43 Waiting for app: TestDuplicateWithEventingRebOut to get un-deployed
2021/12/08 22:51:48 Waiting for app: TestDuplicateWithEventingRebOut to get un-deployed
2021/12/08 22:51:53 Waiting for app: TestDuplicateWithEventingRebOut to get un-deployed
2021/12/08 22:51:58 Waiting for app: TestDuplicateWithEventingRebOut to get un-deployed
2021/12/08 22:51:58 App: TestDuplicateWithEventingRebOut got un-deployed
2021/12/08 22:51:58 No eventing-con process running
2021/12/08 22:51:58 Function: TestDuplicateWithEventingRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithEventingRebOut deleting in the background"
}
2021/12/08 22:51:59 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 22:52:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 22:52:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 22:52:06 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 22:52:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 22:52:16 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/08 22:52:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithEventingRebOut (228.61s)
=== RUN   TestDuplicateWithEventingSwapReb
2021/12/08 22:52:26 Waiting for app: TestDuplicateWithEventingSwapReb to get un-deployed
2021/12/08 22:52:26 App: TestDuplicateWithEventingSwapReb got un-deployed
2021/12/08 22:52:26 No eventing-con process running
2021/12/08 22:52:26 Function: TestDuplicateWithEventingSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithEventingSwapReb not found"
 }
}
2021/12/08 22:52:28 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 22:52:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 22:52:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 22:52:35 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 22:52:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 22:52:40 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 22:52:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 22:52:45 Deploying app: TestDuplicateWithEventingSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/08 22:52:46 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithEventingSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/08 22:52:51 Waiting for app: TestDuplicateWithEventingSwapReb to get deployed
2021/12/08 22:52:56 Waiting for app: TestDuplicateWithEventingSwapReb to get deployed
2021/12/08 22:53:01 Waiting for app: TestDuplicateWithEventingSwapReb to get deployed
2021/12/08 22:53:06 Waiting for app: TestDuplicateWithEventingSwapReb to get deployed
2021/12/08 22:53:06 App: TestDuplicateWithEventingSwapReb got deployed
2021/12/08 22:53:06 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/08 22:53:06 Starting bucket ops to source bucket
2021/12/08 22:53:06 Removing node(s): [] from the cluster
2021/12/08 22:53:11 Rebalance progress: 25
2021/12/08 22:53:16 Rebalance progress: 25
2021/12/08 22:53:21 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 22:53:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 22:53:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 22:53:21 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9302/api/v1/stats": dial tcp 127.0.0.1:9302: connect: connection refused
2021/12/08 22:53:21 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/08 22:53:21 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/08 22:53:29 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/08 22:53:34 Rebalance progress: 16.66666666666667
2021/12/08 22:53:39 Rebalance progress: 16.66666666666667
2021/12/08 22:53:44 Rebalance progress: 16.66666666666667
2021/12/08 22:53:49 Rebalance progress: 72.22222222222223
2021/12/08 22:53:54 Rebalance progress: 72.22222222222223
2021/12/08 22:53:59 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 22:53:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 22:53:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 22:53:59 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/08 22:53:59 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/08 22:54:04 Rebalance progress: 16.66666666666667
2021/12/08 22:54:09 Rebalance progress: 16.66666666666667
2021/12/08 22:54:14 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 22:54:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 22:54:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 22:54:14 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/08 22:54:14 Waiting for src bucket item count to get to: 200000 curr count: 98921
2021/12/08 22:54:19 Waiting for src bucket item count to get to: 200000 curr count: 98921
2021/12/08 22:54:24 Waiting for src bucket item count to get to: 200000 curr count: 115527
2021/12/08 22:54:29 Waiting for src bucket item count to get to: 200000 curr count: 115527
2021/12/08 22:54:34 Waiting for src bucket item count to get to: 200000 curr count: 132244
2021/12/08 22:54:39 Waiting for src bucket item count to get to: 200000 curr count: 132244
2021/12/08 22:54:44 Waiting for src bucket item count to get to: 200000 curr count: 148554
2021/12/08 22:54:49 Waiting for src bucket item count to get to: 200000 curr count: 148554
2021/12/08 22:54:54 Waiting for src bucket item count to get to: 200000 curr count: 165134
2021/12/08 22:54:59 Waiting for src bucket item count to get to: 200000 curr count: 165134
2021/12/08 22:55:04 Waiting for src bucket item count to get to: 200000 curr count: 181645
2021/12/08 22:55:09 Waiting for src bucket item count to get to: 200000 curr count: 181645
2021/12/08 22:55:14 Waiting for src bucket item count to get to: 200000 curr count: 198357
2021/12/08 22:55:19 Waiting for src bucket item count to get to: 200000 curr count: 198357
2021/12/08 22:55:24 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/08 22:55:24 Post SET ops. Expected item count: 200000 got 200000
2021/12/08 22:55:24 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 22:55:29 Waiting for app: TestDuplicateWithEventingSwapReb to get un-deployed
2021/12/08 22:55:34 Waiting for app: TestDuplicateWithEventingSwapReb to get un-deployed
2021/12/08 22:55:39 Waiting for app: TestDuplicateWithEventingSwapReb to get un-deployed
2021/12/08 22:55:44 Waiting for app: TestDuplicateWithEventingSwapReb to get un-deployed
2021/12/08 22:55:44 App: TestDuplicateWithEventingSwapReb got un-deployed
2021/12/08 22:55:44 No eventing-con process running
2021/12/08 22:55:44 Function: TestDuplicateWithEventingSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithEventingSwapReb deleting in the background"
}
2021/12/08 22:55:46 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 22:55:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 22:55:56 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/08 22:55:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 22:55:57 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 22:56:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 22:56:07 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/08 22:56:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithEventingSwapReb (231.33s)
=== RUN   TestDuplicateWithKVRebIn
2021/12/08 22:56:18 Waiting for app: TestDuplicateWithKVRebIn to get un-deployed
2021/12/08 22:56:18 App: TestDuplicateWithKVRebIn got un-deployed
2021/12/08 22:56:18 No eventing-con process running
2021/12/08 22:56:18 Function: TestDuplicateWithKVRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithKVRebIn not found"
 }
}
2021/12/08 22:56:19 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 22:56:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 22:56:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 22:56:26 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 22:56:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 22:56:31 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 22:56:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 22:56:36 Deploying app: TestDuplicateWithKVRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/08 22:56:37 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithKVRebIn' in metakv",
  "warnings": null
 }
}
2021/12/08 22:56:42 Waiting for app: TestDuplicateWithKVRebIn to get deployed
2021/12/08 22:56:47 Waiting for app: TestDuplicateWithKVRebIn to get deployed
2021/12/08 22:56:52 Waiting for app: TestDuplicateWithKVRebIn to get deployed
2021/12/08 22:56:57 Waiting for app: TestDuplicateWithKVRebIn to get deployed
2021/12/08 22:56:57 App: TestDuplicateWithKVRebIn got deployed
2021/12/08 22:56:57 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/08 22:56:57 Starting bucket ops to source bucket
2021/12/08 22:56:57 Removing node(s): [] from the cluster
2021/12/08 22:57:02 Rebalance progress: 16.66666666666667
2021/12/08 22:57:07 Rebalance progress: 16.66666666666667
2021/12/08 22:57:12 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 22:57:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 22:57:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 22:57:13 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/08 22:57:13 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/08 22:57:18 Rebalance progress: 16.66666666666667
2021/12/08 22:57:23 Rebalance progress: 16.66666666666667
2021/12/08 22:57:28 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 22:57:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 22:57:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 22:57:28 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/08 22:57:28 Waiting for src bucket item count to get to: 200000 curr count: 35189
2021/12/08 22:57:33 Waiting for src bucket item count to get to: 200000 curr count: 51886
2021/12/08 22:57:38 Waiting for src bucket item count to get to: 200000 curr count: 51886
2021/12/08 22:57:43 Waiting for src bucket item count to get to: 200000 curr count: 68840
2021/12/08 22:57:48 Waiting for src bucket item count to get to: 200000 curr count: 68840
2021/12/08 22:57:53 Waiting for src bucket item count to get to: 200000 curr count: 84868
2021/12/08 22:57:58 Waiting for src bucket item count to get to: 200000 curr count: 84868
2021/12/08 22:58:03 Waiting for src bucket item count to get to: 200000 curr count: 101211
2021/12/08 22:58:08 Waiting for src bucket item count to get to: 200000 curr count: 101211
2021/12/08 22:58:13 Waiting for src bucket item count to get to: 200000 curr count: 101211
2021/12/08 22:58:18 Waiting for src bucket item count to get to: 200000 curr count: 117685
2021/12/08 22:58:23 Waiting for src bucket item count to get to: 200000 curr count: 117685
2021/12/08 22:58:28 Waiting for src bucket item count to get to: 200000 curr count: 134209
2021/12/08 22:58:33 Waiting for src bucket item count to get to: 200000 curr count: 134209
2021/12/08 22:58:38 Waiting for src bucket item count to get to: 200000 curr count: 150584
2021/12/08 22:58:43 Waiting for src bucket item count to get to: 200000 curr count: 150584
2021/12/08 22:58:48 Waiting for src bucket item count to get to: 200000 curr count: 167434
2021/12/08 22:58:53 Waiting for src bucket item count to get to: 200000 curr count: 167434
2021/12/08 22:58:58 Waiting for src bucket item count to get to: 200000 curr count: 183582
2021/12/08 22:59:03 Waiting for src bucket item count to get to: 200000 curr count: 183582
2021/12/08 22:59:08 Waiting for src bucket item count to get to: 200000 curr count: 199982
2021/12/08 22:59:13 Waiting for src bucket item count to get to: 200000 curr count: 199982
2021/12/08 22:59:18 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/08 22:59:18 Post SET ops. Expected item count: 200000 got 200000
2021/12/08 22:59:18 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 22:59:23 Waiting for app: TestDuplicateWithKVRebIn to get un-deployed
2021/12/08 22:59:28 Waiting for app: TestDuplicateWithKVRebIn to get un-deployed
2021/12/08 22:59:33 Waiting for app: TestDuplicateWithKVRebIn to get un-deployed
2021/12/08 22:59:38 Waiting for app: TestDuplicateWithKVRebIn to get un-deployed
2021/12/08 22:59:38 App: TestDuplicateWithKVRebIn got un-deployed
2021/12/08 22:59:38 No eventing-con process running
2021/12/08 22:59:38 Function: TestDuplicateWithKVRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithKVRebIn deleting in the background"
}
2021/12/08 22:59:40 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 22:59:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 22:59:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 22:59:46 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 22:59:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 22:59:56 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/08 22:59:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithKVRebIn (228.88s)
=== RUN   TestDuplicateWithKVRebOut
2021/12/08 23:00:01 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/08 23:00:02 Removing node(s): [] from the cluster
2021/12/08 23:00:07 Rebalance progress: 100
2021/12/08 23:00:12 Waiting for app: TestDuplicateWithKVRebOut to get un-deployed
2021/12/08 23:00:12 App: TestDuplicateWithKVRebOut got un-deployed
2021/12/08 23:00:12 No eventing-con process running
2021/12/08 23:00:12 Function: TestDuplicateWithKVRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithKVRebOut not found"
 }
}
2021/12/08 23:00:13 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:00:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 23:00:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 23:00:20 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:00:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 23:00:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:00:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 23:00:30 Deploying app: TestDuplicateWithKVRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/08 23:00:31 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithKVRebOut' in metakv",
  "warnings": null
 }
}
2021/12/08 23:00:36 Waiting for app: TestDuplicateWithKVRebOut to get deployed
2021/12/08 23:00:41 Waiting for app: TestDuplicateWithKVRebOut to get deployed
2021/12/08 23:00:46 Waiting for app: TestDuplicateWithKVRebOut to get deployed
2021/12/08 23:00:51 Waiting for app: TestDuplicateWithKVRebOut to get deployed
2021/12/08 23:00:51 App: TestDuplicateWithKVRebOut got deployed
2021/12/08 23:00:51 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/08 23:00:51 Starting bucket ops to source bucket
2021/12/08 23:00:56 Rebalance progress: 16.66666666666667
2021/12/08 23:01:01 Rebalance progress: 16.66666666666667
2021/12/08 23:01:06 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:01:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:01:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 23:01:06 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/08 23:01:06 Waiting for src bucket item count to get to: 200000 curr count: 12468
2021/12/08 23:01:11 Waiting for src bucket item count to get to: 200000 curr count: 29004
2021/12/08 23:01:17 Waiting for src bucket item count to get to: 200000 curr count: 29004
2021/12/08 23:01:22 Waiting for src bucket item count to get to: 200000 curr count: 45668
2021/12/08 23:01:27 Waiting for src bucket item count to get to: 200000 curr count: 45668
2021/12/08 23:01:32 Waiting for src bucket item count to get to: 200000 curr count: 62124
2021/12/08 23:01:37 Waiting for src bucket item count to get to: 200000 curr count: 62124
2021/12/08 23:01:42 Waiting for src bucket item count to get to: 200000 curr count: 62124
2021/12/08 23:01:47 Waiting for src bucket item count to get to: 200000 curr count: 77816
2021/12/08 23:01:52 Waiting for src bucket item count to get to: 200000 curr count: 77816
2021/12/08 23:01:57 Waiting for src bucket item count to get to: 200000 curr count: 93792
2021/12/08 23:02:02 Waiting for src bucket item count to get to: 200000 curr count: 93792
2021/12/08 23:02:07 Waiting for src bucket item count to get to: 200000 curr count: 110137
2021/12/08 23:02:12 Waiting for src bucket item count to get to: 200000 curr count: 110137
2021/12/08 23:02:17 Waiting for src bucket item count to get to: 200000 curr count: 126509
2021/12/08 23:02:22 Waiting for src bucket item count to get to: 200000 curr count: 126509
2021/12/08 23:02:27 Waiting for src bucket item count to get to: 200000 curr count: 142979
2021/12/08 23:02:32 Waiting for src bucket item count to get to: 200000 curr count: 142979
2021/12/08 23:02:37 Waiting for src bucket item count to get to: 200000 curr count: 159268
2021/12/08 23:02:42 Waiting for src bucket item count to get to: 200000 curr count: 159268
2021/12/08 23:02:47 Waiting for src bucket item count to get to: 200000 curr count: 175031
2021/12/08 23:02:52 Waiting for src bucket item count to get to: 200000 curr count: 175031
2021/12/08 23:02:57 Waiting for src bucket item count to get to: 200000 curr count: 191015
2021/12/08 23:03:02 Waiting for src bucket item count to get to: 200000 curr count: 191015
2021/12/08 23:03:07 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/08 23:03:07 Post SET ops. Expected item count: 200000 got 200000
2021/12/08 23:03:07 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 23:03:12 Waiting for app: TestDuplicateWithKVRebOut to get un-deployed
2021/12/08 23:03:17 Waiting for app: TestDuplicateWithKVRebOut to get un-deployed
2021/12/08 23:03:22 Waiting for app: TestDuplicateWithKVRebOut to get un-deployed
2021/12/08 23:03:27 Waiting for app: TestDuplicateWithKVRebOut to get un-deployed
2021/12/08 23:03:27 App: TestDuplicateWithKVRebOut got un-deployed
2021/12/08 23:03:27 No eventing-con process running
2021/12/08 23:03:27 Function: TestDuplicateWithKVRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithKVRebOut deleting in the background"
}
2021/12/08 23:03:29 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:03:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 23:03:39 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/08 23:03:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 23:03:40 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:03:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 23:03:50 Waiting for dst bucket item count to get to: 0 curr count: 76663
2021/12/08 23:03:55 Waiting for dst bucket item count to get to: 0 curr count: 76663
2021/12/08 23:03:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithKVRebOut (238.91s)
=== RUN   TestDuplicateWithKVSwapReb
2021/12/08 23:04:05 Waiting for app: TestDuplicateWithKVSwapReb to get un-deployed
2021/12/08 23:04:05 App: TestDuplicateWithKVSwapReb got un-deployed
2021/12/08 23:04:05 No eventing-con process running
2021/12/08 23:04:05 Function: TestDuplicateWithKVSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithKVSwapReb not found"
 }
}
2021/12/08 23:04:07 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:04:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 23:04:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 23:04:14 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:04:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 23:04:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:04:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 23:04:24 Deploying app: TestDuplicateWithKVSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/08 23:04:25 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithKVSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/08 23:04:30 Waiting for app: TestDuplicateWithKVSwapReb to get deployed
2021/12/08 23:04:35 Waiting for app: TestDuplicateWithKVSwapReb to get deployed
2021/12/08 23:04:40 Waiting for app: TestDuplicateWithKVSwapReb to get deployed
2021/12/08 23:04:45 Waiting for app: TestDuplicateWithKVSwapReb to get deployed
2021/12/08 23:04:45 App: TestDuplicateWithKVSwapReb got deployed
2021/12/08 23:04:45 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/08 23:04:45 Starting bucket ops to source bucket
2021/12/08 23:04:45 Removing node(s): [] from the cluster
2021/12/08 23:04:50 Rebalance progress: 16.66666666666667
2021/12/08 23:04:55 Rebalance progress: 16.66666666666667
2021/12/08 23:05:00 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:05:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:05:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 23:05:00 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/08 23:05:00 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/08 23:05:01 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/08 23:05:06 Rebalance progress: 16.66666666666667
2021/12/08 23:05:11 Rebalance progress: 16.66666666666667
2021/12/08 23:05:16 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:05:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:05:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 23:05:16 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/08 23:05:16 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/08 23:05:21 Rebalance progress: 16.66666666666667
2021/12/08 23:05:26 Rebalance progress: 16.66666666666667
2021/12/08 23:05:31 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:05:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:05:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 23:05:31 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/08 23:05:31 Waiting for src bucket item count to get to: 200000 curr count: 70521
2021/12/08 23:05:36 Waiting for src bucket item count to get to: 200000 curr count: 70521
2021/12/08 23:05:41 Waiting for src bucket item count to get to: 200000 curr count: 86331
2021/12/08 23:05:46 Waiting for src bucket item count to get to: 200000 curr count: 86331
2021/12/08 23:05:51 Waiting for src bucket item count to get to: 200000 curr count: 102850
2021/12/08 23:05:56 Waiting for src bucket item count to get to: 200000 curr count: 102850
2021/12/08 23:06:01 Waiting for src bucket item count to get to: 200000 curr count: 119485
2021/12/08 23:06:06 Waiting for src bucket item count to get to: 200000 curr count: 119485
2021/12/08 23:06:11 Waiting for src bucket item count to get to: 200000 curr count: 135919
2021/12/08 23:06:16 Waiting for src bucket item count to get to: 200000 curr count: 135919
2021/12/08 23:06:21 Waiting for src bucket item count to get to: 200000 curr count: 152105
2021/12/08 23:06:26 Waiting for src bucket item count to get to: 200000 curr count: 152105
2021/12/08 23:06:31 Waiting for src bucket item count to get to: 200000 curr count: 168778
2021/12/08 23:06:36 Waiting for src bucket item count to get to: 200000 curr count: 168778
2021/12/08 23:06:41 Waiting for src bucket item count to get to: 200000 curr count: 184895
2021/12/08 23:06:47 Waiting for src bucket item count to get to: 200000 curr count: 184895
2021/12/08 23:06:52 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/08 23:06:52 Post SET ops. Expected item count: 200000 got 200000
2021/12/08 23:06:52 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 23:06:57 Waiting for app: TestDuplicateWithKVSwapReb to get un-deployed
2021/12/08 23:07:02 Waiting for app: TestDuplicateWithKVSwapReb to get un-deployed
2021/12/08 23:07:07 Waiting for app: TestDuplicateWithKVSwapReb to get un-deployed
2021/12/08 23:07:12 Waiting for app: TestDuplicateWithKVSwapReb to get un-deployed
2021/12/08 23:07:12 App: TestDuplicateWithKVSwapReb got un-deployed
2021/12/08 23:07:12 No eventing-con process running
2021/12/08 23:07:12 Function: TestDuplicateWithKVSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithKVSwapReb deleting in the background"
}
2021/12/08 23:07:13 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:07:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 23:07:23 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/08 23:07:28 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/08 23:07:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 23:07:30 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:07:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 23:07:40 Waiting for dst bucket item count to get to: 0 curr count: 57199
2021/12/08 23:07:45 Waiting for dst bucket item count to get to: 0 curr count: 57199
2021/12/08 23:07:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithKVSwapReb (229.88s)
=== RUN   TestEventingRebKVOpsOnUpdateBucketOpOneByOne
2021/12/08 23:07:55 Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/08 23:07:55 App: TestEventingRebKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/08 23:07:55 No eventing-con process running
2021/12/08 23:07:55 Function: TestEventingRebKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpOneByOne not found"
 }
}
2021/12/08 23:07:57 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:08:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 23:08:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 23:08:04 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:08:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 23:08:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:08:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 23:08:14 Deploying app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne. Item count src bucket: 0 dst bucket: 0
2021/12/08 23:08:15 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebKVOpsOnUpdateBucketOpOneByOne' in metakv",
  "warnings": null
 }
}
2021/12/08 23:08:20 Starting bucket ops to source bucket
2021/12/08 23:08:25 Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/08 23:08:30 Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/08 23:08:35 Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/08 23:08:35 App: TestEventingRebKVOpsOnUpdateBucketOpOneByOne got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:08:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:08:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 23:08:35 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/08 23:08:35 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/08 23:08:35 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/08 23:08:40 Rebalance progress: 16.66666666666667
2021/12/08 23:08:45 Rebalance progress: 16.66666666666667
2021/12/08 23:08:50 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:08:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:08:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 23:08:50 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/08 23:08:50 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/08 23:08:51 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/08 23:08:56 Rebalance progress: 16.66666666666667
2021/12/08 23:09:01 Rebalance progress: 16.66666666666667
2021/12/08 23:09:06 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:09:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:09:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 23:09:06 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/08 23:09:06 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/08 23:09:11 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/08 23:09:16 Rebalance progress: 12.5
2021/12/08 23:09:21 Rebalance progress: 12.5
2021/12/08 23:09:26 Rebalance progress: 12.5
2021/12/08 23:09:31 Rebalance progress: 70.83333333333334
2021/12/08 23:09:36 Rebalance progress: 70.83333333333334
2021/12/08 23:09:41 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:09:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:09:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 23:09:41 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/08 23:09:46 Rebalance progress: 12.5
2021/12/08 23:09:51 Rebalance progress: 12.5
2021/12/08 23:09:56 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:09:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:09:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 23:09:56 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/08 23:10:01 Rebalance progress: 12.5
2021/12/08 23:10:06 Rebalance progress: 12.5
2021/12/08 23:10:11 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:10:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:10:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 23:10:11 clusterSetup.go:253: Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/08 23:10:16 Rebalance progress: 12.5
2021/12/08 23:10:21 Rebalance progress: 12.5
2021/12/08 23:10:26 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:10:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:10:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 23:10:27 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 23:10:32 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/08 23:10:37 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/08 23:10:42 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/08 23:10:47 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/08 23:10:47 clusterSetup.go:405: App: TestEventingRebKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/08 23:10:47 eventingSetup.go:46: No eventing-con process running
2021/12/08 23:10:47 eventingSetup.go:418: Function: TestEventingRebKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpOneByOne deleting in the background"
}
2021/12/08 23:10:48 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:10:53 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 23:10:58 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12665
2021/12/08 23:10:58 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 23:11:00 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:11:05 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 23:11:10 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3759
2021/12/08 23:11:15 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3759
2021/12/08 23:11:15 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebKVOpsOnUpdateBucketOpOneByOne (210.02s)
=== RUN   TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce
2021/12/08 23:11:25 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get un-deployed
2021/12/08 23:11:25 clusterSetup.go:405: App: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce got un-deployed
2021/12/08 23:11:25 eventingSetup.go:46: No eventing-con process running
2021/12/08 23:11:25 eventingSetup.go:418: Function: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce not found"
 }
}
2021/12/08 23:11:27 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:11:32 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 23:11:32 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 23:11:34 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:11:39 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 23:11:39 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:11:44 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 23:11:44 eventingSetup.go:125: Deploying app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce. Item count src bucket: 0 dst bucket: 0
2021/12/08 23:11:45 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce' in metakv",
  "warnings": null
 }
}
2021/12/08 23:11:50 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/08 23:11:55 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get deployed
2021/12/08 23:12:00 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get deployed
2021/12/08 23:12:05 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get deployed
2021/12/08 23:12:05 clusterSetup.go:354: App: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:12:05 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:12:05 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 23:12:05 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/08 23:12:05 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/08 23:12:06 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/08 23:12:06 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/08 23:12:11 Rebalance progress: 12.5
2021/12/08 23:12:16 Rebalance progress: 12.5
2021/12/08 23:12:21 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:12:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:12:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 23:12:21 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002 127.0.0.1:19003] from the cluster
2021/12/08 23:12:26 Rebalance progress: 12.5
2021/12/08 23:12:31 Rebalance progress: 12.5
2021/12/08 23:12:36 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:12:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:12:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 23:12:36 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 23:12:41 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get un-deployed
2021/12/08 23:12:46 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get un-deployed
2021/12/08 23:12:51 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get un-deployed
2021/12/08 23:12:56 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get un-deployed
2021/12/08 23:12:56 clusterSetup.go:405: App: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce got un-deployed
2021/12/08 23:12:56 eventingSetup.go:46: No eventing-con process running
2021/12/08 23:12:56 eventingSetup.go:418: Function: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce deleting in the background"
}
2021/12/08 23:12:58 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:13:03 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 23:13:08 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4617
2021/12/08 23:13:08 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 23:13:10 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:13:15 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 23:13:15 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce (119.66s)
=== RUN   TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings
2021/12/08 23:13:25 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/08 23:13:25 clusterSetup.go:405: App: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings got un-deployed
2021/12/08 23:13:25 eventingSetup.go:46: No eventing-con process running
2021/12/08 23:13:25 eventingSetup.go:418: Function: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings not found"
 }
}
2021/12/08 23:13:27 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:13:32 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 23:13:32 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 23:13:33 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:13:38 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 23:13:38 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:13:43 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 23:13:43 eventingSetup.go:125: Deploying app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/08 23:13:45 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/08 23:13:50 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/08 23:13:55 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get deployed
2021/12/08 23:14:00 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get deployed
2021/12/08 23:14:05 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get deployed
2021/12/08 23:14:05 clusterSetup.go:354: App: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:14:05 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:14:05 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 23:14:05 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/08 23:14:05 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/08 23:14:10 Rebalance progress: 12.5
2021/12/08 23:14:15 Rebalance progress: 12.5
2021/12/08 23:14:20 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:14:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:14:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 23:14:20 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/08 23:14:20 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/08 23:14:25 Rebalance progress: 12.5
2021/12/08 23:14:30 Rebalance progress: 12.5
2021/12/08 23:14:35 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:14:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:14:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 23:14:36 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/08 23:14:36 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/08 23:14:41 Rebalance progress: 12.5
2021/12/08 23:14:46 Rebalance progress: 12.5
2021/12/08 23:14:51 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:14:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:14:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 23:14:51 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/08 23:14:56 Rebalance progress: 12.5
2021/12/08 23:15:01 Rebalance progress: 12.5
2021/12/08 23:15:06 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:15:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:15:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 23:15:06 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/08 23:15:11 Rebalance progress: 12.5
2021/12/08 23:15:16 Rebalance progress: 12.5
2021/12/08 23:15:21 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:15:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:15:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 23:15:21 clusterSetup.go:253: Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/08 23:15:26 Rebalance progress: 12.5
2021/12/08 23:15:31 Rebalance progress: 12.5
2021/12/08 23:15:36 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:15:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:15:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 23:15:36 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 23:15:41 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/08 23:15:47 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/08 23:15:52 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/08 23:15:57 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/08 23:15:57 clusterSetup.go:405: App: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings got un-deployed
2021/12/08 23:15:57 eventingSetup.go:46: No eventing-con process running
2021/12/08 23:15:57 eventingSetup.go:418: Function: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings deleting in the background"
}
2021/12/08 23:15:58 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:16:03 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 23:16:08 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 10673
2021/12/08 23:16:08 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 23:16:10 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:16:15 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 23:16:20 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1777
2021/12/08 23:16:25 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1777
2021/12/08 23:16:25 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings (190.19s)
=== RUN   TestEventingSwapRebOnUpdateBucketOp
2021/12/08 23:16:35 clusterSetup.go:397: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get un-deployed
2021/12/08 23:16:35 clusterSetup.go:405: App: TestEventingSwapRebOnUpdateBucketOp got un-deployed
2021/12/08 23:16:35 eventingSetup.go:46: No eventing-con process running
2021/12/08 23:16:35 eventingSetup.go:418: Function: TestEventingSwapRebOnUpdateBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingSwapRebOnUpdateBucketOp not found"
 }
}
2021/12/08 23:16:37 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:16:42 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 23:16:42 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 23:16:44 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:16:49 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 23:16:49 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:16:49 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 23:16:49 eventingSetup.go:125: Deploying app: TestEventingSwapRebOnUpdateBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/08 23:16:50 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingSwapRebOnUpdateBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/08 23:16:55 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/08 23:17:00 clusterSetup.go:346: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get deployed
2021/12/08 23:17:05 clusterSetup.go:346: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get deployed
2021/12/08 23:17:10 clusterSetup.go:346: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get deployed
2021/12/08 23:17:10 clusterSetup.go:354: App: TestEventingSwapRebOnUpdateBucketOp got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:17:10 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:17:10 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 23:17:10 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/08 23:17:10 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/08 23:17:15 Rebalance progress: 12.5
2021/12/08 23:17:20 Rebalance progress: 12.5
2021/12/08 23:17:25 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:17:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:17:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 23:17:25 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/08 23:17:25 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/08 23:17:30 Rebalance progress: 12.5
2021/12/08 23:17:35 Rebalance progress: 12.5
2021/12/08 23:17:40 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:17:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:17:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 23:17:40 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/08 23:17:45 Rebalance progress: 12.5
2021/12/08 23:17:50 Rebalance progress: 12.5
2021/12/08 23:17:55 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:17:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:17:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 23:17:55 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 23:18:00 clusterSetup.go:397: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get un-deployed
2021/12/08 23:18:05 clusterSetup.go:397: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get un-deployed
2021/12/08 23:18:10 clusterSetup.go:397: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get un-deployed
2021/12/08 23:18:15 clusterSetup.go:397: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get un-deployed
2021/12/08 23:18:15 clusterSetup.go:405: App: TestEventingSwapRebOnUpdateBucketOp got un-deployed
2021/12/08 23:18:15 eventingSetup.go:46: No eventing-con process running
2021/12/08 23:18:15 eventingSetup.go:418: Function: TestEventingSwapRebOnUpdateBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingSwapRebOnUpdateBucketOp deleting in the background"
}
2021/12/08 23:18:17 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:18:22 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 23:18:27 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6057
2021/12/08 23:18:27 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 23:18:29 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:18:34 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 23:18:39 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6057
2021/12/08 23:18:39 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingSwapRebOnUpdateBucketOp (133.92s)
=== RUN   TestMetaRollbackWithEventingReb
2021/12/08 23:18:44 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/08 23:18:44 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/08 23:18:49 Rebalance progress: 100
2021/12/08 23:18:54 Waiting for app: TestMetaRollbackWithEventingReb to get un-deployed
2021/12/08 23:18:54 App: TestMetaRollbackWithEventingReb got un-deployed
2021/12/08 23:18:54 No eventing-con process running
2021/12/08 23:18:54 Function: TestMetaRollbackWithEventingReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMetaRollbackWithEventingReb not found"
 }
}
2021/12/08 23:18:56 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:19:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 23:19:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 23:19:03 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:19:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 23:19:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:19:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 23:19:08 Deploying app: TestMetaRollbackWithEventingReb. Item count src bucket: 0 dst bucket: 0
2021/12/08 23:19:09 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMetaRollbackWithEventingReb' in metakv",
  "warnings": null
 }
}
2021/12/08 23:19:14 Starting bucket ops to source bucket
2021/12/08 23:19:19 Waiting for app: TestMetaRollbackWithEventingReb to get deployed
2021/12/08 23:19:24 Waiting for app: TestMetaRollbackWithEventingReb to get deployed
2021/12/08 23:19:29 Waiting for app: TestMetaRollbackWithEventingReb to get deployed
2021/12/08 23:19:29 App: TestMetaRollbackWithEventingReb got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:19:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:19:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 23:19:29 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/08 23:19:30 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/08 23:19:35 Rebalance progress: 12.5
2021/12/08 23:19:40 Rebalance progress: 12.5
2021/12/08 23:19:45 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:19:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:19:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 23:19:45 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002] from the cluster
2021/12/08 23:19:45 bucketOps.go:134: Metakv lookup failed, err: Get "/_metakv/eventing/tempApps/TestMetaRollbackWithEventingReb/0": Unable to initialize cbauth's revrpc: cbauth environment variable CBAUTH_REVRPC_URL is not set
2021/12/08 23:19:50 Rebalance progress: 12.5
2021/12/08 23:19:55 Rebalance progress: 12.5
2021/12/08 23:20:00 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:20:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:20:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 23:20:00 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 23:20:05 clusterSetup.go:397: Waiting for app: TestMetaRollbackWithEventingReb to get un-deployed
2021/12/08 23:20:10 clusterSetup.go:397: Waiting for app: TestMetaRollbackWithEventingReb to get un-deployed
2021/12/08 23:20:15 clusterSetup.go:397: Waiting for app: TestMetaRollbackWithEventingReb to get un-deployed
2021/12/08 23:20:20 clusterSetup.go:397: Waiting for app: TestMetaRollbackWithEventingReb to get un-deployed
2021/12/08 23:20:20 clusterSetup.go:405: App: TestMetaRollbackWithEventingReb got un-deployed
2021/12/08 23:20:20 eventingSetup.go:46: No eventing-con process running
2021/12/08 23:20:20 eventingSetup.go:418: Function: TestMetaRollbackWithEventingReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMetaRollbackWithEventingReb deleting in the background"
}
2021/12/08 23:20:22 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:20:27 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 23:20:32 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4549
2021/12/08 23:20:37 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4549
2021/12/08 23:20:37 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 23:20:38 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:20:43 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 23:20:48 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4549
2021/12/08 23:20:48 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestMetaRollbackWithEventingReb (129.48s)
=== RUN   TestMetaPartialRollbackWithEventingReb
2021/12/08 23:20:53 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/08 23:20:54 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/08 23:20:59 Rebalance progress: 100
2021/12/08 23:21:04 Waiting for app: TestMetaPartialRollbackWithEventingReb to get un-deployed
2021/12/08 23:21:04 App: TestMetaPartialRollbackWithEventingReb got un-deployed
2021/12/08 23:21:04 No eventing-con process running
2021/12/08 23:21:04 Function: TestMetaPartialRollbackWithEventingReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMetaPartialRollbackWithEventingReb not found"
 }
}
2021/12/08 23:21:06 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:21:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 23:21:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 23:21:12 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:21:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 23:21:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:21:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 23:21:17 Deploying app: TestMetaPartialRollbackWithEventingReb. Item count src bucket: 0 dst bucket: 0
2021/12/08 23:21:19 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMetaPartialRollbackWithEventingReb' in metakv",
  "warnings": null
 }
}
2021/12/08 23:21:24 Starting bucket ops to source bucket
2021/12/08 23:21:29 Waiting for app: TestMetaPartialRollbackWithEventingReb to get deployed
2021/12/08 23:21:34 Waiting for app: TestMetaPartialRollbackWithEventingReb to get deployed
2021/12/08 23:21:39 Waiting for app: TestMetaPartialRollbackWithEventingReb to get deployed
2021/12/08 23:21:39 App: TestMetaPartialRollbackWithEventingReb got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:21:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:21:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 23:21:39 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/08 23:21:39 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/08 23:21:44 Rebalance progress: 12.5
2021/12/08 23:21:49 Rebalance progress: 12.5
2021/12/08 23:21:54 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:21:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:21:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 23:21:54 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002] from the cluster
2021/12/08 23:21:54 bucketOps.go:65: Metakv lookup failed, err: Get "/_metakv/eventing/tempApps/TestMetaPartialRollbackWithEventingReb/0": Unable to initialize cbauth's revrpc: cbauth environment variable CBAUTH_REVRPC_URL is not set
2021/12/08 23:21:59 Rebalance progress: 12.5
2021/12/08 23:22:04 Rebalance progress: 12.5
2021/12/08 23:22:09 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:22:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:22:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 23:22:09 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 23:22:09 bucketOps.go:65: Metakv lookup failed, err: Get "/_metakv/eventing/tempApps/TestMetaPartialRollbackWithEventingReb/0": Unable to initialize cbauth's revrpc: cbauth environment variable CBAUTH_REVRPC_URL is not set
2021/12/08 23:22:14 clusterSetup.go:397: Waiting for app: TestMetaPartialRollbackWithEventingReb to get un-deployed
2021/12/08 23:22:19 clusterSetup.go:397: Waiting for app: TestMetaPartialRollbackWithEventingReb to get un-deployed
2021/12/08 23:22:24 clusterSetup.go:397: Waiting for app: TestMetaPartialRollbackWithEventingReb to get un-deployed
2021/12/08 23:22:29 clusterSetup.go:397: Waiting for app: TestMetaPartialRollbackWithEventingReb to get un-deployed
2021/12/08 23:22:29 clusterSetup.go:405: App: TestMetaPartialRollbackWithEventingReb got un-deployed
2021/12/08 23:22:29 eventingSetup.go:46: No eventing-con process running
2021/12/08 23:22:29 eventingSetup.go:418: Function: TestMetaPartialRollbackWithEventingReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMetaPartialRollbackWithEventingReb deleting in the background"
}
2021/12/08 23:22:31 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:22:36 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 23:22:41 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 37782
2021/12/08 23:22:46 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 37782
2021/12/08 23:22:46 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 23:22:48 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:22:53 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 23:22:58 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 37782
2021/12/08 23:22:58 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestMetaPartialRollbackWithEventingReb (129.49s)
=== RUN   TestEventingRebMultipleHandlersOneByOne
2021/12/08 23:23:08 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get un-deployed
2021/12/08 23:23:08 clusterSetup.go:405: App: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op got un-deployed
2021/12/08 23:23:08 eventingSetup.go:46: No eventing-con process running
2021/12/08 23:23:08 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op not found"
 }
}
2021/12/08 23:23:10 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:23:15 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 23:23:15 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 23:23:16 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:23:21 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 23:23:21 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:23:26 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 23:23:26 eventingSetup.go:125: Deploying app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op. Item count src bucket: 0 dst bucket: 0
2021/12/08 23:23:28 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op' in metakv",
  "warnings": null
 }
}
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:23:28 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 23:23:28 eventingSetup.go:125: Deploying app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/08 23:23:29 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebMultipleHandlersOneByOnebucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/08 23:23:34 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/08 23:23:39 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get deployed
2021/12/08 23:23:44 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get deployed
2021/12/08 23:23:49 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get deployed
2021/12/08 23:23:49 clusterSetup.go:354: App: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op got deployed
2021/12/08 23:23:54 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get deployed
2021/12/08 23:23:59 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get deployed
2021/12/08 23:24:04 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get deployed
2021/12/08 23:24:04 clusterSetup.go:354: App: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:24:04 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:24:04 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 23:24:04 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/08 23:24:04 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/08 23:24:09 Rebalance progress: 12.5
2021/12/08 23:24:14 Rebalance progress: 12.5
2021/12/08 23:24:19 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:24:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:24:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 23:24:19 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/08 23:24:19 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/08 23:24:25 Rebalance progress: 12.5
2021/12/08 23:24:30 Rebalance progress: 12.5
2021/12/08 23:24:35 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:24:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:24:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 23:24:35 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/08 23:24:35 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/08 23:24:40 Rebalance progress: 12.5
2021/12/08 23:24:45 Rebalance progress: 12.5
2021/12/08 23:24:50 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:24:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:24:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 23:24:50 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/08 23:24:55 Rebalance progress: 12.5
2021/12/08 23:25:00 Rebalance progress: 12.5
2021/12/08 23:25:05 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:25:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:25:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 23:25:05 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/08 23:25:10 Rebalance progress: 12.5
2021/12/08 23:25:15 Rebalance progress: 12.5
2021/12/08 23:25:20 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:25:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:25:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 23:25:20 clusterSetup.go:253: Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/08 23:25:26 Rebalance progress: 12.5
2021/12/08 23:25:31 Rebalance progress: 12.5
2021/12/08 23:25:36 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:25:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:25:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 23:25:36 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 23:25:41 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get un-deployed
2021/12/08 23:25:46 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get un-deployed
2021/12/08 23:25:51 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get un-deployed
2021/12/08 23:25:56 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get un-deployed
2021/12/08 23:25:56 clusterSetup.go:405: App: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op got un-deployed
2021/12/08 23:25:56 eventingSetup.go:46: No eventing-con process running
2021/12/08 23:25:56 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op deleting in the background"
}
2021/12/08 23:25:59 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:26:04 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 23:26:04 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 23:26:06 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:26:11 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 23:26:16 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12168
2021/12/08 23:26:16 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 23:26:16 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 23:26:21 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get un-deployed
2021/12/08 23:26:26 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get un-deployed
2021/12/08 23:26:31 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get un-deployed
2021/12/08 23:26:36 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get un-deployed
2021/12/08 23:26:36 clusterSetup.go:405: App: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update got un-deployed
2021/12/08 23:26:37 eventingSetup.go:46: No eventing-con process running
2021/12/08 23:26:37 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update deleting in the background"
}
2021/12/08 23:26:38 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:26:43 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 23:26:43 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 23:26:45 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:26:50 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 23:26:50 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebMultipleHandlersOneByOne (232.06s)
=== RUN   TestEventingRebMultipleHandlersAllAtOnce
2021/12/08 23:27:00 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get un-deployed
2021/12/08 23:27:00 clusterSetup.go:405: App: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op got un-deployed
2021/12/08 23:27:00 eventingSetup.go:46: No eventing-con process running
2021/12/08 23:27:00 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op not found"
 }
}
2021/12/08 23:27:02 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:27:07 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 23:27:07 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 23:27:09 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:27:14 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 23:27:14 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:27:19 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 23:27:19 eventingSetup.go:125: Deploying app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op. Item count src bucket: 0 dst bucket: 0
2021/12/08 23:27:20 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op' in metakv",
  "warnings": null
 }
}
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:27:20 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 23:27:20 eventingSetup.go:125: Deploying app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/08 23:27:21 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/08 23:27:26 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/08 23:27:31 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get deployed
2021/12/08 23:27:36 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get deployed
2021/12/08 23:27:41 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get deployed
2021/12/08 23:27:41 clusterSetup.go:354: App: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op got deployed
2021/12/08 23:27:46 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get deployed
2021/12/08 23:27:51 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get deployed
2021/12/08 23:27:56 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get deployed
2021/12/08 23:27:56 clusterSetup.go:354: App: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:27:56 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:27:56 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 23:27:56 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/08 23:27:56 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/08 23:27:57 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/08 23:27:57 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/08 23:28:02 Rebalance progress: 12.5
2021/12/08 23:28:07 Rebalance progress: 12.5
2021/12/08 23:28:12 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:28:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:28:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 23:28:12 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002 127.0.0.1:19003] from the cluster
2021/12/08 23:28:17 Rebalance progress: 12.5
2021/12/08 23:28:22 Rebalance progress: 12.5
2021/12/08 23:28:27 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:28:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:28:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 23:28:27 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 23:28:32 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get un-deployed
2021/12/08 23:28:37 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get un-deployed
2021/12/08 23:28:42 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get un-deployed
2021/12/08 23:28:47 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get un-deployed
2021/12/08 23:28:47 clusterSetup.go:405: App: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op got un-deployed
2021/12/08 23:28:47 eventingSetup.go:46: No eventing-con process running
2021/12/08 23:28:47 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op deleting in the background"
}
2021/12/08 23:28:51 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:28:56 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 23:29:01 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 242
2021/12/08 23:29:06 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 242
2021/12/08 23:29:06 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 23:29:07 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:29:12 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 23:29:17 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6112
2021/12/08 23:29:17 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 23:29:18 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 23:29:23 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get un-deployed
2021/12/08 23:29:28 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get un-deployed
2021/12/08 23:29:33 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get un-deployed
2021/12/08 23:29:38 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get un-deployed
2021/12/08 23:29:38 clusterSetup.go:405: App: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update got un-deployed
2021/12/08 23:29:38 eventingSetup.go:46: No eventing-con process running
2021/12/08 23:29:38 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update deleting in the background"
}
2021/12/08 23:29:39 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:29:44 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 23:29:44 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 23:29:46 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:29:51 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 23:29:51 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebMultipleHandlersAllAtOnce (181.12s)
=== RUN   TestEventingSwapRebMultipleHandlers
2021/12/08 23:30:01 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get un-deployed
2021/12/08 23:30:01 clusterSetup.go:405: App: TestEventingSwapRebMultipleHandlersbucket_op_on_update got un-deployed
2021/12/08 23:30:01 eventingSetup.go:46: No eventing-con process running
2021/12/08 23:30:01 eventingSetup.go:418: Function: TestEventingSwapRebMultipleHandlersbucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingSwapRebMultipleHandlersbucket_op_on_update not found"
 }
}
2021/12/08 23:30:03 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:30:08 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 23:30:08 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 23:30:10 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:30:15 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 23:30:15 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 23:30:20 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get un-deployed
2021/12/08 23:30:20 clusterSetup.go:405: App: TestEventingSwapRebMultipleHandlerssys_test_bucket_op got un-deployed
2021/12/08 23:30:20 eventingSetup.go:46: No eventing-con process running
2021/12/08 23:30:20 eventingSetup.go:418: Function: TestEventingSwapRebMultipleHandlerssys_test_bucket_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingSwapRebMultipleHandlerssys_test_bucket_op not found"
 }
}
2021/12/08 23:30:21 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:30:26 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 23:30:26 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 23:30:28 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:30:33 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 23:30:33 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:30:33 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 23:30:33 eventingSetup.go:125: Deploying app: TestEventingSwapRebMultipleHandlersbucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/08 23:30:34 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingSwapRebMultipleHandlersbucket_op_on_update' in metakv",
  "warnings": null
 }
}
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:30:34 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 23:30:34 eventingSetup.go:125: Deploying app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op. Item count src bucket: 0 dst bucket: 0
2021/12/08 23:30:35 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingSwapRebMultipleHandlerssys_test_bucket_op' in metakv",
  "warnings": null
 }
}
2021/12/08 23:30:40 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/08 23:30:45 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get deployed
2021/12/08 23:30:50 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get deployed
2021/12/08 23:30:55 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get deployed
2021/12/08 23:30:55 clusterSetup.go:354: App: TestEventingSwapRebMultipleHandlersbucket_op_on_update got deployed
2021/12/08 23:31:00 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get deployed
2021/12/08 23:31:06 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get deployed
2021/12/08 23:31:11 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get deployed
2021/12/08 23:31:11 clusterSetup.go:354: App: TestEventingSwapRebMultipleHandlerssys_test_bucket_op got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:31:11 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:31:11 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 23:31:11 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/08 23:31:11 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/08 23:31:16 Rebalance progress: 12.5
2021/12/08 23:31:21 Rebalance progress: 12.5
2021/12/08 23:31:26 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:31:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:31:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 23:31:26 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/08 23:31:26 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/08 23:31:31 Rebalance progress: 12.5
2021/12/08 23:31:36 Rebalance progress: 12.5
2021/12/08 23:31:41 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:31:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:31:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 23:31:41 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/08 23:31:46 Rebalance progress: 12.5
2021/12/08 23:31:51 Rebalance progress: 12.5
2021/12/08 23:31:56 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:31:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:31:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 23:31:56 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 23:32:01 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get un-deployed
2021/12/08 23:32:06 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get un-deployed
2021/12/08 23:32:11 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get un-deployed
2021/12/08 23:32:16 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get un-deployed
2021/12/08 23:32:16 clusterSetup.go:405: App: TestEventingSwapRebMultipleHandlersbucket_op_on_update got un-deployed
2021/12/08 23:32:16 eventingSetup.go:46: No eventing-con process running
2021/12/08 23:32:16 eventingSetup.go:418: Function: TestEventingSwapRebMultipleHandlersbucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingSwapRebMultipleHandlersbucket_op_on_update deleting in the background"
}
2021/12/08 23:32:19 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:32:24 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 23:32:29 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7576
2021/12/08 23:32:29 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 23:32:31 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:32:36 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 23:32:41 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7576
2021/12/08 23:32:46 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7576
2021/12/08 23:32:46 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 23:32:46 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 23:32:51 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get un-deployed
2021/12/08 23:32:56 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get un-deployed
2021/12/08 23:33:01 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get un-deployed
2021/12/08 23:33:06 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get un-deployed
2021/12/08 23:33:06 clusterSetup.go:405: App: TestEventingSwapRebMultipleHandlerssys_test_bucket_op got un-deployed
2021/12/08 23:33:06 eventingSetup.go:46: No eventing-con process running
2021/12/08 23:33:06 eventingSetup.go:418: Function: TestEventingSwapRebMultipleHandlerssys_test_bucket_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingSwapRebMultipleHandlerssys_test_bucket_op deleting in the background"
}
2021/12/08 23:33:08 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:33:13 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 23:33:13 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 23:33:15 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:33:20 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 23:33:20 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingSwapRebMultipleHandlers (208.71s)
=== RUN   TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne
2021/12/08 23:33:30 clusterSetup.go:397: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/08 23:33:30 clusterSetup.go:405: App: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/08 23:33:30 eventingSetup.go:46: No eventing-con process running
2021/12/08 23:33:30 eventingSetup.go:418: Function: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne not found"
 }
}
2021/12/08 23:33:32 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:33:37 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 23:33:37 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 23:33:38 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:33:43 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 23:33:43 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:33:48 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 23:33:48 eventingSetup.go:125: Deploying app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne. Item count src bucket: 0 dst bucket: 0
2021/12/08 23:33:50 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne' in metakv",
  "warnings": null
 }
}
2021/12/08 23:33:55 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/08 23:34:00 clusterSetup.go:346: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/08 23:34:05 clusterSetup.go:346: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/08 23:34:10 clusterSetup.go:346: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/08 23:34:10 clusterSetup.go:354: App: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:34:10 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:34:10 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 23:34:10 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/08 23:34:10 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/08 23:34:30 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:34:31 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:34:31 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:35:01 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:35:01 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 23:35:01 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/08 23:35:06 Rebalance progress: 12.5
2021/12/08 23:35:11 Rebalance progress: 12.5
2021/12/08 23:35:16 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:35:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:35:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 23:35:16 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/08 23:35:16 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/08 23:35:21 Rebalance progress: 12.5
2021/12/08 23:35:26 Rebalance progress: 12.5
2021/12/08 23:35:31 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:35:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:35:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 23:35:31 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/08 23:35:37 Rebalance progress: 12.5
2021/12/08 23:35:42 Rebalance progress: 12.5
2021/12/08 23:35:47 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:35:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:35:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 23:35:47 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 23:35:52 clusterSetup.go:397: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/08 23:35:57 clusterSetup.go:397: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/08 23:36:02 clusterSetup.go:397: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/08 23:36:07 clusterSetup.go:397: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/08 23:36:07 clusterSetup.go:405: App: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/08 23:36:07 eventingSetup.go:46: No eventing-con process running
2021/12/08 23:36:07 eventingSetup.go:418: Function: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne deleting in the background"
}
2021/12/08 23:36:08 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:36:13 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 23:36:18 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 11194
2021/12/08 23:36:18 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 23:36:20 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:36:25 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 23:36:30 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3947
2021/12/08 23:36:35 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3947
2021/12/08 23:36:35 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne (195.39s)
=== RUN   TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne
2021/12/08 23:36:45 clusterSetup.go:397: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/08 23:36:45 clusterSetup.go:405: App: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/08 23:36:45 eventingSetup.go:46: No eventing-con process running
2021/12/08 23:36:45 eventingSetup.go:418: Function: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne not found"
 }
}
2021/12/08 23:36:47 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:36:52 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 23:36:52 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 23:36:54 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:36:59 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 23:36:59 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:37:04 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 23:37:04 eventingSetup.go:125: Deploying app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne. Item count src bucket: 0 dst bucket: 0
2021/12/08 23:37:05 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne' in metakv",
  "warnings": null
 }
}
2021/12/08 23:37:10 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/08 23:37:15 clusterSetup.go:346: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/08 23:37:20 clusterSetup.go:346: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/08 23:37:25 clusterSetup.go:346: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/08 23:37:25 clusterSetup.go:354: App: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:37:25 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:37:25 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 23:37:25 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/08 23:37:25 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/08 23:37:45 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:37:46 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:37:46 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:37:56 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:37:56 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 23:37:56 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/08 23:37:56 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/08 23:38:16 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:38:17 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:38:17 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:38:27 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:38:27 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 23:38:27 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/08 23:38:27 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/08 23:38:47 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:38:47 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:38:47 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:38:57 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:38:57 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 23:38:57 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/08 23:38:58 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/08 23:39:18 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:39:18 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:39:18 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:39:28 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:39:28 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 23:39:28 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/08 23:39:29 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/08 23:39:49 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:39:49 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:39:49 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:39:59 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:39:59 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 23:39:59 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/08 23:40:04 Rebalance progress: 12.5
2021/12/08 23:40:09 Rebalance progress: 12.5
2021/12/08 23:40:14 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:40:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:40:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 23:40:14 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/08 23:40:14 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/08 23:40:19 Rebalance progress: 12.5
2021/12/08 23:40:24 Rebalance progress: 12.5
2021/12/08 23:40:29 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:40:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:40:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 23:40:29 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/08 23:40:35 Rebalance progress: 12.5
2021/12/08 23:40:40 Rebalance progress: 12.5
2021/12/08 23:40:45 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:40:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:40:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 23:40:45 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 23:40:50 clusterSetup.go:397: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/08 23:40:55 clusterSetup.go:397: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/08 23:41:00 clusterSetup.go:397: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/08 23:41:05 clusterSetup.go:397: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/08 23:41:05 clusterSetup.go:405: App: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/08 23:41:05 eventingSetup.go:46: No eventing-con process running
2021/12/08 23:41:05 eventingSetup.go:418: Function: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne deleting in the background"
}
2021/12/08 23:41:06 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:41:11 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 23:41:17 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 21464
2021/12/08 23:41:17 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 23:41:18 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:41:23 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 23:41:28 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 21464
2021/12/08 23:41:28 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne (292.98s)
=== RUN   TestEventingFailoverOnUpdateBucketOp
2021/12/08 23:41:38 clusterSetup.go:397: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get un-deployed
2021/12/08 23:41:38 clusterSetup.go:405: App: TestEventingFailoverOnUpdateBucketOp got un-deployed
2021/12/08 23:41:38 eventingSetup.go:46: No eventing-con process running
2021/12/08 23:41:38 eventingSetup.go:418: Function: TestEventingFailoverOnUpdateBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingFailoverOnUpdateBucketOp not found"
 }
}
2021/12/08 23:41:40 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:41:45 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 23:41:45 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 23:41:47 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:41:52 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 23:41:52 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:41:57 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 23:41:57 eventingSetup.go:125: Deploying app: TestEventingFailoverOnUpdateBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/08 23:41:58 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingFailoverOnUpdateBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/08 23:42:03 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/08 23:42:08 clusterSetup.go:346: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get deployed
2021/12/08 23:42:13 clusterSetup.go:346: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get deployed
2021/12/08 23:42:18 clusterSetup.go:346: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get deployed
2021/12/08 23:42:18 clusterSetup.go:354: App: TestEventingFailoverOnUpdateBucketOp got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:42:18 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:42:18 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 23:42:18 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/08 23:42:18 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/08 23:42:23 Rebalance progress: 12.5
2021/12/08 23:42:28 Rebalance progress: 12.5
2021/12/08 23:42:33 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:42:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:42:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 23:42:33 clusterSetup.go:229: Failing over: [127.0.0.1:19001]
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:42:43 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:42:43 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 23:42:43 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/08 23:42:49 Rebalance progress: 12.5
2021/12/08 23:42:54 Rebalance progress: 12.5
2021/12/08 23:42:59 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:42:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:42:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 23:42:59 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/08 23:42:59 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/08 23:43:04 Rebalance progress: 12.5
2021/12/08 23:43:09 Rebalance progress: 12.5
2021/12/08 23:43:14 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:43:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:43:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 23:43:14 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/08 23:43:19 Rebalance progress: 12.5
2021/12/08 23:43:24 Rebalance progress: 12.5
2021/12/08 23:43:29 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:43:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:43:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 23:43:29 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 23:43:34 clusterSetup.go:397: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get un-deployed
2021/12/08 23:43:39 clusterSetup.go:397: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get un-deployed
2021/12/08 23:43:44 clusterSetup.go:397: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get un-deployed
2021/12/08 23:43:49 clusterSetup.go:397: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get un-deployed
2021/12/08 23:43:49 clusterSetup.go:405: App: TestEventingFailoverOnUpdateBucketOp got un-deployed
2021/12/08 23:43:49 eventingSetup.go:46: No eventing-con process running
2021/12/08 23:43:49 eventingSetup.go:418: Function: TestEventingFailoverOnUpdateBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingFailoverOnUpdateBucketOp deleting in the background"
}
2021/12/08 23:43:51 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:43:56 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 23:44:01 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 8577
2021/12/08 23:44:06 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 8577
2021/12/08 23:44:06 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 23:44:08 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:44:13 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 23:44:18 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 8607
2021/12/08 23:44:18 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingFailoverOnUpdateBucketOp (169.51s)
=== RUN   TestEventingKVRebalanceOnUpdateBucketOp
2021/12/08 23:44:28 clusterSetup.go:397: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get un-deployed
2021/12/08 23:44:28 clusterSetup.go:405: App: TestEventingKVRebalanceOnUpdateBucketOp got un-deployed
2021/12/08 23:44:28 eventingSetup.go:46: No eventing-con process running
2021/12/08 23:44:28 eventingSetup.go:418: Function: TestEventingKVRebalanceOnUpdateBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingKVRebalanceOnUpdateBucketOp not found"
 }
}
2021/12/08 23:44:30 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:44:35 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 23:44:35 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 23:44:36 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:44:41 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 23:44:41 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:44:46 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 23:44:46 eventingSetup.go:125: Deploying app: TestEventingKVRebalanceOnUpdateBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/08 23:44:47 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingKVRebalanceOnUpdateBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/08 23:44:52 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/08 23:44:57 clusterSetup.go:346: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get deployed
2021/12/08 23:45:02 clusterSetup.go:346: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get deployed
2021/12/08 23:45:07 clusterSetup.go:346: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get deployed
2021/12/08 23:45:07 clusterSetup.go:354: App: TestEventingKVRebalanceOnUpdateBucketOp got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:45:07 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:45:07 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 23:45:07 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing,kv to the cluster
2021/12/08 23:45:08 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/08 23:45:13 Rebalance progress: 12.5
2021/12/08 23:45:18 Rebalance progress: 12.5
2021/12/08 23:45:23 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:45:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:45:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 23:45:23 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing,kv to the cluster
2021/12/08 23:45:23 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/08 23:45:28 Rebalance progress: 12.5
2021/12/08 23:45:33 Rebalance progress: 12.5
2021/12/08 23:45:38 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:45:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:45:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 23:45:38 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/08 23:45:43 Rebalance progress: 12.5
2021/12/08 23:45:48 Rebalance progress: 12.5
2021/12/08 23:45:53 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:45:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:45:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 23:45:53 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 23:45:58 clusterSetup.go:397: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get un-deployed
2021/12/08 23:46:04 clusterSetup.go:397: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get un-deployed
2021/12/08 23:46:09 clusterSetup.go:397: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get un-deployed
2021/12/08 23:46:14 clusterSetup.go:397: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get un-deployed
2021/12/08 23:46:14 clusterSetup.go:405: App: TestEventingKVRebalanceOnUpdateBucketOp got un-deployed
2021/12/08 23:46:14 eventingSetup.go:46: No eventing-con process running
2021/12/08 23:46:14 eventingSetup.go:418: Function: TestEventingKVRebalanceOnUpdateBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingKVRebalanceOnUpdateBucketOp deleting in the background"
}
2021/12/08 23:46:15 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:46:20 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 23:46:25 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6083
2021/12/08 23:46:25 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 23:46:27 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:46:32 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 23:46:37 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6083
2021/12/08 23:46:37 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingKVRebalanceOnUpdateBucketOp (139.32s)
=== RUN   TestEventingRebInWithOps
2021/12/08 23:46:47 clusterSetup.go:397: Waiting for app: TestEventingRebInWithOps to get un-deployed
2021/12/08 23:46:47 clusterSetup.go:405: App: TestEventingRebInWithOps got un-deployed
2021/12/08 23:46:47 eventingSetup.go:46: No eventing-con process running
2021/12/08 23:46:47 eventingSetup.go:418: Function: TestEventingRebInWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebInWithOps not found"
 }
}
2021/12/08 23:46:49 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:46:54 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 23:46:54 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 23:46:56 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:47:01 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 23:47:01 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:47:06 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 23:47:06 eventingSetup.go:125: Deploying app: TestEventingRebInWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/08 23:47:07 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebInWithOps' in metakv",
  "warnings": null
 }
}
2021/12/08 23:47:12 clusterSetup.go:346: Waiting for app: TestEventingRebInWithOps to get deployed
2021/12/08 23:47:17 clusterSetup.go:346: Waiting for app: TestEventingRebInWithOps to get deployed
2021/12/08 23:47:22 clusterSetup.go:346: Waiting for app: TestEventingRebInWithOps to get deployed
2021/12/08 23:47:27 clusterSetup.go:346: Waiting for app: TestEventingRebInWithOps to get deployed
2021/12/08 23:47:27 clusterSetup.go:354: App: TestEventingRebInWithOps got deployed
2021/12/08 23:47:27 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/08 23:47:27 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/08 23:47:27 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/08 23:47:33 Rebalance progress: 12.5
2021/12/08 23:47:38 Rebalance progress: 12.5
2021/12/08 23:47:43 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:47:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:47:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 23:47:43 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/08 23:47:48 Rebalance progress: 12.5
2021/12/08 23:47:53 Rebalance progress: 12.5
2021/12/08 23:47:58 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:47:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:47:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 23:47:58 Waiting for src bucket item count to get to: 200000 curr count: 34357
2021/12/08 23:48:03 Waiting for src bucket item count to get to: 200000 curr count: 50655
2021/12/08 23:48:08 Waiting for src bucket item count to get to: 200000 curr count: 50655
2021/12/08 23:48:13 Waiting for src bucket item count to get to: 200000 curr count: 66964
2021/12/08 23:48:18 Waiting for src bucket item count to get to: 200000 curr count: 66964
2021/12/08 23:48:23 Waiting for src bucket item count to get to: 200000 curr count: 82325
2021/12/08 23:48:28 Waiting for src bucket item count to get to: 200000 curr count: 82325
2021/12/08 23:48:33 Waiting for src bucket item count to get to: 200000 curr count: 98481
2021/12/08 23:48:38 Waiting for src bucket item count to get to: 200000 curr count: 98481
2021/12/08 23:48:43 Waiting for src bucket item count to get to: 200000 curr count: 115138
2021/12/08 23:48:48 Waiting for src bucket item count to get to: 200000 curr count: 115138
2021/12/08 23:48:53 Waiting for src bucket item count to get to: 200000 curr count: 131684
2021/12/08 23:48:58 Waiting for src bucket item count to get to: 200000 curr count: 131684
2021/12/08 23:49:03 Waiting for src bucket item count to get to: 200000 curr count: 148267
2021/12/08 23:49:08 Waiting for src bucket item count to get to: 200000 curr count: 148267
2021/12/08 23:49:13 Waiting for src bucket item count to get to: 200000 curr count: 148267
2021/12/08 23:49:18 Waiting for src bucket item count to get to: 200000 curr count: 164705
2021/12/08 23:49:23 Waiting for src bucket item count to get to: 200000 curr count: 164705
2021/12/08 23:49:28 Waiting for src bucket item count to get to: 200000 curr count: 180035
2021/12/08 23:49:33 Waiting for src bucket item count to get to: 200000 curr count: 180035
2021/12/08 23:49:38 Waiting for src bucket item count to get to: 200000 curr count: 196188
2021/12/08 23:49:43 Waiting for src bucket item count to get to: 200000 curr count: 196188
2021/12/08 23:49:48 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/08 23:49:48 Post SET ops. Expected item count: 200000 got 200000
2021/12/08 23:49:48 Starting bucket ops to source bucket
2021/12/08 23:53:33 Waiting for dst bucket item count to get to: 0 curr count: 16565
2021/12/08 23:53:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 23:53:33 Post DELETE ops. Expected item count: 0 got 0
2021/12/08 23:53:33 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 23:53:38 Waiting for app: TestEventingRebInWithOps to get un-deployed
2021/12/08 23:53:43 Waiting for app: TestEventingRebInWithOps to get un-deployed
2021/12/08 23:53:48 Waiting for app: TestEventingRebInWithOps to get un-deployed
2021/12/08 23:53:53 Waiting for app: TestEventingRebInWithOps to get un-deployed
2021/12/08 23:53:53 App: TestEventingRebInWithOps got un-deployed
2021/12/08 23:53:53 No eventing-con process running
2021/12/08 23:53:54 Function: TestEventingRebInWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebInWithOps deleting in the background"
}
2021/12/08 23:53:55 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:54:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 23:54:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 23:54:02 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:54:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 23:54:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebInWithOps (449.90s)
=== RUN   TestEventingRebOutWithOps
2021/12/08 23:54:12 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/08 23:54:12 Removing node(s): [] from the cluster
2021/12/08 23:54:17 Rebalance progress: 100
2021/12/08 23:54:22 Waiting for app: TestEventingRebOutWithOps to get un-deployed
2021/12/08 23:54:22 App: TestEventingRebOutWithOps got un-deployed
2021/12/08 23:54:22 No eventing-con process running
2021/12/08 23:54:22 Function: TestEventingRebOutWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebOutWithOps not found"
 }
}
2021/12/08 23:54:24 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:54:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 23:54:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 23:54:31 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:54:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 23:54:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:54:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 23:54:41 Deploying app: TestEventingRebOutWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/08 23:54:42 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebOutWithOps' in metakv",
  "warnings": null
 }
}
2021/12/08 23:54:47 Waiting for app: TestEventingRebOutWithOps to get deployed
2021/12/08 23:54:52 Waiting for app: TestEventingRebOutWithOps to get deployed
2021/12/08 23:54:57 Waiting for app: TestEventingRebOutWithOps to get deployed
2021/12/08 23:55:02 Waiting for app: TestEventingRebOutWithOps to get deployed
2021/12/08 23:55:02 App: TestEventingRebOutWithOps got deployed
2021/12/08 23:55:02 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/08 23:55:02 Starting bucket ops to source bucket
2021/12/08 23:55:07 Rebalance progress: 12.5
2021/12/08 23:55:12 Rebalance progress: 12.5
2021/12/08 23:55:17 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:55:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 23:55:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 23:55:17 Waiting for src bucket item count to get to: 200000 curr count: 11083
2021/12/08 23:55:22 Waiting for src bucket item count to get to: 200000 curr count: 27421
2021/12/08 23:55:27 Waiting for src bucket item count to get to: 200000 curr count: 27421
2021/12/08 23:55:32 Waiting for src bucket item count to get to: 200000 curr count: 44042
2021/12/08 23:55:37 Waiting for src bucket item count to get to: 200000 curr count: 44042
2021/12/08 23:55:42 Waiting for src bucket item count to get to: 200000 curr count: 60488
2021/12/08 23:55:47 Waiting for src bucket item count to get to: 200000 curr count: 60488
2021/12/08 23:55:52 Waiting for src bucket item count to get to: 200000 curr count: 60488
2021/12/08 23:55:57 Waiting for src bucket item count to get to: 200000 curr count: 75772
2021/12/08 23:56:02 Waiting for src bucket item count to get to: 200000 curr count: 75772
2021/12/08 23:56:07 Waiting for src bucket item count to get to: 200000 curr count: 91601
2021/12/08 23:56:12 Waiting for src bucket item count to get to: 200000 curr count: 91601
2021/12/08 23:56:17 Waiting for src bucket item count to get to: 200000 curr count: 107973
2021/12/08 23:56:22 Waiting for src bucket item count to get to: 200000 curr count: 107973
2021/12/08 23:56:27 Waiting for src bucket item count to get to: 200000 curr count: 124627
2021/12/08 23:56:32 Waiting for src bucket item count to get to: 200000 curr count: 124627
2021/12/08 23:56:37 Waiting for src bucket item count to get to: 200000 curr count: 140916
2021/12/08 23:56:42 Waiting for src bucket item count to get to: 200000 curr count: 140916
2021/12/08 23:56:47 Waiting for src bucket item count to get to: 200000 curr count: 157374
2021/12/08 23:56:52 Waiting for src bucket item count to get to: 200000 curr count: 157374
2021/12/08 23:56:57 Waiting for src bucket item count to get to: 200000 curr count: 172728
2021/12/08 23:57:02 Waiting for src bucket item count to get to: 200000 curr count: 172728
2021/12/08 23:57:07 Waiting for src bucket item count to get to: 200000 curr count: 188856
2021/12/08 23:57:12 Waiting for src bucket item count to get to: 200000 curr count: 188856
2021/12/08 23:57:17 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/08 23:57:17 Post SET ops. Expected item count: 200000 got 200000
2021/12/08 23:57:17 Starting bucket ops to source bucket
2021/12/09 00:01:02 Waiting for dst bucket item count to get to: 0 curr count: 15066
2021/12/09 00:01:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 00:01:02 Post DELETE ops. Expected item count: 0 got 0
2021/12/09 00:01:03 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 00:01:08 Waiting for app: TestEventingRebOutWithOps to get un-deployed
2021/12/09 00:01:13 Waiting for app: TestEventingRebOutWithOps to get un-deployed
2021/12/09 00:01:18 Waiting for app: TestEventingRebOutWithOps to get un-deployed
2021/12/09 00:01:23 Waiting for app: TestEventingRebOutWithOps to get un-deployed
2021/12/09 00:01:23 App: TestEventingRebOutWithOps got un-deployed
2021/12/09 00:01:23 No eventing-con process running
2021/12/09 00:01:23 Function: TestEventingRebOutWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebOutWithOps deleting in the background"
}
2021/12/09 00:01:24 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 00:01:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 00:01:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 00:01:31 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 00:01:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 00:01:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebOutWithOps (449.14s)
=== RUN   TestEventingSwapRebWithOps
2021/12/09 00:01:46 Waiting for app: TestEventingSwapRebWithOps to get un-deployed
2021/12/09 00:01:46 App: TestEventingSwapRebWithOps got un-deployed
2021/12/09 00:01:46 No eventing-con process running
2021/12/09 00:01:46 Function: TestEventingSwapRebWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingSwapRebWithOps not found"
 }
}
2021/12/09 00:01:48 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 00:01:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 00:01:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 00:01:55 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 00:02:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 00:02:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 00:02:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 00:02:05 Deploying app: TestEventingSwapRebWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/09 00:02:06 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingSwapRebWithOps' in metakv",
  "warnings": null
 }
}
2021/12/09 00:02:11 Waiting for app: TestEventingSwapRebWithOps to get deployed
2021/12/09 00:02:16 Waiting for app: TestEventingSwapRebWithOps to get deployed
2021/12/09 00:02:21 Waiting for app: TestEventingSwapRebWithOps to get deployed
2021/12/09 00:02:26 Waiting for app: TestEventingSwapRebWithOps to get deployed
2021/12/09 00:02:26 App: TestEventingSwapRebWithOps got deployed
2021/12/09 00:02:26 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/09 00:02:26 Starting bucket ops to source bucket
2021/12/09 00:02:26 Removing node(s): [] from the cluster
2021/12/09 00:02:31 Rebalance progress: 12.5
2021/12/09 00:02:36 Rebalance progress: 12.5
2021/12/09 00:02:41 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 00:02:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 00:02:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 00:02:41 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/09 00:02:42 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/09 00:02:47 Rebalance progress: 12.5
2021/12/09 00:02:52 Rebalance progress: 12.5
2021/12/09 00:02:57 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 00:02:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 00:02:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 00:02:57 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/09 00:03:02 Rebalance progress: 12.5
2021/12/09 00:03:07 Rebalance progress: 12.5
2021/12/09 00:03:12 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 00:03:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 00:03:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 00:03:12 Waiting for src bucket item count to get to: 200000 curr count: 67949
2021/12/09 00:03:17 Waiting for src bucket item count to get to: 200000 curr count: 67949
2021/12/09 00:03:22 Waiting for src bucket item count to get to: 200000 curr count: 82972
2021/12/09 00:03:27 Waiting for src bucket item count to get to: 200000 curr count: 82972
2021/12/09 00:03:32 Waiting for src bucket item count to get to: 200000 curr count: 99191
2021/12/09 00:03:37 Waiting for src bucket item count to get to: 200000 curr count: 99191
2021/12/09 00:03:42 Waiting for src bucket item count to get to: 200000 curr count: 115831
2021/12/09 00:03:47 Waiting for src bucket item count to get to: 200000 curr count: 115831
2021/12/09 00:03:52 Waiting for src bucket item count to get to: 200000 curr count: 132330
2021/12/09 00:03:57 Waiting for src bucket item count to get to: 200000 curr count: 132330
2021/12/09 00:04:02 Waiting for src bucket item count to get to: 200000 curr count: 148829
2021/12/09 00:04:07 Waiting for src bucket item count to get to: 200000 curr count: 148829
2021/12/09 00:04:12 Waiting for src bucket item count to get to: 200000 curr count: 165420
2021/12/09 00:04:17 Waiting for src bucket item count to get to: 200000 curr count: 165420
2021/12/09 00:04:22 Waiting for src bucket item count to get to: 200000 curr count: 180925
2021/12/09 00:04:27 Waiting for src bucket item count to get to: 200000 curr count: 180925
2021/12/09 00:04:32 Waiting for src bucket item count to get to: 200000 curr count: 197171
2021/12/09 00:04:37 Waiting for src bucket item count to get to: 200000 curr count: 197171
2021/12/09 00:04:42 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/09 00:04:42 Post SET ops. Expected item count: 200000 got 200000
2021/12/09 00:04:42 Starting bucket ops to source bucket
2021/12/09 00:08:27 Waiting for dst bucket item count to get to: 0 curr count: 5014
2021/12/09 00:08:32 Waiting for dst bucket item count to get to: 0 curr count: 5014
2021/12/09 00:08:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 00:08:32 Post DELETE ops. Expected item count: 0 got 0
2021/12/09 00:08:32 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 00:08:37 Waiting for app: TestEventingSwapRebWithOps to get un-deployed
2021/12/09 00:08:42 Waiting for app: TestEventingSwapRebWithOps to get un-deployed
2021/12/09 00:08:47 Waiting for app: TestEventingSwapRebWithOps to get un-deployed
2021/12/09 00:08:52 Waiting for app: TestEventingSwapRebWithOps to get un-deployed
2021/12/09 00:08:52 App: TestEventingSwapRebWithOps got un-deployed
2021/12/09 00:08:52 No eventing-con process running
2021/12/09 00:08:52 Function: TestEventingSwapRebWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingSwapRebWithOps deleting in the background"
}
2021/12/09 00:08:54 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 00:08:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 00:08:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 00:09:01 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 00:09:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 00:09:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingSwapRebWithOps (449.60s)
=== RUN   TestKVRebInWithOps
2021/12/09 00:09:16 Waiting for app: TestKVRebInWithOps to get un-deployed
2021/12/09 00:09:16 App: TestKVRebInWithOps got un-deployed
2021/12/09 00:09:16 No eventing-con process running
2021/12/09 00:09:16 Function: TestKVRebInWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVRebInWithOps not found"
 }
}
2021/12/09 00:09:17 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 00:09:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 00:09:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 00:09:24 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 00:09:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 00:09:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 00:09:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 00:09:34 Deploying app: TestKVRebInWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/09 00:09:35 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVRebInWithOps' in metakv",
  "warnings": null
 }
}
2021/12/09 00:09:40 Waiting for app: TestKVRebInWithOps to get deployed
2021/12/09 00:09:45 Waiting for app: TestKVRebInWithOps to get deployed
2021/12/09 00:09:50 Waiting for app: TestKVRebInWithOps to get deployed
2021/12/09 00:09:55 Waiting for app: TestKVRebInWithOps to get deployed
2021/12/09 00:09:55 App: TestKVRebInWithOps got deployed
2021/12/09 00:09:55 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/09 00:09:55 Starting bucket ops to source bucket
2021/12/09 00:09:56 Removing node(s): [] from the cluster
2021/12/09 00:10:01 Rebalance progress: 12.5
2021/12/09 00:10:06 Rebalance progress: 12.5
2021/12/09 00:10:11 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 00:10:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 00:10:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 00:10:11 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/09 00:10:16 Rebalance progress: 12.5
2021/12/09 00:10:21 Rebalance progress: 12.5
2021/12/09 00:10:26 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 00:10:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 00:10:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 00:10:26 Waiting for src bucket item count to get to: 200000 curr count: 37368
2021/12/09 00:10:31 Waiting for src bucket item count to get to: 200000 curr count: 53445
2021/12/09 00:10:36 Waiting for src bucket item count to get to: 200000 curr count: 53445
2021/12/09 00:10:41 Waiting for src bucket item count to get to: 200000 curr count: 70185
2021/12/09 00:10:46 Waiting for src bucket item count to get to: 200000 curr count: 70185
2021/12/09 00:10:51 Waiting for src bucket item count to get to: 200000 curr count: 85729
2021/12/09 00:10:56 Waiting for src bucket item count to get to: 200000 curr count: 85729
2021/12/09 00:11:01 Waiting for src bucket item count to get to: 200000 curr count: 101891
2021/12/09 00:11:06 Waiting for src bucket item count to get to: 200000 curr count: 101891
2021/12/09 00:11:11 Waiting for src bucket item count to get to: 200000 curr count: 101891
2021/12/09 00:11:16 Waiting for src bucket item count to get to: 200000 curr count: 118523
2021/12/09 00:11:21 Waiting for src bucket item count to get to: 200000 curr count: 118523
2021/12/09 00:11:26 Waiting for src bucket item count to get to: 200000 curr count: 135110
2021/12/09 00:11:31 Waiting for src bucket item count to get to: 200000 curr count: 135110
2021/12/09 00:11:36 Waiting for src bucket item count to get to: 200000 curr count: 151720
2021/12/09 00:11:41 Waiting for src bucket item count to get to: 200000 curr count: 151720
2021/12/09 00:11:46 Waiting for src bucket item count to get to: 200000 curr count: 168055
2021/12/09 00:11:51 Waiting for src bucket item count to get to: 200000 curr count: 168055
2021/12/09 00:11:56 Waiting for src bucket item count to get to: 200000 curr count: 183415
2021/12/09 00:12:01 Waiting for src bucket item count to get to: 200000 curr count: 183415
2021/12/09 00:12:06 Waiting for src bucket item count to get to: 200000 curr count: 199855
2021/12/09 00:12:11 Waiting for src bucket item count to get to: 200000 curr count: 199855
2021/12/09 00:12:16 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/09 00:12:16 Post SET ops. Expected item count: 200000 got 200000
2021/12/09 00:12:16 Starting bucket ops to source bucket
2021/12/09 00:16:00 Waiting for dst bucket item count to get to: 0 curr count: 10336
2021/12/09 00:16:05 Waiting for dst bucket item count to get to: 0 curr count: 10336
2021/12/09 00:16:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 00:16:05 Post DELETE ops. Expected item count: 0 got 0
2021/12/09 00:16:05 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 00:16:10 Waiting for app: TestKVRebInWithOps to get un-deployed
2021/12/09 00:16:15 Waiting for app: TestKVRebInWithOps to get un-deployed
2021/12/09 00:16:20 Waiting for app: TestKVRebInWithOps to get un-deployed
2021/12/09 00:16:25 Waiting for app: TestKVRebInWithOps to get un-deployed
2021/12/09 00:16:25 App: TestKVRebInWithOps got un-deployed
2021/12/09 00:16:25 No eventing-con process running
2021/12/09 00:16:25 Function: TestKVRebInWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVRebInWithOps deleting in the background"
}
2021/12/09 00:16:27 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 00:16:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 00:16:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 00:16:34 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 00:16:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 00:16:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebInWithOps (453.04s)
=== RUN   TestKVRebOutWithOps
2021/12/09 00:16:44 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/09 00:16:44 Removing node(s): [] from the cluster
2021/12/09 00:16:49 Rebalance progress: 100
2021/12/09 00:16:54 Waiting for app: TestKVRebOutWithOps to get un-deployed
2021/12/09 00:16:54 App: TestKVRebOutWithOps got un-deployed
2021/12/09 00:16:54 No eventing-con process running
2021/12/09 00:16:54 Function: TestKVRebOutWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVRebOutWithOps not found"
 }
}
2021/12/09 00:16:56 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 00:17:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 00:17:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 00:17:03 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 00:17:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 00:17:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 00:17:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 00:17:13 Deploying app: TestKVRebOutWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/09 00:17:14 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVRebOutWithOps' in metakv",
  "warnings": null
 }
}
2021/12/09 00:17:19 Waiting for app: TestKVRebOutWithOps to get deployed
2021/12/09 00:17:24 Waiting for app: TestKVRebOutWithOps to get deployed
2021/12/09 00:17:29 Waiting for app: TestKVRebOutWithOps to get deployed
2021/12/09 00:17:34 Waiting for app: TestKVRebOutWithOps to get deployed
2021/12/09 00:17:34 App: TestKVRebOutWithOps got deployed
2021/12/09 00:17:34 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/09 00:17:34 Starting bucket ops to source bucket
2021/12/09 00:17:39 Rebalance progress: 12.5
2021/12/09 00:17:44 Rebalance progress: 12.5
2021/12/09 00:17:49 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 00:17:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 00:17:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 00:17:49 Waiting for src bucket item count to get to: 200000 curr count: 8244
2021/12/09 00:17:54 Waiting for src bucket item count to get to: 200000 curr count: 24407
2021/12/09 00:17:59 Waiting for src bucket item count to get to: 200000 curr count: 24407
2021/12/09 00:18:04 Waiting for src bucket item count to get to: 200000 curr count: 40580
2021/12/09 00:18:09 Waiting for src bucket item count to get to: 200000 curr count: 40580
2021/12/09 00:18:14 Waiting for src bucket item count to get to: 200000 curr count: 57105
2021/12/09 00:18:19 Waiting for src bucket item count to get to: 200000 curr count: 57105
2021/12/09 00:18:24 Waiting for src bucket item count to get to: 200000 curr count: 57105
2021/12/09 00:18:29 Waiting for src bucket item count to get to: 200000 curr count: 73323
2021/12/09 00:18:34 Waiting for src bucket item count to get to: 200000 curr count: 73323
2021/12/09 00:18:39 Waiting for src bucket item count to get to: 200000 curr count: 89071
2021/12/09 00:18:44 Waiting for src bucket item count to get to: 200000 curr count: 89071
2021/12/09 00:18:49 Waiting for src bucket item count to get to: 200000 curr count: 105227
2021/12/09 00:18:54 Waiting for src bucket item count to get to: 200000 curr count: 105227
2021/12/09 00:18:59 Waiting for src bucket item count to get to: 200000 curr count: 121519
2021/12/09 00:19:04 Waiting for src bucket item count to get to: 200000 curr count: 121519
2021/12/09 00:19:09 Waiting for src bucket item count to get to: 200000 curr count: 137994
2021/12/09 00:19:14 Waiting for src bucket item count to get to: 200000 curr count: 154280
2021/12/09 00:19:19 Waiting for src bucket item count to get to: 200000 curr count: 154280
2021/12/09 00:19:24 Waiting for src bucket item count to get to: 200000 curr count: 170433
2021/12/09 00:19:29 Waiting for src bucket item count to get to: 200000 curr count: 170433
2021/12/09 00:19:34 Waiting for src bucket item count to get to: 200000 curr count: 186120
2021/12/09 00:19:39 Waiting for src bucket item count to get to: 200000 curr count: 186120
2021/12/09 00:19:44 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/09 00:19:44 Post SET ops. Expected item count: 200000 got 200000
2021/12/09 00:19:44 Starting bucket ops to source bucket
2021/12/09 00:23:29 Waiting for dst bucket item count to get to: 0 curr count: 8125
2021/12/09 00:23:34 Waiting for dst bucket item count to get to: 0 curr count: 8125
2021/12/09 00:23:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 00:23:34 Post DELETE ops. Expected item count: 0 got 0
2021/12/09 00:23:34 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 00:23:39 Waiting for app: TestKVRebOutWithOps to get un-deployed
2021/12/09 00:23:44 Waiting for app: TestKVRebOutWithOps to get un-deployed
2021/12/09 00:23:49 Waiting for app: TestKVRebOutWithOps to get un-deployed
2021/12/09 00:23:54 Waiting for app: TestKVRebOutWithOps to get un-deployed
2021/12/09 00:23:54 App: TestKVRebOutWithOps got un-deployed
2021/12/09 00:23:54 No eventing-con process running
2021/12/09 00:23:54 Function: TestKVRebOutWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVRebOutWithOps deleting in the background"
}
2021/12/09 00:23:56 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 00:24:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 00:24:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 00:24:03 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 00:24:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 00:24:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebOutWithOps (448.99s)
=== RUN   TestKVSwapRebWithOps
2021/12/09 00:24:18 Waiting for app: TestKVSwapRebWithOps to get un-deployed
2021/12/09 00:24:18 App: TestKVSwapRebWithOps got un-deployed
2021/12/09 00:24:18 No eventing-con process running
2021/12/09 00:24:18 Function: TestKVSwapRebWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVSwapRebWithOps not found"
 }
}
2021/12/09 00:24:19 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 00:24:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 00:24:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 00:24:26 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 00:24:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 00:24:31 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 00:24:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 00:24:36 Deploying app: TestKVSwapRebWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/09 00:24:37 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVSwapRebWithOps' in metakv",
  "warnings": null
 }
}
2021/12/09 00:24:42 Waiting for app: TestKVSwapRebWithOps to get deployed
2021/12/09 00:24:47 Waiting for app: TestKVSwapRebWithOps to get deployed
2021/12/09 00:24:52 Waiting for app: TestKVSwapRebWithOps to get deployed
2021/12/09 00:24:57 Waiting for app: TestKVSwapRebWithOps to get deployed
2021/12/09 00:24:57 App: TestKVSwapRebWithOps got deployed
2021/12/09 00:24:57 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/09 00:24:57 Starting bucket ops to source bucket
2021/12/09 00:24:58 Removing node(s): [] from the cluster
2021/12/09 00:25:03 Rebalance progress: 12.5
2021/12/09 00:25:08 Rebalance progress: 12.5
2021/12/09 00:25:13 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 00:25:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 00:25:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 00:25:13 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/09 00:25:13 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/09 00:25:18 Rebalance progress: 12.5
2021/12/09 00:25:23 Rebalance progress: 12.5
2021/12/09 00:25:28 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 00:25:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 00:25:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 00:25:28 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/09 00:25:33 Rebalance progress: 12.5
2021/12/09 00:25:38 Rebalance progress: 12.5
2021/12/09 00:25:43 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 00:25:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 00:25:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 00:25:43 Waiting for src bucket item count to get to: 200000 curr count: 64959
2021/12/09 00:25:48 Waiting for src bucket item count to get to: 200000 curr count: 64959
2021/12/09 00:25:53 Waiting for src bucket item count to get to: 200000 curr count: 80095
2021/12/09 00:25:58 Waiting for src bucket item count to get to: 200000 curr count: 80095
2021/12/09 00:26:03 Waiting for src bucket item count to get to: 200000 curr count: 96523
2021/12/09 00:26:08 Waiting for src bucket item count to get to: 200000 curr count: 96523
2021/12/09 00:26:13 Waiting for src bucket item count to get to: 200000 curr count: 112773
2021/12/09 00:26:18 Waiting for src bucket item count to get to: 200000 curr count: 112773
2021/12/09 00:26:23 Waiting for src bucket item count to get to: 200000 curr count: 129147
2021/12/09 00:26:28 Waiting for src bucket item count to get to: 200000 curr count: 129147
2021/12/09 00:26:33 Waiting for src bucket item count to get to: 200000 curr count: 145619
2021/12/09 00:26:38 Waiting for src bucket item count to get to: 200000 curr count: 145619
2021/12/09 00:26:43 Waiting for src bucket item count to get to: 200000 curr count: 145619
2021/12/09 00:26:48 Waiting for src bucket item count to get to: 200000 curr count: 162128
2021/12/09 00:26:53 Waiting for src bucket item count to get to: 200000 curr count: 162128
2021/12/09 00:26:58 Waiting for src bucket item count to get to: 200000 curr count: 177299
2021/12/09 00:27:03 Waiting for src bucket item count to get to: 200000 curr count: 177299
2021/12/09 00:27:08 Waiting for src bucket item count to get to: 200000 curr count: 193604
2021/12/09 00:27:13 Waiting for src bucket item count to get to: 200000 curr count: 193604
2021/12/09 00:27:18 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/09 00:27:18 Post SET ops. Expected item count: 200000 got 200000
2021/12/09 00:27:18 Starting bucket ops to source bucket
2021/12/09 00:31:03 Waiting for dst bucket item count to get to: 0 curr count: 15485
2021/12/09 00:31:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 00:31:03 Post DELETE ops. Expected item count: 0 got 0
2021/12/09 00:31:03 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 00:31:08 Waiting for app: TestKVSwapRebWithOps to get un-deployed
2021/12/09 00:31:13 Waiting for app: TestKVSwapRebWithOps to get un-deployed
2021/12/09 00:31:18 Waiting for app: TestKVSwapRebWithOps to get un-deployed
2021/12/09 00:31:23 Waiting for app: TestKVSwapRebWithOps to get un-deployed
2021/12/09 00:31:23 App: TestKVSwapRebWithOps got un-deployed
2021/12/09 00:31:23 No eventing-con process running
2021/12/09 00:31:23 Function: TestKVSwapRebWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVSwapRebWithOps deleting in the background"
}
2021/12/09 00:31:25 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 00:31:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 00:31:30 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 00:31:31 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 00:31:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 00:31:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVSwapRebWithOps (448.71s)
=== RUN   TestEventingRebInWithTimers
2021/12/09 00:31:46 Waiting for app: TestEventingRebInWithTimers to get un-deployed
2021/12/09 00:31:46 App: TestEventingRebInWithTimers got un-deployed
2021/12/09 00:31:46 No eventing-con process running
2021/12/09 00:31:46 Function: TestEventingRebInWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebInWithTimers not found"
 }
}
2021/12/09 00:31:48 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 00:31:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 00:31:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 00:31:55 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 00:32:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 00:32:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 00:32:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 00:32:05 Deploying app: TestEventingRebInWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/09 00:32:06 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebInWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/09 00:32:11 Waiting for app: TestEventingRebInWithTimers to get deployed
2021/12/09 00:32:16 Waiting for app: TestEventingRebInWithTimers to get deployed
2021/12/09 00:32:21 Waiting for app: TestEventingRebInWithTimers to get deployed
2021/12/09 00:32:26 Waiting for app: TestEventingRebInWithTimers to get deployed
2021/12/09 00:32:26 App: TestEventingRebInWithTimers got deployed
2021/12/09 00:32:26 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/09 00:32:26 Starting bucket ops to source bucket
2021/12/09 00:32:26 Removing node(s): [] from the cluster
2021/12/09 00:32:31 Rebalance progress: 12.5
2021/12/09 00:32:36 Rebalance progress: 12.5
2021/12/09 00:32:42 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 00:32:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 00:32:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 00:32:42 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/09 00:32:47 Rebalance progress: 12.5
2021/12/09 00:32:52 Rebalance progress: 12.5
2021/12/09 00:32:57 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 00:32:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 00:32:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 00:32:57 Waiting for src bucket item count to get to: 50000 curr count: 28559
2021/12/09 00:33:02 Waiting for src bucket item count to get to: 50000 curr count: 39815
2021/12/09 00:33:07 Waiting for src bucket item count to get to: 50000 curr count: 39815
2021/12/09 00:33:17 Waiting for dst bucket item count to get to: 50000 curr count: 37993
2021/12/09 00:33:22 Waiting for dst bucket item count to get to: 50000 curr count: 37993
2021/12/09 00:33:27 Waiting for dst bucket item count to get to: 50000 curr count: 46991
2021/12/09 00:33:32 Waiting for dst bucket item count to get to: 50000 curr count: 46991
2021/12/09 00:33:32 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/09 00:33:32 Post SET ops. Expected item count: 50000 got 50000
2021/12/09 00:33:32 Starting bucket ops to source bucket
2021/12/09 00:34:40 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/09 00:34:45 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/09 00:34:50 Waiting for dst bucket item count to get to: 0 curr count: 48240
2021/12/09 00:34:55 Waiting for dst bucket item count to get to: 0 curr count: 48240
2021/12/09 00:35:00 Waiting for dst bucket item count to get to: 0 curr count: 39804
2021/12/09 00:35:05 Waiting for dst bucket item count to get to: 0 curr count: 39804
2021/12/09 00:35:10 Waiting for dst bucket item count to get to: 0 curr count: 26291
2021/12/09 00:35:15 Waiting for dst bucket item count to get to: 0 curr count: 26291
2021/12/09 00:35:20 Waiting for dst bucket item count to get to: 0 curr count: 1991
2021/12/09 00:35:25 Waiting for dst bucket item count to get to: 0 curr count: 1991
2021/12/09 00:35:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 00:35:25 Post DELETE ops. Expected item count: 0 got 0
2021/12/09 00:35:26 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 00:35:31 Waiting for app: TestEventingRebInWithTimers to get un-deployed
2021/12/09 00:35:36 Waiting for app: TestEventingRebInWithTimers to get un-deployed
2021/12/09 00:35:41 Waiting for app: TestEventingRebInWithTimers to get un-deployed
2021/12/09 00:35:46 Waiting for app: TestEventingRebInWithTimers to get un-deployed
2021/12/09 00:35:46 App: TestEventingRebInWithTimers got un-deployed
2021/12/09 00:35:46 No eventing-con process running
2021/12/09 00:35:46 Function: TestEventingRebInWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebInWithTimers deleting in the background"
}
2021/12/09 00:35:47 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 00:35:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 00:35:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 00:35:54 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 00:35:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 00:35:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebInWithTimers (262.68s)
=== RUN   TestEventingRebOutWithTimers
2021/12/09 00:36:04 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/09 00:36:04 Removing node(s): [] from the cluster
2021/12/09 00:36:09 Rebalance progress: 100
2021/12/09 00:36:14 Waiting for app: TestEventingRebOutWithTimers to get un-deployed
2021/12/09 00:36:14 App: TestEventingRebOutWithTimers got un-deployed
2021/12/09 00:36:14 No eventing-con process running
2021/12/09 00:36:14 Function: TestEventingRebOutWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebOutWithTimers not found"
 }
}
2021/12/09 00:36:16 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 00:36:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 00:36:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 00:36:23 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 00:36:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 00:36:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 00:36:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 00:36:33 Deploying app: TestEventingRebOutWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/09 00:36:34 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebOutWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/09 00:36:39 Waiting for app: TestEventingRebOutWithTimers to get deployed
2021/12/09 00:36:44 Waiting for app: TestEventingRebOutWithTimers to get deployed
2021/12/09 00:36:49 Waiting for app: TestEventingRebOutWithTimers to get deployed
2021/12/09 00:36:54 Waiting for app: TestEventingRebOutWithTimers to get deployed
2021/12/09 00:36:54 App: TestEventingRebOutWithTimers got deployed
2021/12/09 00:36:54 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/09 00:36:54 Starting bucket ops to source bucket
2021/12/09 00:36:59 Rebalance progress: 12.5
2021/12/09 00:37:04 Rebalance progress: 12.5
2021/12/09 00:37:09 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 00:37:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 00:37:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 00:37:09 Waiting for src bucket item count to get to: 50000 curr count: 6758
2021/12/09 00:37:14 Waiting for src bucket item count to get to: 50000 curr count: 20901
2021/12/09 00:37:19 Waiting for src bucket item count to get to: 50000 curr count: 20901
2021/12/09 00:37:24 Waiting for src bucket item count to get to: 50000 curr count: 32591
2021/12/09 00:37:29 Waiting for src bucket item count to get to: 50000 curr count: 32591
2021/12/09 00:37:34 Waiting for src bucket item count to get to: 50000 curr count: 43993
2021/12/09 00:37:39 Waiting for src bucket item count to get to: 50000 curr count: 43993
2021/12/09 00:37:44 Waiting for src bucket item count to get to: 50000 curr count: 43993
2021/12/09 00:37:54 Waiting for dst bucket item count to get to: 50000 curr count: 35764
2021/12/09 00:37:59 Waiting for dst bucket item count to get to: 50000 curr count: 35764
2021/12/09 00:38:04 Waiting for dst bucket item count to get to: 50000 curr count: 46208
2021/12/09 00:38:04 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/09 00:38:04 Post SET ops. Expected item count: 50000 got 50000
2021/12/09 00:38:04 Starting bucket ops to source bucket
2021/12/09 00:39:13 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/09 00:39:18 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/09 00:39:23 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/09 00:39:28 Waiting for dst bucket item count to get to: 0 curr count: 46470
2021/12/09 00:39:33 Waiting for dst bucket item count to get to: 0 curr count: 46470
2021/12/09 00:39:38 Waiting for dst bucket item count to get to: 0 curr count: 28747
2021/12/09 00:39:43 Waiting for dst bucket item count to get to: 0 curr count: 28747
2021/12/09 00:39:48 Waiting for dst bucket item count to get to: 0 curr count: 13020
2021/12/09 00:39:53 Waiting for dst bucket item count to get to: 0 curr count: 13020
2021/12/09 00:39:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 00:39:53 Post DELETE ops. Expected item count: 0 got 0
2021/12/09 00:39:53 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 00:39:58 Waiting for app: TestEventingRebOutWithTimers to get un-deployed
2021/12/09 00:40:03 Waiting for app: TestEventingRebOutWithTimers to get un-deployed
2021/12/09 00:40:08 Waiting for app: TestEventingRebOutWithTimers to get un-deployed
2021/12/09 00:40:13 Waiting for app: TestEventingRebOutWithTimers to get un-deployed
2021/12/09 00:40:13 App: TestEventingRebOutWithTimers got un-deployed
2021/12/09 00:40:13 No eventing-con process running
2021/12/09 00:40:13 Function: TestEventingRebOutWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebOutWithTimers deleting in the background"
}
2021/12/09 00:40:15 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 00:40:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 00:40:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 00:40:22 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 00:40:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 00:40:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebOutWithTimers (267.62s)
=== RUN   TestEventingSwapRebWithTimers
2021/12/09 00:40:37 Waiting for app: TestEventingSwapRebWithTimers to get un-deployed
2021/12/09 00:40:37 App: TestEventingSwapRebWithTimers got un-deployed
2021/12/09 00:40:37 No eventing-con process running
2021/12/09 00:40:37 Function: TestEventingSwapRebWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingSwapRebWithTimers not found"
 }
}
2021/12/09 00:40:38 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 00:40:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 00:40:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 00:40:45 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 00:40:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 00:40:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 00:40:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 00:40:55 Deploying app: TestEventingSwapRebWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/09 00:40:56 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingSwapRebWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/09 00:41:01 Waiting for app: TestEventingSwapRebWithTimers to get deployed
2021/12/09 00:41:06 Waiting for app: TestEventingSwapRebWithTimers to get deployed
2021/12/09 00:41:11 Waiting for app: TestEventingSwapRebWithTimers to get deployed
2021/12/09 00:41:16 Waiting for app: TestEventingSwapRebWithTimers to get deployed
2021/12/09 00:41:16 App: TestEventingSwapRebWithTimers got deployed
2021/12/09 00:41:16 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/09 00:41:16 Starting bucket ops to source bucket
2021/12/09 00:41:17 Removing node(s): [] from the cluster
2021/12/09 00:41:22 Rebalance progress: 12.5
2021/12/09 00:41:27 Rebalance progress: 12.5
2021/12/09 00:41:32 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 00:41:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 00:41:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 00:41:32 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/09 00:41:33 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/09 00:41:38 Rebalance progress: 12.5
2021/12/09 00:41:43 Rebalance progress: 12.5
2021/12/09 00:41:48 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 00:41:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 00:41:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 00:41:48 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/09 00:41:53 Rebalance progress: 12.5
2021/12/09 00:41:58 Rebalance progress: 12.5
2021/12/09 00:42:03 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 00:42:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 00:42:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 00:42:03 Waiting for src bucket item count to get to: 50000 curr count: 49017
2021/12/09 00:42:08 Waiting for src bucket item count to get to: 50000 curr count: 49017
2021/12/09 00:42:18 Waiting for dst bucket item count to get to: 50000 curr count: 47639
2021/12/09 00:42:23 Waiting for dst bucket item count to get to: 50000 curr count: 47639
2021/12/09 00:42:23 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/09 00:42:23 Post SET ops. Expected item count: 50000 got 50000
2021/12/09 00:42:23 Starting bucket ops to source bucket
2021/12/09 00:43:32 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/09 00:43:37 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/09 00:43:42 Waiting for dst bucket item count to get to: 0 curr count: 47733
2021/12/09 00:43:47 Waiting for dst bucket item count to get to: 0 curr count: 47733
2021/12/09 00:43:52 Waiting for dst bucket item count to get to: 0 curr count: 39356
2021/12/09 00:43:57 Waiting for dst bucket item count to get to: 0 curr count: 39356
2021/12/09 00:44:02 Waiting for dst bucket item count to get to: 0 curr count: 24946
2021/12/09 00:44:07 Waiting for dst bucket item count to get to: 0 curr count: 24946
2021/12/09 00:44:12 Waiting for dst bucket item count to get to: 0 curr count: 2345
2021/12/09 00:44:17 Waiting for dst bucket item count to get to: 0 curr count: 2345
2021/12/09 00:44:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 00:44:17 Post DELETE ops. Expected item count: 0 got 0
2021/12/09 00:44:17 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 00:44:22 Waiting for app: TestEventingSwapRebWithTimers to get un-deployed
2021/12/09 00:44:27 Waiting for app: TestEventingSwapRebWithTimers to get un-deployed
2021/12/09 00:44:32 Waiting for app: TestEventingSwapRebWithTimers to get un-deployed
2021/12/09 00:44:37 Waiting for app: TestEventingSwapRebWithTimers to get un-deployed
2021/12/09 00:44:37 App: TestEventingSwapRebWithTimers got un-deployed
2021/12/09 00:44:37 No eventing-con process running
2021/12/09 00:44:37 Function: TestEventingSwapRebWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingSwapRebWithTimers deleting in the background"
}
2021/12/09 00:44:39 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 00:44:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 00:44:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 00:44:46 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 00:44:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 00:44:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingSwapRebWithTimers (263.85s)
=== RUN   TestKVRebInWithTimers
2021/12/09 00:45:01 Waiting for app: TestKVRebInWithTimers to get un-deployed
2021/12/09 00:45:01 App: TestKVRebInWithTimers got un-deployed
2021/12/09 00:45:01 No eventing-con process running
2021/12/09 00:45:01 Function: TestKVRebInWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVRebInWithTimers not found"
 }
}
2021/12/09 00:45:02 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 00:45:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 00:45:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 00:45:09 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 00:45:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 00:45:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 00:45:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 00:45:19 Deploying app: TestKVRebInWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/09 00:45:20 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVRebInWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/09 00:45:25 Waiting for app: TestKVRebInWithTimers to get deployed
2021/12/09 00:45:30 Waiting for app: TestKVRebInWithTimers to get deployed
2021/12/09 00:45:35 Waiting for app: TestKVRebInWithTimers to get deployed
2021/12/09 00:45:40 Waiting for app: TestKVRebInWithTimers to get deployed
2021/12/09 00:45:40 App: TestKVRebInWithTimers got deployed
2021/12/09 00:45:40 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/09 00:45:40 Starting bucket ops to source bucket
2021/12/09 00:45:41 Removing node(s): [] from the cluster
2021/12/09 00:45:46 Rebalance progress: 12.5
2021/12/09 00:45:51 Rebalance progress: 12.5
2021/12/09 00:45:56 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 00:45:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 00:45:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 00:45:56 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/09 00:46:01 Rebalance progress: 12.5
2021/12/09 00:46:06 Rebalance progress: 12.5
2021/12/09 00:46:11 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 00:46:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 00:46:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 00:46:11 Waiting for src bucket item count to get to: 50000 curr count: 36879
2021/12/09 00:46:16 Waiting for src bucket item count to get to: 50000 curr count: 36879
2021/12/09 00:46:21 Waiting for src bucket item count to get to: 50000 curr count: 47667
2021/12/09 00:46:26 Waiting for src bucket item count to get to: 50000 curr count: 47667
2021/12/09 00:46:36 Waiting for dst bucket item count to get to: 50000 curr count: 38406
2021/12/09 00:46:41 Waiting for dst bucket item count to get to: 50000 curr count: 38406
2021/12/09 00:46:41 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/09 00:46:41 Post SET ops. Expected item count: 50000 got 50000
2021/12/09 00:46:41 Starting bucket ops to source bucket
2021/12/09 00:47:50 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/09 00:47:55 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/09 00:48:00 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/09 00:48:05 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/09 00:48:10 Waiting for dst bucket item count to get to: 0 curr count: 42451
2021/12/09 00:48:15 Waiting for dst bucket item count to get to: 0 curr count: 42451
2021/12/09 00:48:20 Waiting for dst bucket item count to get to: 0 curr count: 22342
2021/12/09 00:48:25 Waiting for dst bucket item count to get to: 0 curr count: 22342
2021/12/09 00:48:30 Waiting for dst bucket item count to get to: 0 curr count: 10933
2021/12/09 00:48:35 Waiting for dst bucket item count to get to: 0 curr count: 10933
2021/12/09 00:48:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 00:48:35 Post DELETE ops. Expected item count: 0 got 0
2021/12/09 00:48:35 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 00:48:40 Waiting for app: TestKVRebInWithTimers to get un-deployed
2021/12/09 00:48:45 Waiting for app: TestKVRebInWithTimers to get un-deployed
2021/12/09 00:48:50 Waiting for app: TestKVRebInWithTimers to get un-deployed
2021/12/09 00:48:55 Waiting for app: TestKVRebInWithTimers to get un-deployed
2021/12/09 00:48:55 App: TestKVRebInWithTimers got un-deployed
2021/12/09 00:48:55 No eventing-con process running
2021/12/09 00:48:55 Function: TestKVRebInWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVRebInWithTimers deleting in the background"
}
2021/12/09 00:48:57 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 00:49:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 00:49:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 00:49:04 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 00:49:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 00:49:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebInWithTimers (258.00s)
=== RUN   TestKVRebOutWithTimers
2021/12/09 00:49:14 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/09 00:49:14 Removing node(s): [] from the cluster
2021/12/09 00:49:19 Rebalance progress: 100
2021/12/09 00:49:24 Waiting for app: TestKVRebOutWithTimers to get un-deployed
2021/12/09 00:49:24 App: TestKVRebOutWithTimers got un-deployed
2021/12/09 00:49:24 No eventing-con process running
2021/12/09 00:49:24 Function: TestKVRebOutWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVRebOutWithTimers not found"
 }
}
2021/12/09 00:49:26 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 00:49:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 00:49:31 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 00:49:32 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 00:49:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 00:49:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 00:49:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 00:49:42 Deploying app: TestKVRebOutWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/09 00:49:43 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVRebOutWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/09 00:49:48 Waiting for app: TestKVRebOutWithTimers to get deployed
2021/12/09 00:49:53 Waiting for app: TestKVRebOutWithTimers to get deployed
2021/12/09 00:49:58 Waiting for app: TestKVRebOutWithTimers to get deployed
2021/12/09 00:50:04 Waiting for app: TestKVRebOutWithTimers to get deployed
2021/12/09 00:50:04 App: TestKVRebOutWithTimers got deployed
2021/12/09 00:50:04 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/09 00:50:04 Starting bucket ops to source bucket
2021/12/09 00:50:09 Rebalance progress: 12.5
2021/12/09 00:50:14 Rebalance progress: 12.5
2021/12/09 00:50:19 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 00:50:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 00:50:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 00:50:19 Waiting for src bucket item count to get to: 50000 curr count: 7748
2021/12/09 00:50:24 Waiting for src bucket item count to get to: 50000 curr count: 20809
2021/12/09 00:50:29 Waiting for src bucket item count to get to: 50000 curr count: 20809
2021/12/09 00:50:34 Waiting for src bucket item count to get to: 50000 curr count: 32816
2021/12/09 00:50:39 Waiting for src bucket item count to get to: 50000 curr count: 32816
2021/12/09 00:50:44 Waiting for src bucket item count to get to: 50000 curr count: 44221
2021/12/09 00:50:49 Waiting for src bucket item count to get to: 50000 curr count: 44221
2021/12/09 00:50:54 Waiting for src bucket item count to get to: 50000 curr count: 44221
2021/12/09 00:51:04 Waiting for dst bucket item count to get to: 50000 curr count: 39576
2021/12/09 00:51:09 Waiting for dst bucket item count to get to: 50000 curr count: 39576
2021/12/09 00:51:14 Waiting for dst bucket item count to get to: 50000 curr count: 49628
2021/12/09 00:51:19 Waiting for dst bucket item count to get to: 50000 curr count: 49628
2021/12/09 00:51:19 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/09 00:51:19 Post SET ops. Expected item count: 50000 got 50000
2021/12/09 00:51:19 Starting bucket ops to source bucket
2021/12/09 00:52:28 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/09 00:52:33 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/09 00:52:38 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/09 00:52:43 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/09 00:52:48 Waiting for dst bucket item count to get to: 0 curr count: 34286
2021/12/09 00:52:53 Waiting for dst bucket item count to get to: 0 curr count: 34286
2021/12/09 00:52:58 Waiting for dst bucket item count to get to: 0 curr count: 19936
2021/12/09 00:53:03 Waiting for dst bucket item count to get to: 0 curr count: 19936
2021/12/09 00:53:08 Waiting for dst bucket item count to get to: 0 curr count: 1533
2021/12/09 00:53:13 Waiting for dst bucket item count to get to: 0 curr count: 1533
2021/12/09 00:53:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 00:53:13 Post DELETE ops. Expected item count: 0 got 0
2021/12/09 00:53:13 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 00:53:18 Waiting for app: TestKVRebOutWithTimers to get un-deployed
2021/12/09 00:53:23 Waiting for app: TestKVRebOutWithTimers to get un-deployed
2021/12/09 00:53:28 Waiting for app: TestKVRebOutWithTimers to get un-deployed
2021/12/09 00:53:33 Waiting for app: TestKVRebOutWithTimers to get un-deployed
2021/12/09 00:53:33 App: TestKVRebOutWithTimers got un-deployed
2021/12/09 00:53:33 No eventing-con process running
2021/12/09 00:53:33 Function: TestKVRebOutWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVRebOutWithTimers deleting in the background"
}
2021/12/09 00:53:35 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 00:53:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 00:53:40 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 00:53:42 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 00:53:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 00:53:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebOutWithTimers (278.06s)
=== RUN   TestKVSwapRebWithTimers
2021/12/09 00:53:57 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/09 00:53:57 App: TestKVSwapRebWithTimers got un-deployed
2021/12/09 00:53:57 No eventing-con process running
2021/12/09 00:53:57 Function: TestKVSwapRebWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVSwapRebWithTimers not found"
 }
}
2021/12/09 00:53:58 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 00:54:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 00:54:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 00:54:05 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 00:54:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 00:54:10 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 00:54:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 00:54:15 Deploying app: TestKVSwapRebWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/09 00:54:16 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVSwapRebWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/09 00:54:21 Waiting for app: TestKVSwapRebWithTimers to get deployed
2021/12/09 00:54:26 Waiting for app: TestKVSwapRebWithTimers to get deployed
2021/12/09 00:54:31 Waiting for app: TestKVSwapRebWithTimers to get deployed
2021/12/09 00:54:36 Waiting for app: TestKVSwapRebWithTimers to get deployed
2021/12/09 00:54:36 App: TestKVSwapRebWithTimers got deployed
2021/12/09 00:54:36 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/09 00:54:36 Starting bucket ops to source bucket
2021/12/09 00:54:37 Removing node(s): [] from the cluster
2021/12/09 00:54:42 Rebalance progress: 12.5
2021/12/09 00:54:47 Rebalance progress: 12.5
2021/12/09 00:54:52 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 00:54:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 00:54:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 00:54:52 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/09 00:54:53 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/09 00:54:58 Rebalance progress: 12.5
2021/12/09 00:55:03 Rebalance progress: 12.5
2021/12/09 00:55:08 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 00:55:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 00:55:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 00:55:08 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/09 00:55:13 Rebalance progress: 12.5
2021/12/09 00:55:18 Rebalance progress: 12.5
2021/12/09 00:55:23 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 00:55:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 00:55:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 00:55:28 Waiting for dst bucket item count to get to: 50000 curr count: 36744
2021/12/09 00:55:33 Waiting for dst bucket item count to get to: 50000 curr count: 36744
2021/12/09 00:55:38 Waiting for dst bucket item count to get to: 50000 curr count: 46176
2021/12/09 00:55:43 Waiting for dst bucket item count to get to: 50000 curr count: 46176
2021/12/09 00:55:43 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/09 00:55:43 Post SET ops. Expected item count: 50000 got 50000
2021/12/09 00:55:43 Starting bucket ops to source bucket
2021/12/09 00:56:53 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/09 00:56:58 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/09 00:57:03 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/09 00:57:08 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/09 00:57:13 Waiting for dst bucket item count to get to: 0 curr count: 42151
2021/12/09 00:57:18 Waiting for dst bucket item count to get to: 0 curr count: 42151
2021/12/09 00:57:23 Waiting for dst bucket item count to get to: 0 curr count: 32607
2021/12/09 00:57:28 Waiting for dst bucket item count to get to: 0 curr count: 32607
2021/12/09 00:57:33 Waiting for dst bucket item count to get to: 0 curr count: 6776
2021/12/09 00:57:38 Waiting for dst bucket item count to get to: 0 curr count: 6776
2021/12/09 00:57:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 00:57:38 Post DELETE ops. Expected item count: 0 got 0
2021/12/09 00:57:38 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 00:57:43 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/09 00:57:48 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/09 00:57:53 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/09 00:57:58 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/09 00:58:03 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/09 00:58:08 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/09 00:58:08 App: TestKVSwapRebWithTimers got un-deployed
2021/12/09 00:58:08 No eventing-con process running
2021/12/09 00:58:08 Function: TestKVSwapRebWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVSwapRebWithTimers deleting in the background"
}
2021/12/09 00:58:10 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 00:58:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 00:58:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 00:58:16 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 00:58:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 00:58:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVSwapRebWithTimers (274.81s)
=== RUN   TestDuplicateTimerCreationEventingRebIn
2021/12/09 00:58:31 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/09 00:58:31 App: TestDuplicateTimerCreationEventingRebIn got un-deployed
2021/12/09 00:58:31 No eventing-con process running
2021/12/09 00:58:31 Function: TestDuplicateTimerCreationEventingRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationEventingRebIn not found"
 }
}
2021/12/09 00:58:33 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 00:58:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 00:58:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 00:58:40 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 00:58:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 00:58:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 00:58:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 00:58:50 Deploying app: TestDuplicateTimerCreationEventingRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/09 00:58:51 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationEventingRebIn' in metakv",
  "warnings": null
 }
}
2021/12/09 00:58:56 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get deployed
2021/12/09 00:59:01 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get deployed
2021/12/09 00:59:06 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get deployed
2021/12/09 00:59:11 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get deployed
2021/12/09 00:59:11 App: TestDuplicateTimerCreationEventingRebIn got deployed
2021/12/09 00:59:11 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/09 00:59:11 Starting bucket ops to source bucket
2021/12/09 00:59:11 Removing node(s): [] from the cluster
2021/12/09 00:59:16 Rebalance progress: 12.5
2021/12/09 00:59:22 Rebalance progress: 12.5
2021/12/09 00:59:26 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 00:59:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 00:59:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 00:59:27 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/09 00:59:32 Rebalance progress: 12.5
2021/12/09 00:59:37 Rebalance progress: 12.5
2021/12/09 00:59:42 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 00:59:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 00:59:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 00:59:42 Waiting for src bucket item count to get to: 50000 curr count: 37330
2021/12/09 00:59:47 Waiting for src bucket item count to get to: 50000 curr count: 37330
2021/12/09 00:59:52 Waiting for src bucket item count to get to: 50000 curr count: 46206
2021/12/09 00:59:57 Waiting for src bucket item count to get to: 50000 curr count: 46206
2021/12/09 01:00:07 Waiting for dst bucket item count to get to: 50000 curr count: 29667
2021/12/09 01:00:12 Waiting for dst bucket item count to get to: 50000 curr count: 29667
2021/12/09 01:00:17 Waiting for dst bucket item count to get to: 50000 curr count: 43489
2021/12/09 01:00:22 Waiting for dst bucket item count to get to: 50000 curr count: 43489
2021/12/09 01:00:22 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/09 01:00:22 Post SET ops. Expected item count: 50000 got 50000
2021/12/09 01:00:22 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 01:00:27 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/09 01:00:32 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/09 01:00:37 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/09 01:00:42 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/09 01:00:47 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/09 01:00:52 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/09 01:00:52 App: TestDuplicateTimerCreationEventingRebIn got un-deployed
2021/12/09 01:00:52 No eventing-con process running
2021/12/09 01:00:52 Function: TestDuplicateTimerCreationEventingRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationEventingRebIn deleting in the background"
}
2021/12/09 01:00:54 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:00:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 01:01:04 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/09 01:01:09 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/09 01:01:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 01:01:10 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:01:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 01:01:20 Waiting for dst bucket item count to get to: 0 curr count: 27831
2021/12/09 01:01:25 Waiting for dst bucket item count to get to: 0 curr count: 27831
2021/12/09 01:01:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationEventingRebIn (184.01s)
=== RUN   TestDuplicateTimerCreationEventingRebOut
2021/12/09 01:01:30 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/09 01:01:31 Removing node(s): [] from the cluster
2021/12/09 01:01:37 Rebalance progress: 100
2021/12/09 01:01:42 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/09 01:01:42 App: TestDuplicateTimerCreationEventingRebOut got un-deployed
2021/12/09 01:01:42 No eventing-con process running
2021/12/09 01:01:42 Function: TestDuplicateTimerCreationEventingRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationEventingRebOut not found"
 }
}
2021/12/09 01:01:43 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:01:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 01:01:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 01:01:50 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:01:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 01:01:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:02:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 01:02:00 Deploying app: TestDuplicateTimerCreationEventingRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/09 01:02:01 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationEventingRebOut' in metakv",
  "warnings": null
 }
}
2021/12/09 01:02:06 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get deployed
2021/12/09 01:02:11 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get deployed
2021/12/09 01:02:16 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get deployed
2021/12/09 01:02:21 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get deployed
2021/12/09 01:02:21 App: TestDuplicateTimerCreationEventingRebOut got deployed
2021/12/09 01:02:21 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/09 01:02:21 Starting bucket ops to source bucket
2021/12/09 01:02:26 Rebalance progress: 12.5
2021/12/09 01:02:31 Rebalance progress: 12.5
2021/12/09 01:02:36 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:02:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:02:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 01:02:36 Waiting for src bucket item count to get to: 50000 curr count: 10598
2021/12/09 01:02:41 Waiting for src bucket item count to get to: 50000 curr count: 25065
2021/12/09 01:02:46 Waiting for src bucket item count to get to: 50000 curr count: 25065
2021/12/09 01:02:51 Waiting for src bucket item count to get to: 50000 curr count: 39195
2021/12/09 01:02:56 Waiting for src bucket item count to get to: 50000 curr count: 39195
2021/12/09 01:03:01 Waiting for src bucket item count to get to: 50000 curr count: 48560
2021/12/09 01:03:06 Waiting for src bucket item count to get to: 50000 curr count: 48560
2021/12/09 01:03:11 Waiting for src bucket item count to get to: 50000 curr count: 48560
2021/12/09 01:03:21 Waiting for dst bucket item count to get to: 50000 curr count: 29591
2021/12/09 01:03:26 Waiting for dst bucket item count to get to: 50000 curr count: 29591
2021/12/09 01:03:31 Waiting for dst bucket item count to get to: 50000 curr count: 45856
2021/12/09 01:03:36 Waiting for dst bucket item count to get to: 50000 curr count: 45856
2021/12/09 01:03:36 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/09 01:03:36 Post SET ops. Expected item count: 50000 got 50000
2021/12/09 01:03:36 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 01:03:41 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/09 01:03:47 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/09 01:03:52 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/09 01:03:57 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/09 01:04:02 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/09 01:04:07 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/09 01:04:07 App: TestDuplicateTimerCreationEventingRebOut got un-deployed
2021/12/09 01:04:07 No eventing-con process running
2021/12/09 01:04:07 Function: TestDuplicateTimerCreationEventingRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationEventingRebOut deleting in the background"
}
2021/12/09 01:04:09 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:04:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 01:04:19 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/09 01:04:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 01:04:20 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:04:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 01:04:30 Waiting for dst bucket item count to get to: 0 curr count: 21259
2021/12/09 01:04:35 Waiting for dst bucket item count to get to: 0 curr count: 21259
2021/12/09 01:04:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationEventingRebOut (189.88s)
=== RUN   TestDuplicateTimerCreationEventingSwapReb
2021/12/09 01:04:45 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/09 01:04:45 App: TestDuplicateTimerCreationEventingSwapReb got un-deployed
2021/12/09 01:04:45 No eventing-con process running
2021/12/09 01:04:45 Function: TestDuplicateTimerCreationEventingSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationEventingSwapReb not found"
 }
}
2021/12/09 01:04:47 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:04:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 01:04:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 01:04:54 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:04:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 01:04:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:05:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 01:05:04 Deploying app: TestDuplicateTimerCreationEventingSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/09 01:05:05 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationEventingSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/09 01:05:10 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get deployed
2021/12/09 01:05:15 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get deployed
2021/12/09 01:05:20 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get deployed
2021/12/09 01:05:25 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get deployed
2021/12/09 01:05:25 App: TestDuplicateTimerCreationEventingSwapReb got deployed
2021/12/09 01:05:25 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/09 01:05:25 Starting bucket ops to source bucket
2021/12/09 01:05:26 Removing node(s): [] from the cluster
2021/12/09 01:05:31 Rebalance progress: 12.5
2021/12/09 01:05:36 Rebalance progress: 12.5
2021/12/09 01:05:41 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:05:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:05:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 01:05:41 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/09 01:05:41 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/09 01:05:46 Rebalance progress: 12.5
2021/12/09 01:05:51 Rebalance progress: 12.5
2021/12/09 01:05:56 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:05:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:05:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 01:05:56 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/09 01:06:01 Rebalance progress: 12.5
2021/12/09 01:06:06 Rebalance progress: 12.5
2021/12/09 01:06:11 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:06:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:06:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 01:06:12 Waiting for src bucket item count to get to: 50000 curr count: 49748
2021/12/09 01:06:17 Waiting for src bucket item count to get to: 50000 curr count: 49748
2021/12/09 01:06:27 Waiting for dst bucket item count to get to: 50000 curr count: 39040
2021/12/09 01:06:32 Waiting for dst bucket item count to get to: 50000 curr count: 39040
2021/12/09 01:06:32 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/09 01:06:32 Post SET ops. Expected item count: 50000 got 50000
2021/12/09 01:06:32 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 01:06:37 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/09 01:06:42 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/09 01:06:47 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/09 01:06:52 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/09 01:06:57 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/09 01:07:02 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/09 01:07:02 App: TestDuplicateTimerCreationEventingSwapReb got un-deployed
2021/12/09 01:07:02 No eventing-con process running
2021/12/09 01:07:02 Function: TestDuplicateTimerCreationEventingSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationEventingSwapReb deleting in the background"
}
2021/12/09 01:07:04 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:07:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 01:07:14 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/09 01:07:19 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/09 01:07:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 01:07:20 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:07:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 01:07:30 Waiting for dst bucket item count to get to: 0 curr count: 26558
2021/12/09 01:07:35 Waiting for dst bucket item count to get to: 0 curr count: 26558
2021/12/09 01:07:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationEventingSwapReb (180.11s)
=== RUN   TestDuplicateTimerCreationKVRebIn
2021/12/09 01:07:45 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/09 01:07:45 App: TestDuplicateTimerCreationKVRebIn got un-deployed
2021/12/09 01:07:45 No eventing-con process running
2021/12/09 01:07:45 Function: TestDuplicateTimerCreationKVRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationKVRebIn not found"
 }
}
2021/12/09 01:07:47 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:07:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 01:07:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 01:07:54 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:07:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 01:07:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:08:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 01:08:04 Deploying app: TestDuplicateTimerCreationKVRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/09 01:08:05 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationKVRebIn' in metakv",
  "warnings": null
 }
}
2021/12/09 01:08:10 Waiting for app: TestDuplicateTimerCreationKVRebIn to get deployed
2021/12/09 01:08:15 Waiting for app: TestDuplicateTimerCreationKVRebIn to get deployed
2021/12/09 01:08:20 Waiting for app: TestDuplicateTimerCreationKVRebIn to get deployed
2021/12/09 01:08:25 Waiting for app: TestDuplicateTimerCreationKVRebIn to get deployed
2021/12/09 01:08:25 App: TestDuplicateTimerCreationKVRebIn got deployed
2021/12/09 01:08:25 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/09 01:08:25 Starting bucket ops to source bucket
2021/12/09 01:08:25 Removing node(s): [] from the cluster
2021/12/09 01:08:31 Rebalance progress: 12.5
2021/12/09 01:08:36 Rebalance progress: 12.5
2021/12/09 01:08:41 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:08:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:08:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 01:08:41 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/09 01:08:46 Rebalance progress: 12.5
2021/12/09 01:08:51 Rebalance progress: 12.5
2021/12/09 01:08:56 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:08:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:08:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 01:08:56 Waiting for src bucket item count to get to: 50000 curr count: 31729
2021/12/09 01:09:01 Waiting for src bucket item count to get to: 50000 curr count: 42805
2021/12/09 01:09:06 Waiting for src bucket item count to get to: 50000 curr count: 42805
2021/12/09 01:09:16 Waiting for dst bucket item count to get to: 50000 curr count: 27735
2021/12/09 01:09:21 Waiting for dst bucket item count to get to: 50000 curr count: 27735
2021/12/09 01:09:26 Waiting for dst bucket item count to get to: 50000 curr count: 36235
2021/12/09 01:09:31 Waiting for dst bucket item count to get to: 50000 curr count: 36235
2021/12/09 01:09:36 Waiting for dst bucket item count to get to: 50000 curr count: 49743
2021/12/09 01:09:41 Waiting for dst bucket item count to get to: 50000 curr count: 49743
2021/12/09 01:09:41 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/09 01:09:41 Post SET ops. Expected item count: 50000 got 50000
2021/12/09 01:09:41 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 01:09:46 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/09 01:09:51 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/09 01:09:56 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/09 01:10:01 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/09 01:10:06 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/09 01:10:06 App: TestDuplicateTimerCreationKVRebIn got un-deployed
2021/12/09 01:10:06 No eventing-con process running
2021/12/09 01:10:06 Function: TestDuplicateTimerCreationKVRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationKVRebIn deleting in the background"
}
2021/12/09 01:10:08 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:10:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 01:10:18 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/09 01:10:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 01:10:20 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:10:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 01:10:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationKVRebIn (169.25s)
=== RUN   TestDuplicateTimerCreationKVRebOut
2021/12/09 01:10:30 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/09 01:10:30 Removing node(s): [] from the cluster
2021/12/09 01:10:35 Rebalance progress: 100
2021/12/09 01:10:40 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/09 01:10:40 App: TestDuplicateTimerCreationKVRebOut got un-deployed
2021/12/09 01:10:40 No eventing-con process running
2021/12/09 01:10:40 Function: TestDuplicateTimerCreationKVRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationKVRebOut not found"
 }
}
2021/12/09 01:10:42 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:10:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 01:10:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 01:10:48 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:10:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 01:10:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:10:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 01:10:58 Deploying app: TestDuplicateTimerCreationKVRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/09 01:10:59 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationKVRebOut' in metakv",
  "warnings": null
 }
}
2021/12/09 01:11:04 Waiting for app: TestDuplicateTimerCreationKVRebOut to get deployed
2021/12/09 01:11:09 Waiting for app: TestDuplicateTimerCreationKVRebOut to get deployed
2021/12/09 01:11:15 Waiting for app: TestDuplicateTimerCreationKVRebOut to get deployed
2021/12/09 01:11:20 Waiting for app: TestDuplicateTimerCreationKVRebOut to get deployed
2021/12/09 01:11:20 App: TestDuplicateTimerCreationKVRebOut got deployed
2021/12/09 01:11:20 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/09 01:11:20 Starting bucket ops to source bucket
2021/12/09 01:11:25 Rebalance progress: 12.5
2021/12/09 01:11:30 Rebalance progress: 12.5
2021/12/09 01:11:35 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:11:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:11:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 01:11:35 Waiting for src bucket item count to get to: 50000 curr count: 12545
2021/12/09 01:11:40 Waiting for src bucket item count to get to: 50000 curr count: 12545
2021/12/09 01:11:45 Waiting for src bucket item count to get to: 50000 curr count: 26607
2021/12/09 01:11:50 Waiting for src bucket item count to get to: 50000 curr count: 26607
2021/12/09 01:11:55 Waiting for src bucket item count to get to: 50000 curr count: 38761
2021/12/09 01:12:00 Waiting for src bucket item count to get to: 50000 curr count: 49239
2021/12/09 01:12:05 Waiting for src bucket item count to get to: 50000 curr count: 49239
2021/12/09 01:12:10 Waiting for src bucket item count to get to: 50000 curr count: 49239
2021/12/09 01:12:20 Waiting for dst bucket item count to get to: 50000 curr count: 35445
2021/12/09 01:12:25 Waiting for dst bucket item count to get to: 50000 curr count: 35445
2021/12/09 01:12:30 Waiting for dst bucket item count to get to: 50000 curr count: 44441
2021/12/09 01:12:35 Waiting for dst bucket item count to get to: 50000 curr count: 44441
2021/12/09 01:12:35 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/09 01:12:35 Post SET ops. Expected item count: 50000 got 50000
2021/12/09 01:12:35 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 01:12:40 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/09 01:12:45 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/09 01:12:50 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/09 01:12:55 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/09 01:13:00 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/09 01:13:05 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/09 01:13:05 App: TestDuplicateTimerCreationKVRebOut got un-deployed
2021/12/09 01:13:05 No eventing-con process running
2021/12/09 01:13:05 Function: TestDuplicateTimerCreationKVRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationKVRebOut deleting in the background"
}
2021/12/09 01:13:07 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:13:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 01:13:17 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/09 01:13:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 01:13:19 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:13:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 01:13:29 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/09 01:13:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationKVRebOut (183.97s)
=== RUN   TestDuplicateTimerCreationKVSwapReb
2021/12/09 01:13:39 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/09 01:13:39 App: TestDuplicateTimerCreationKVSwapReb got un-deployed
2021/12/09 01:13:39 No eventing-con process running
2021/12/09 01:13:39 Function: TestDuplicateTimerCreationKVSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationKVSwapReb not found"
 }
}
2021/12/09 01:13:40 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:13:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 01:13:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 01:13:47 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:13:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 01:13:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:13:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 01:13:57 Deploying app: TestDuplicateTimerCreationKVSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/09 01:13:58 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationKVSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/09 01:14:03 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get deployed
2021/12/09 01:14:08 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get deployed
2021/12/09 01:14:13 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get deployed
2021/12/09 01:14:18 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get deployed
2021/12/09 01:14:18 App: TestDuplicateTimerCreationKVSwapReb got deployed
2021/12/09 01:14:18 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/09 01:14:18 Starting bucket ops to source bucket
2021/12/09 01:14:19 Removing node(s): [] from the cluster
2021/12/09 01:14:24 Rebalance progress: 12.5
2021/12/09 01:14:29 Rebalance progress: 12.5
2021/12/09 01:14:34 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:14:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:14:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 01:14:34 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/09 01:14:34 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/09 01:14:39 Rebalance progress: 12.5
2021/12/09 01:14:44 Rebalance progress: 12.5
2021/12/09 01:14:49 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:14:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:14:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 01:14:49 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/09 01:14:54 Rebalance progress: 12.5
2021/12/09 01:14:59 Rebalance progress: 12.5
2021/12/09 01:15:04 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:15:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:15:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 01:15:04 Waiting for src bucket item count to get to: 50000 curr count: 47162
2021/12/09 01:15:09 Waiting for src bucket item count to get to: 50000 curr count: 47162
2021/12/09 01:15:19 Waiting for dst bucket item count to get to: 50000 curr count: 37577
2021/12/09 01:15:24 Waiting for dst bucket item count to get to: 50000 curr count: 37577
2021/12/09 01:15:29 Waiting for dst bucket item count to get to: 50000 curr count: 44636
2021/12/09 01:15:34 Waiting for dst bucket item count to get to: 50000 curr count: 44636
2021/12/09 01:15:34 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/09 01:15:34 Post SET ops. Expected item count: 50000 got 50000
2021/12/09 01:15:34 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 01:15:39 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/09 01:15:45 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/09 01:15:50 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/09 01:15:55 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/09 01:16:00 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/09 01:16:05 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/09 01:16:05 App: TestDuplicateTimerCreationKVSwapReb got un-deployed
2021/12/09 01:16:05 No eventing-con process running
2021/12/09 01:16:05 Function: TestDuplicateTimerCreationKVSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationKVSwapReb deleting in the background"
}
2021/12/09 01:16:07 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:16:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 01:16:17 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/09 01:16:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 01:16:18 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:16:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 01:16:28 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/09 01:16:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationKVSwapReb (179.61s)
=== RUN   TestDuplicateTimerFiringEventingRebIn
2021/12/09 01:16:38 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/09 01:16:38 App: TestDuplicateTimerFiringEventingRebIn got un-deployed
2021/12/09 01:16:38 No eventing-con process running
2021/12/09 01:16:38 Function: TestDuplicateTimerFiringEventingRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringEventingRebIn not found"
 }
}
2021/12/09 01:16:40 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:16:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 01:16:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 01:16:47 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:16:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 01:16:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:16:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 01:16:57 Deploying app: TestDuplicateTimerFiringEventingRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/09 01:16:58 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringEventingRebIn' in metakv",
  "warnings": null
 }
}
2021/12/09 01:17:03 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get deployed
2021/12/09 01:17:08 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get deployed
2021/12/09 01:17:13 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get deployed
2021/12/09 01:17:18 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get deployed
2021/12/09 01:17:18 App: TestDuplicateTimerFiringEventingRebIn got deployed
2021/12/09 01:17:18 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/09 01:17:18 Starting bucket ops to source bucket
2021/12/09 01:17:19 Removing node(s): [] from the cluster
2021/12/09 01:17:24 Rebalance progress: 12.5
2021/12/09 01:17:29 Rebalance progress: 12.5
2021/12/09 01:17:34 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:17:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:17:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 01:17:34 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/09 01:17:39 Rebalance progress: 12.5
2021/12/09 01:17:44 Rebalance progress: 12.5
2021/12/09 01:17:49 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:17:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:17:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 01:17:49 Waiting for src bucket item count to get to: 50000 curr count: 28144
2021/12/09 01:17:54 Waiting for src bucket item count to get to: 50000 curr count: 37722
2021/12/09 01:17:59 Waiting for src bucket item count to get to: 50000 curr count: 37722
2021/12/09 01:18:04 Waiting for src bucket item count to get to: 50000 curr count: 49597
2021/12/09 01:18:09 Waiting for src bucket item count to get to: 50000 curr count: 49597
2021/12/09 01:18:19 Waiting for dst bucket item count to get to: 50000 curr count: 42484
2021/12/09 01:18:24 Waiting for dst bucket item count to get to: 50000 curr count: 42484
2021/12/09 01:18:29 Waiting for dst bucket item count to get to: 50000 curr count: 49657
2021/12/09 01:18:34 Waiting for dst bucket item count to get to: 50000 curr count: 49657
2021/12/09 01:18:34 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/09 01:18:34 Post SET ops. Expected item count: 50000 got 50000
2021/12/09 01:18:34 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 01:18:39 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/09 01:18:44 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/09 01:18:49 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/09 01:18:54 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/09 01:18:59 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/09 01:19:04 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/09 01:19:04 App: TestDuplicateTimerFiringEventingRebIn got un-deployed
2021/12/09 01:19:04 No eventing-con process running
2021/12/09 01:19:04 Function: TestDuplicateTimerFiringEventingRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringEventingRebIn deleting in the background"
}
2021/12/09 01:19:06 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:19:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 01:19:16 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/09 01:19:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 01:19:18 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:19:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 01:19:28 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/09 01:19:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringEventingRebIn (179.47s)
=== RUN   TestDuplicateTimerFiringEventingRebOut
2021/12/09 01:19:33 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/09 01:19:33 Removing node(s): [] from the cluster
2021/12/09 01:19:38 Rebalance progress: 100
2021/12/09 01:19:43 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/09 01:19:43 App: TestDuplicateTimerFiringEventingRebOut got un-deployed
2021/12/09 01:19:43 No eventing-con process running
2021/12/09 01:19:43 Function: TestDuplicateTimerFiringEventingRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringEventingRebOut not found"
 }
}
2021/12/09 01:19:45 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:19:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 01:19:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 01:19:52 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:19:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 01:19:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:20:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 01:20:02 Deploying app: TestDuplicateTimerFiringEventingRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/09 01:20:03 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringEventingRebOut' in metakv",
  "warnings": null
 }
}
2021/12/09 01:20:08 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get deployed
2021/12/09 01:20:13 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get deployed
2021/12/09 01:20:18 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get deployed
2021/12/09 01:20:23 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get deployed
2021/12/09 01:20:23 App: TestDuplicateTimerFiringEventingRebOut got deployed
2021/12/09 01:20:23 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/09 01:20:23 Starting bucket ops to source bucket
2021/12/09 01:20:28 Rebalance progress: 12.5
2021/12/09 01:20:33 Rebalance progress: 12.5
2021/12/09 01:20:38 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:20:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:20:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 01:20:38 Waiting for src bucket item count to get to: 50000 curr count: 8291
2021/12/09 01:20:43 Waiting for src bucket item count to get to: 50000 curr count: 20623
2021/12/09 01:20:48 Waiting for src bucket item count to get to: 50000 curr count: 20623
2021/12/09 01:20:53 Waiting for src bucket item count to get to: 50000 curr count: 31979
2021/12/09 01:20:58 Waiting for src bucket item count to get to: 50000 curr count: 31979
2021/12/09 01:21:03 Waiting for src bucket item count to get to: 50000 curr count: 31979
2021/12/09 01:21:08 Waiting for src bucket item count to get to: 50000 curr count: 42889
2021/12/09 01:21:13 Waiting for src bucket item count to get to: 50000 curr count: 42889
2021/12/09 01:21:23 Waiting for dst bucket item count to get to: 50000 curr count: 38834
2021/12/09 01:21:28 Waiting for dst bucket item count to get to: 50000 curr count: 38834
2021/12/09 01:21:28 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/09 01:21:28 Post SET ops. Expected item count: 50000 got 50000
2021/12/09 01:21:28 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 01:21:33 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/09 01:21:38 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/09 01:21:43 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/09 01:21:48 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/09 01:21:53 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/09 01:21:58 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/09 01:21:58 App: TestDuplicateTimerFiringEventingRebOut got un-deployed
2021/12/09 01:21:58 No eventing-con process running
2021/12/09 01:21:58 Function: TestDuplicateTimerFiringEventingRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringEventingRebOut deleting in the background"
}
2021/12/09 01:22:00 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:22:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 01:22:10 Waiting for dst bucket item count to get to: 0 curr count: 12356
2021/12/09 01:22:15 Waiting for dst bucket item count to get to: 0 curr count: 12356
2021/12/09 01:22:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 01:22:17 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:22:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 01:22:27 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/09 01:22:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringEventingRebOut (179.22s)
=== RUN   TestDuplicateTimerFiringEventingSwapReb
2021/12/09 01:22:37 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/09 01:22:37 App: TestDuplicateTimerFiringEventingSwapReb got un-deployed
2021/12/09 01:22:37 No eventing-con process running
2021/12/09 01:22:37 Function: TestDuplicateTimerFiringEventingSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringEventingSwapReb not found"
 }
}
2021/12/09 01:22:39 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:22:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 01:22:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 01:22:46 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:22:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 01:22:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:22:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 01:22:56 Deploying app: TestDuplicateTimerFiringEventingSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/09 01:22:57 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringEventingSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/09 01:23:02 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get deployed
2021/12/09 01:23:07 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get deployed
2021/12/09 01:23:12 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get deployed
2021/12/09 01:23:17 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get deployed
2021/12/09 01:23:17 App: TestDuplicateTimerFiringEventingSwapReb got deployed
2021/12/09 01:23:17 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/09 01:23:17 Starting bucket ops to source bucket
2021/12/09 01:23:17 Removing node(s): [] from the cluster
2021/12/09 01:23:22 Rebalance progress: 12.5
2021/12/09 01:23:27 Rebalance progress: 12.5
2021/12/09 01:23:32 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:23:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:23:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 01:23:32 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/09 01:23:33 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/09 01:23:38 Rebalance progress: 12.5
2021/12/09 01:23:43 Rebalance progress: 12.5
2021/12/09 01:23:48 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:23:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:23:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 01:23:48 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/09 01:23:53 Rebalance progress: 12.5
2021/12/09 01:23:58 Rebalance progress: 12.5
2021/12/09 01:24:03 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:24:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:24:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 01:24:03 Waiting for src bucket item count to get to: 50000 curr count: 49600
2021/12/09 01:24:08 Waiting for src bucket item count to get to: 50000 curr count: 49600
2021/12/09 01:24:18 Waiting for dst bucket item count to get to: 50000 curr count: 47966
2021/12/09 01:24:23 Waiting for dst bucket item count to get to: 50000 curr count: 47966
2021/12/09 01:24:23 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/09 01:24:23 Post SET ops. Expected item count: 50000 got 50000
2021/12/09 01:24:23 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 01:24:28 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/09 01:24:33 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/09 01:24:38 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/09 01:24:43 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/09 01:24:48 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/09 01:24:53 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/09 01:24:53 App: TestDuplicateTimerFiringEventingSwapReb got un-deployed
2021/12/09 01:24:53 No eventing-con process running
2021/12/09 01:24:53 Function: TestDuplicateTimerFiringEventingSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringEventingSwapReb deleting in the background"
}
2021/12/09 01:24:55 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:25:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 01:25:05 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/09 01:25:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 01:25:07 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:25:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 01:25:17 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/09 01:25:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringEventingSwapReb (169.88s)
=== RUN   TestDuplicateTimerFiringKVRebIn
2021/12/09 01:25:27 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/09 01:25:27 App: TestDuplicateTimerFiringKVRebIn got un-deployed
2021/12/09 01:25:27 No eventing-con process running
2021/12/09 01:25:27 Function: TestDuplicateTimerFiringKVRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringKVRebIn not found"
 }
}
2021/12/09 01:25:29 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:25:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 01:25:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 01:25:35 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:25:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 01:25:40 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:25:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 01:25:45 Deploying app: TestDuplicateTimerFiringKVRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/09 01:25:46 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringKVRebIn' in metakv",
  "warnings": null
 }
}
2021/12/09 01:25:51 Waiting for app: TestDuplicateTimerFiringKVRebIn to get deployed
2021/12/09 01:25:56 Waiting for app: TestDuplicateTimerFiringKVRebIn to get deployed
2021/12/09 01:26:02 Waiting for app: TestDuplicateTimerFiringKVRebIn to get deployed
2021/12/09 01:26:07 Waiting for app: TestDuplicateTimerFiringKVRebIn to get deployed
2021/12/09 01:26:07 App: TestDuplicateTimerFiringKVRebIn got deployed
2021/12/09 01:26:07 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/09 01:26:07 Starting bucket ops to source bucket
2021/12/09 01:26:07 Removing node(s): [] from the cluster
2021/12/09 01:26:12 Rebalance progress: 12.5
2021/12/09 01:26:17 Rebalance progress: 12.5
2021/12/09 01:26:22 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:26:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:26:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 01:26:22 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/09 01:26:27 Rebalance progress: 12.5
2021/12/09 01:26:32 Rebalance progress: 12.5
2021/12/09 01:26:37 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:26:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:26:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 01:26:37 Waiting for src bucket item count to get to: 50000 curr count: 27487
2021/12/09 01:26:42 Waiting for src bucket item count to get to: 50000 curr count: 38262
2021/12/09 01:26:47 Waiting for src bucket item count to get to: 50000 curr count: 38262
2021/12/09 01:26:52 Waiting for src bucket item count to get to: 50000 curr count: 49534
2021/12/09 01:26:57 Waiting for src bucket item count to get to: 50000 curr count: 49534
2021/12/09 01:27:07 Waiting for dst bucket item count to get to: 50000 curr count: 45075
2021/12/09 01:27:12 Waiting for dst bucket item count to get to: 50000 curr count: 45075
2021/12/09 01:27:12 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/09 01:27:12 Post SET ops. Expected item count: 50000 got 50000
2021/12/09 01:27:12 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 01:27:17 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/09 01:27:22 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/09 01:27:27 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/09 01:27:33 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/09 01:27:38 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/09 01:27:43 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/09 01:27:43 App: TestDuplicateTimerFiringKVRebIn got un-deployed
2021/12/09 01:27:43 No eventing-con process running
2021/12/09 01:27:43 Function: TestDuplicateTimerFiringKVRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringKVRebIn deleting in the background"
}
2021/12/09 01:27:44 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:27:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 01:27:54 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/09 01:27:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 01:27:56 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:28:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 01:28:06 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/09 01:28:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringKVRebIn (169.28s)
=== RUN   TestDuplicateTimerFiringKVRebOut
2021/12/09 01:28:11 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/09 01:28:11 Removing node(s): [] from the cluster
2021/12/09 01:28:16 Rebalance progress: 100
2021/12/09 01:28:21 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/09 01:28:21 App: TestDuplicateTimerFiringKVRebOut got un-deployed
2021/12/09 01:28:21 No eventing-con process running
2021/12/09 01:28:21 Function: TestDuplicateTimerFiringKVRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringKVRebOut not found"
 }
}
2021/12/09 01:28:23 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:28:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 01:28:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 01:28:30 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:28:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 01:28:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:28:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 01:28:40 Deploying app: TestDuplicateTimerFiringKVRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/09 01:28:41 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringKVRebOut' in metakv",
  "warnings": null
 }
}
2021/12/09 01:28:46 Waiting for app: TestDuplicateTimerFiringKVRebOut to get deployed
2021/12/09 01:28:51 Waiting for app: TestDuplicateTimerFiringKVRebOut to get deployed
2021/12/09 01:28:56 Waiting for app: TestDuplicateTimerFiringKVRebOut to get deployed
2021/12/09 01:29:01 Waiting for app: TestDuplicateTimerFiringKVRebOut to get deployed
2021/12/09 01:29:01 App: TestDuplicateTimerFiringKVRebOut got deployed
2021/12/09 01:29:01 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/09 01:29:01 Starting bucket ops to source bucket
2021/12/09 01:29:06 Rebalance progress: 12.5
2021/12/09 01:29:11 Rebalance progress: 12.5
2021/12/09 01:29:16 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:29:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:29:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 01:29:16 Waiting for src bucket item count to get to: 50000 curr count: 11098
2021/12/09 01:29:21 Waiting for src bucket item count to get to: 50000 curr count: 24161
2021/12/09 01:29:26 Waiting for src bucket item count to get to: 50000 curr count: 24161
2021/12/09 01:29:31 Waiting for src bucket item count to get to: 50000 curr count: 34857
2021/12/09 01:29:36 Waiting for src bucket item count to get to: 50000 curr count: 34857
2021/12/09 01:29:41 Waiting for src bucket item count to get to: 50000 curr count: 34857
2021/12/09 01:29:46 Waiting for src bucket item count to get to: 50000 curr count: 45787
2021/12/09 01:29:51 Waiting for src bucket item count to get to: 50000 curr count: 45787
2021/12/09 01:30:01 Waiting for dst bucket item count to get to: 50000 curr count: 39221
2021/12/09 01:30:06 Waiting for dst bucket item count to get to: 50000 curr count: 39221
2021/12/09 01:30:06 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/09 01:30:06 Post SET ops. Expected item count: 50000 got 50000
2021/12/09 01:30:06 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 01:30:11 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/09 01:30:16 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/09 01:30:21 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/09 01:30:27 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/09 01:30:32 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/09 01:30:37 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/09 01:30:37 App: TestDuplicateTimerFiringKVRebOut got un-deployed
2021/12/09 01:30:37 No eventing-con process running
2021/12/09 01:30:37 Function: TestDuplicateTimerFiringKVRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringKVRebOut deleting in the background"
}
2021/12/09 01:30:38 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:30:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 01:30:48 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/09 01:30:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 01:30:50 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:30:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 01:31:00 Waiting for dst bucket item count to get to: 0 curr count: 19789
2021/12/09 01:31:05 Waiting for dst bucket item count to get to: 0 curr count: 19789
2021/12/09 01:31:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringKVRebOut (179.22s)
=== RUN   TestDuplicateTimerFiringKVSwapReb
2021/12/09 01:31:15 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/09 01:31:15 App: TestDuplicateTimerFiringKVSwapReb got un-deployed
2021/12/09 01:31:15 No eventing-con process running
2021/12/09 01:31:15 Function: TestDuplicateTimerFiringKVSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringKVSwapReb not found"
 }
}
2021/12/09 01:31:17 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:31:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 01:31:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 01:31:24 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:31:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 01:31:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:31:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 01:31:34 Deploying app: TestDuplicateTimerFiringKVSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/09 01:31:35 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringKVSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/09 01:31:40 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get deployed
2021/12/09 01:31:45 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get deployed
2021/12/09 01:31:50 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get deployed
2021/12/09 01:31:55 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get deployed
2021/12/09 01:31:55 App: TestDuplicateTimerFiringKVSwapReb got deployed
2021/12/09 01:31:55 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/09 01:31:55 Starting bucket ops to source bucket
2021/12/09 01:31:56 Removing node(s): [] from the cluster
2021/12/09 01:32:01 Rebalance progress: 12.5
2021/12/09 01:32:06 Rebalance progress: 12.5
2021/12/09 01:32:11 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:32:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:32:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 01:32:11 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/09 01:32:11 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/09 01:32:16 Rebalance progress: 12.5
2021/12/09 01:32:21 Rebalance progress: 12.5
2021/12/09 01:32:26 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:32:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:32:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 01:32:26 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/09 01:32:31 Rebalance progress: 12.5
2021/12/09 01:32:36 Rebalance progress: 12.5
2021/12/09 01:32:41 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:32:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:32:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 01:32:41 Waiting for src bucket item count to get to: 50000 curr count: 49958
2021/12/09 01:32:46 Waiting for src bucket item count to get to: 50000 curr count: 49958
2021/12/09 01:32:56 Waiting for dst bucket item count to get to: 50000 curr count: 46430
2021/12/09 01:33:01 Waiting for dst bucket item count to get to: 50000 curr count: 46430
2021/12/09 01:33:01 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/09 01:33:01 Post SET ops. Expected item count: 50000 got 50000
2021/12/09 01:33:01 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 01:33:06 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/09 01:33:11 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/09 01:33:16 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/09 01:33:21 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/09 01:33:26 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/09 01:33:31 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/09 01:33:31 App: TestDuplicateTimerFiringKVSwapReb got un-deployed
2021/12/09 01:33:31 No eventing-con process running
2021/12/09 01:33:31 Function: TestDuplicateTimerFiringKVSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringKVSwapReb deleting in the background"
}
2021/12/09 01:33:33 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:33:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 01:33:43 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/09 01:33:48 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/09 01:33:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 01:33:50 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:33:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 01:34:00 Waiting for dst bucket item count to get to: 0 curr count: 7489
2021/12/09 01:34:05 Waiting for dst bucket item count to get to: 0 curr count: 7489
2021/12/09 01:34:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringKVSwapReb (179.71s)
=== RUN   TestKVRebOnUpdateBucketOpOneByOne
2021/12/09 01:34:15 Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 01:34:15 App: bucket_op_on_update got un-deployed
2021/12/09 01:34:15 No eventing-con process running
2021/12/09 01:34:15 Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/09 01:34:17 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:34:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 01:34:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 01:34:24 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:34:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 01:34:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:34:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 01:34:34 Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/09 01:34:35 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/09 01:34:40 Starting bucket ops to source bucket
2021/12/09 01:34:45 Waiting for app: bucket_op_on_update to get deployed
2021/12/09 01:34:50 Waiting for app: bucket_op_on_update to get deployed
2021/12/09 01:34:55 Waiting for app: bucket_op_on_update to get deployed
2021/12/09 01:34:55 App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:34:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:34:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 01:34:55 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/09 01:34:55 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/09 01:35:00 Rebalance progress: 12.5
2021/12/09 01:35:05 Rebalance progress: 12.5
2021/12/09 01:35:10 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:35:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:35:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 01:35:10 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/09 01:35:11 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/09 01:35:16 Rebalance progress: 12.5
2021/12/09 01:35:21 Rebalance progress: 12.5
2021/12/09 01:35:26 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:35:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:35:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 01:35:26 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: kv to the cluster
2021/12/09 01:35:27 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/09 01:35:32 Rebalance progress: 12.5
2021/12/09 01:35:37 Rebalance progress: 12.5
2021/12/09 01:35:42 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:35:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:35:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 01:35:42 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/09 01:35:47 Rebalance progress: 12.5
2021/12/09 01:35:52 Rebalance progress: 12.5
2021/12/09 01:35:57 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:35:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:35:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 01:35:57 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/09 01:36:02 Rebalance progress: 12.5
2021/12/09 01:36:07 Rebalance progress: 12.5
2021/12/09 01:36:12 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:36:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:36:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 01:36:12 clusterSetup.go:253: Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/09 01:36:17 Rebalance progress: 12.5
2021/12/09 01:36:22 Rebalance progress: 12.5
2021/12/09 01:36:27 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:36:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:36:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 01:36:27 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 01:36:32 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 01:36:37 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 01:36:42 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 01:36:47 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 01:36:52 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 01:36:57 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 01:36:57 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/09 01:36:57 eventingSetup.go:46: No eventing-con process running
2021/12/09 01:36:57 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/09 01:36:59 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:37:04 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 01:37:09 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 10704
2021/12/09 01:37:09 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 01:37:11 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:37:16 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 01:37:21 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 8848
2021/12/09 01:37:26 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 8848
2021/12/09 01:37:26 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebOnUpdateBucketOpOneByOne (200.75s)
=== RUN   TestKVForceFailover
2021/12/09 01:37:41 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 01:37:41 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/09 01:37:41 eventingSetup.go:46: No eventing-con process running
2021/12/09 01:37:41 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/09 01:37:43 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:37:48 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 01:37:48 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 01:37:49 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:37:54 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 01:37:54 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 01:37:54 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/09 01:37:55 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/09 01:38:00 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:38:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 01:38:00 Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/09 01:38:01 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/09 01:38:01 Starting bucket ops to source bucket
2021/12/09 01:38:06 Waiting for app: bucket_op_on_update to get deployed
2021/12/09 01:38:11 Waiting for app: bucket_op_on_update to get deployed
2021/12/09 01:38:16 Waiting for app: bucket_op_on_update to get deployed
2021/12/09 01:38:21 Waiting for app: bucket_op_on_update to get deployed
2021/12/09 01:38:21 App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:38:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:38:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 01:38:21 clusterSetup.go:229: Failing over: [127.0.0.1:19001]
2021/12/09 01:38:41 clusterSetup.go:237: Kicking off failover recovery, type: full
2021/12/09 01:38:41 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/09 01:38:47 Rebalance progress: 12.5
2021/12/09 01:38:52 Rebalance progress: 12.5
2021/12/09 01:38:57 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:38:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:38:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 01:38:57 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002] from the cluster
2021/12/09 01:39:02 Rebalance progress: 12.5
2021/12/09 01:39:07 Rebalance progress: 12.5
2021/12/09 01:39:12 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:39:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:39:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 01:39:12 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 01:39:17 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 01:39:22 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 01:39:27 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 01:39:32 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 01:39:37 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 01:39:42 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 01:39:42 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/09 01:39:42 eventingSetup.go:46: No eventing-con process running
2021/12/09 01:39:42 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/09 01:39:44 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:39:49 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 01:39:54 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7054
2021/12/09 01:39:59 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7054
2021/12/09 01:39:59 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 01:40:01 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:40:06 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 01:40:11 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4806
2021/12/09 01:40:16 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4806
2021/12/09 01:40:16 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVForceFailover (169.82s)
=== RUN   TestKVHardFailoverRecovery
2021/12/09 01:40:26 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 01:40:26 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/09 01:40:26 eventingSetup.go:46: No eventing-con process running
2021/12/09 01:40:26 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/09 01:40:27 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:40:32 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 01:40:32 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 01:40:34 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:40:39 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 01:40:39 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:40:44 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 01:40:44 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/09 01:40:45 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/09 01:40:50 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/09 01:40:55 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/09 01:41:00 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/09 01:41:05 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/09 01:41:05 clusterSetup.go:354: App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:41:05 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:41:05 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 01:41:05 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/09 01:41:06 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/09 01:41:11 Rebalance progress: 12.5
2021/12/09 01:41:16 Rebalance progress: 12.5
2021/12/09 01:41:21 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:41:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:41:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 01:41:21 clusterSetup.go:229: Failing over: [127.0.0.1:19001]
2021/12/09 01:42:21 clusterSetup.go:237: Kicking off failover recovery, type: full
2021/12/09 01:42:26 Rebalance progress: 12.5
2021/12/09 01:42:31 Rebalance progress: 12.5
2021/12/09 01:42:36 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:42:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:42:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 01:42:36 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/09 01:42:41 Rebalance progress: 12.5
2021/12/09 01:42:46 Rebalance progress: 12.5
2021/12/09 01:42:51 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:42:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:42:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 01:42:51 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 01:42:56 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 01:43:01 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 01:43:06 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 01:43:11 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 01:43:16 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 01:43:21 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 01:43:21 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/09 01:43:21 eventingSetup.go:46: No eventing-con process running
2021/12/09 01:43:21 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/09 01:43:23 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:43:28 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 01:43:33 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12053
2021/12/09 01:43:38 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12053
2021/12/09 01:43:38 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 01:43:40 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:43:45 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 01:43:50 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 421
2021/12/09 01:43:55 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 421
2021/12/09 01:43:55 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVHardFailoverRecovery (219.33s)
=== RUN   TestKVRebOnUpdateBucketOpAllAtOnce
2021/12/09 01:44:05 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 01:44:05 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/09 01:44:05 eventingSetup.go:46: No eventing-con process running
2021/12/09 01:44:05 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/09 01:44:07 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:44:12 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 01:44:12 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 01:44:14 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:44:19 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 01:44:19 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:44:24 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 01:44:24 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/09 01:44:25 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/09 01:44:30 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/09 01:44:35 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/09 01:44:40 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/09 01:44:45 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/09 01:44:45 clusterSetup.go:354: App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:44:45 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:44:45 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 01:44:45 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/09 01:44:45 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/09 01:44:46 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: kv to the cluster
2021/12/09 01:44:46 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/09 01:44:51 Rebalance progress: 12.5
2021/12/09 01:44:56 Rebalance progress: 12.5
2021/12/09 01:45:01 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:45:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:45:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 01:45:01 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002 127.0.0.1:19003] from the cluster
2021/12/09 01:45:06 Rebalance progress: 12.5
2021/12/09 01:45:11 Rebalance progress: 12.5
2021/12/09 01:45:16 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:45:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:45:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 01:45:16 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 01:45:21 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 01:45:26 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 01:45:31 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 01:45:36 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 01:45:41 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 01:45:46 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 01:45:46 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/09 01:45:46 eventingSetup.go:46: No eventing-con process running
2021/12/09 01:45:46 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/09 01:45:48 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:45:53 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 01:45:58 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4612
2021/12/09 01:45:58 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 01:46:00 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:46:05 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 01:46:10 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 251
2021/12/09 01:46:15 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 251
2021/12/09 01:46:15 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebOnUpdateBucketOpAllAtOnce (139.99s)
=== RUN   TestKVSwapRebOnUpdateBucketOp
2021/12/09 01:46:25 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 01:46:25 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/09 01:46:25 eventingSetup.go:46: No eventing-con process running
2021/12/09 01:46:25 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/09 01:46:27 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:46:32 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 01:46:32 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 01:46:33 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:46:38 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 01:46:38 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:46:38 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 01:46:38 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/09 01:46:40 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/09 01:46:45 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/09 01:46:50 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/09 01:46:55 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/09 01:47:00 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/09 01:47:00 clusterSetup.go:354: App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:47:00 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:47:00 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 01:47:00 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/09 01:47:00 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/09 01:47:05 Rebalance progress: 12.5
2021/12/09 01:47:10 Rebalance progress: 12.5
2021/12/09 01:47:15 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:47:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:47:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 01:47:15 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/09 01:47:16 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/09 01:47:21 Rebalance progress: 12.5
2021/12/09 01:47:26 Rebalance progress: 12.5
2021/12/09 01:47:31 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:47:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:47:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 01:47:31 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/09 01:47:36 Rebalance progress: 12.5
2021/12/09 01:47:41 Rebalance progress: 12.5
2021/12/09 01:47:46 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:47:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:47:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 01:47:46 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 01:47:51 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 01:47:56 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 01:48:01 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 01:48:06 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 01:48:11 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 01:48:11 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/09 01:48:11 eventingSetup.go:46: No eventing-con process running
2021/12/09 01:48:11 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/09 01:48:13 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:48:18 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 01:48:23 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6120
2021/12/09 01:48:28 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6120
2021/12/09 01:48:28 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 01:48:30 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:48:35 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 01:48:35 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 01:48:40 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 01:48:40 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/09 01:48:40 eventingSetup.go:46: No eventing-con process running
2021/12/09 01:48:40 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/09 01:48:42 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:48:47 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 01:48:47 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 01:48:48 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:48:53 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 01:48:53 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVSwapRebOnUpdateBucketOp (158.45s)
=== RUN   TestKVRebalanceWithMultipleHandlers
2021/12/09 01:49:03 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 01:49:03 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/09 01:49:03 eventingSetup.go:46: No eventing-con process running
2021/12/09 01:49:03 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/09 01:49:05 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:49:10 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 01:49:10 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 01:49:12 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:49:17 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 01:49:17 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 01:49:22 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/09 01:49:22 clusterSetup.go:405: App: sys_test_bucket_op got un-deployed
2021/12/09 01:49:22 eventingSetup.go:46: No eventing-con process running
2021/12/09 01:49:22 eventingSetup.go:418: Function: sys_test_bucket_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: sys_test_bucket_op not found"
 }
}
2021/12/09 01:49:24 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:49:29 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 01:49:29 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 01:49:30 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:49:35 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 01:49:35 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:49:35 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 01:49:35 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/09 01:49:36 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:49:36 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 01:49:37 eventingSetup.go:125: Deploying app: sys_test_bucket_op. Item count src bucket: 0 dst bucket: 0
2021/12/09 01:49:38 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'sys_test_bucket_op' in metakv",
  "warnings": null
 }
}
2021/12/09 01:49:43 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/09 01:49:48 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/09 01:49:53 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/09 01:49:58 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/09 01:49:58 clusterSetup.go:354: App: bucket_op_on_update got deployed
2021/12/09 01:50:03 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/09 01:50:08 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/09 01:50:13 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/09 01:50:13 clusterSetup.go:354: App: sys_test_bucket_op got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:50:13 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:50:13 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 01:50:13 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/09 01:50:13 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/09 01:50:18 Rebalance progress: 12.5
2021/12/09 01:50:23 Rebalance progress: 12.5
2021/12/09 01:50:28 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:50:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:50:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 01:50:28 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/09 01:50:29 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/09 01:50:34 Rebalance progress: 12.5
2021/12/09 01:50:39 Rebalance progress: 12.5
2021/12/09 01:50:44 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:50:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:50:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 01:50:44 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: kv to the cluster
2021/12/09 01:50:44 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/09 01:50:50 Rebalance progress: 12.5
2021/12/09 01:50:55 Rebalance progress: 12.5
2021/12/09 01:51:00 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:51:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:51:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 01:51:00 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/09 01:51:05 Rebalance progress: 12.5
2021/12/09 01:51:10 Rebalance progress: 12.5
2021/12/09 01:51:15 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:51:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:51:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 01:51:15 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/09 01:51:20 Rebalance progress: 12.5
2021/12/09 01:51:25 Rebalance progress: 12.5
2021/12/09 01:51:30 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:51:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:51:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 01:51:30 clusterSetup.go:253: Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/09 01:51:35 Rebalance progress: 12.5
2021/12/09 01:51:40 Rebalance progress: 12.5
2021/12/09 01:51:45 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:51:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:51:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 01:51:45 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 01:51:50 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 01:51:55 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 01:52:00 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 01:52:05 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 01:52:10 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 01:52:10 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/09 01:52:10 eventingSetup.go:46: No eventing-con process running
2021/12/09 01:52:10 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/09 01:52:14 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:52:19 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 01:52:24 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12220
2021/12/09 01:52:29 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12220
2021/12/09 01:52:29 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 01:52:31 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:52:36 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 01:52:41 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 9629
2021/12/09 01:52:46 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 9629
2021/12/09 01:52:46 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 01:52:46 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 01:52:51 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/09 01:52:56 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/09 01:53:01 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/09 01:53:06 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/09 01:53:11 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/09 01:53:16 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/09 01:53:16 clusterSetup.go:405: App: sys_test_bucket_op got un-deployed
2021/12/09 01:53:16 eventingSetup.go:46: No eventing-con process running
2021/12/09 01:53:16 eventingSetup.go:418: Function: sys_test_bucket_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: sys_test_bucket_op deleting in the background"
}
2021/12/09 01:53:18 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:53:23 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 01:53:23 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 01:53:25 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:53:30 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 01:53:30 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebalanceWithMultipleHandlers (276.25s)
=== RUN   TestKVSwapRebalanceWithMultipleHandlers
2021/12/09 01:53:40 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 01:53:40 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/09 01:53:40 eventingSetup.go:46: No eventing-con process running
2021/12/09 01:53:40 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/09 01:53:41 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:53:46 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 01:53:46 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 01:53:48 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:53:53 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 01:53:53 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 01:53:58 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/09 01:53:58 clusterSetup.go:405: App: sys_test_bucket_op got un-deployed
2021/12/09 01:53:58 eventingSetup.go:46: No eventing-con process running
2021/12/09 01:53:58 eventingSetup.go:418: Function: sys_test_bucket_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: sys_test_bucket_op not found"
 }
}
2021/12/09 01:54:00 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:54:05 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 01:54:05 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 01:54:07 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:54:12 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 01:54:12 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:54:12 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 01:54:12 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/09 01:54:13 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:54:13 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 01:54:13 eventingSetup.go:125: Deploying app: sys_test_bucket_op. Item count src bucket: 0 dst bucket: 0
2021/12/09 01:54:14 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'sys_test_bucket_op' in metakv",
  "warnings": null
 }
}
2021/12/09 01:54:19 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/09 01:54:24 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/09 01:54:29 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/09 01:54:34 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/09 01:54:34 clusterSetup.go:354: App: bucket_op_on_update got deployed
2021/12/09 01:54:39 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/09 01:54:44 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/09 01:54:49 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/09 01:54:49 clusterSetup.go:354: App: sys_test_bucket_op got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:54:49 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:54:49 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 01:54:49 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/09 01:54:49 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/09 01:54:55 Rebalance progress: 12.5
2021/12/09 01:55:00 Rebalance progress: 12.5
2021/12/09 01:55:05 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:55:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:55:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 01:55:05 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/09 01:55:05 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/09 01:55:10 Rebalance progress: 12.5
2021/12/09 01:55:15 Rebalance progress: 12.5
2021/12/09 01:55:20 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:55:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:55:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 01:55:20 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/09 01:55:25 Rebalance progress: 12.5
2021/12/09 01:55:30 Rebalance progress: 12.5
2021/12/09 01:55:35 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:55:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:55:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 01:55:35 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 01:55:40 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 01:55:45 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 01:55:50 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 01:55:55 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 01:56:00 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 01:56:05 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 01:56:05 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/09 01:56:05 eventingSetup.go:46: No eventing-con process running
2021/12/09 01:56:05 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/09 01:56:09 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:56:14 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 01:56:19 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7596
2021/12/09 01:56:19 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 01:56:21 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:56:26 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 01:56:31 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5891
2021/12/09 01:56:36 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5891
2021/12/09 01:56:36 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 01:56:36 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 01:56:41 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/09 01:56:46 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/09 01:56:51 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/09 01:56:56 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/09 01:57:01 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/09 01:57:06 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/09 01:57:06 clusterSetup.go:405: App: sys_test_bucket_op got un-deployed
2021/12/09 01:57:06 eventingSetup.go:46: No eventing-con process running
2021/12/09 01:57:06 eventingSetup.go:418: Function: sys_test_bucket_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: sys_test_bucket_op deleting in the background"
}
2021/12/09 01:57:08 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:57:13 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 01:57:13 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 01:57:15 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:57:20 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 01:57:20 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVSwapRebalanceWithMultipleHandlers (230.11s)
=== RUN   TestKVRebStopStartKVOpsOnUpdateBucketOpOneByOne
2021/12/09 01:57:30 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 01:57:30 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/09 01:57:30 eventingSetup.go:46: No eventing-con process running
2021/12/09 01:57:30 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/09 01:57:32 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:57:37 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 01:57:37 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 01:57:38 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:57:43 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 01:57:43 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:57:48 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 01:57:48 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/09 01:57:49 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/09 01:57:54 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/09 01:57:59 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/09 01:58:04 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/09 01:58:09 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/09 01:58:09 clusterSetup.go:354: App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:58:09 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:58:09 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 01:58:09 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/09 01:58:10 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/09 01:58:30 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:58:30 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:58:30 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:58:40 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:58:40 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 01:58:40 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/09 01:58:45 Rebalance progress: 12.5
2021/12/09 01:58:50 Rebalance progress: 12.5
2021/12/09 01:58:55 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:58:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:58:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 01:58:56 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/09 01:58:56 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/09 01:59:01 Rebalance progress: 12.5
2021/12/09 01:59:06 Rebalance progress: 12.5
2021/12/09 01:59:11 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:59:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:59:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 01:59:11 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/09 01:59:16 Rebalance progress: 12.5
2021/12/09 01:59:21 Rebalance progress: 12.5
2021/12/09 01:59:26 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:59:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 01:59:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 01:59:26 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 01:59:31 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 01:59:36 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 01:59:41 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 01:59:47 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 01:59:52 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 01:59:57 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 01:59:57 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/09 01:59:57 eventingSetup.go:46: No eventing-con process running
2021/12/09 01:59:57 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/09 01:59:58 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 02:00:03 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 02:00:08 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 9176
2021/12/09 02:00:08 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 02:00:10 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 02:00:15 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 02:00:20 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2109
2021/12/09 02:00:25 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2109
2021/12/09 02:00:25 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebStopStartKVOpsOnUpdateBucketOpOneByOne (185.38s)
=== RUN   TestKVFailoverOnUpdateBucketOp
2021/12/09 02:00:35 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 02:00:35 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/09 02:00:35 eventingSetup.go:46: No eventing-con process running
2021/12/09 02:00:35 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/09 02:00:37 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 02:00:42 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 02:00:42 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 02:00:44 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 02:00:49 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 02:00:49 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 02:00:54 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 02:00:54 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/09 02:00:55 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/09 02:01:00 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/09 02:01:05 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/09 02:01:10 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/09 02:01:15 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/09 02:01:15 clusterSetup.go:354: App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 02:01:15 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 02:01:15 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 02:01:15 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/09 02:01:15 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/09 02:01:20 Rebalance progress: 12.5
2021/12/09 02:01:25 Rebalance progress: 12.5
2021/12/09 02:01:30 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 02:01:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 02:01:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 02:01:30 clusterSetup.go:229: Failing over: [127.0.0.1:19001]
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 02:01:40 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 02:01:40 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 02:01:40 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/09 02:01:45 Rebalance progress: 12.5
2021/12/09 02:01:50 Rebalance progress: 12.5
2021/12/09 02:01:55 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 02:01:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 02:01:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 02:01:55 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/09 02:01:55 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/09 02:02:00 Rebalance progress: 12.5
2021/12/09 02:02:05 Rebalance progress: 12.5
2021/12/09 02:02:10 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 02:02:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 02:02:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 02:02:11 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/09 02:02:16 Rebalance progress: 12.5
2021/12/09 02:02:21 Rebalance progress: 12.5
2021/12/09 02:02:26 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 02:02:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 02:02:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 02:02:26 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 02:02:31 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 02:02:36 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 02:02:41 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 02:02:46 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 02:02:51 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 02:02:56 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 02:02:56 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/09 02:02:56 eventingSetup.go:46: No eventing-con process running
2021/12/09 02:02:56 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/09 02:02:58 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 02:03:03 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 02:03:08 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 8569
2021/12/09 02:03:08 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 02:03:09 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 02:03:14 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 02:03:14 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVFailoverOnUpdateBucketOp (169.31s)
=== RUN   TestBootstrapAfterKVHardFailover
2021/12/09 02:03:19 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/09 02:03:20 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/09 02:03:25 Rebalance progress: 100
2021/12/09 02:03:25 Failing over: [127.0.0.1:19001]
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 02:03:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 02:03:55 Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/09 02:03:56 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/09 02:04:01 Removing node(s): [] from the cluster
2021/12/09 02:04:01 Starting bucket ops to source bucket
2021/12/09 02:04:06 Rebalance failed. See logs for detailed reason. You can try again.
2021/12/09 02:04:11 Waiting for app: bucket_op_on_update to get deployed
2021/12/09 02:04:16 Waiting for app: bucket_op_on_update to get deployed
2021/12/09 02:04:16 App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 02:04:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 02:04:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 02:04:16 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 02:04:21 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 02:04:26 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 02:04:31 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 02:04:36 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 02:04:41 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 02:04:46 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 02:04:46 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/09 02:04:46 eventingSetup.go:46: No eventing-con process running
2021/12/09 02:04:46 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/09 02:04:48 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 02:04:53 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 02:04:58 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1502
2021/12/09 02:04:58 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 02:05:00 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 02:05:05 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 02:05:10 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 92
2021/12/09 02:05:15 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 92
2021/12/09 02:05:15 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestBootstrapAfterKVHardFailover (120.53s)
PASS
ok  	github.com/couchbase/eventing/tests/functional_tests	12042.198s

Rebalance tests took 3 hours, 20 minutes and 44 seconds.

Testing Succeeded


Version: versions-08.12.2021-18.27.cfg
Build Log: make-08.12.2021-18.27.log
Server Log: logs-08.12.2021-18.27.tar.gz

Finished